What kind of tree is this?

It blooms twice a year, typically late spring and in the fall. VERY, VERY fragrant with purple small, stringy flowers. Immature berries are green, mature ones are a yellowy brown. The leaves are tapered and about 2.5" - 3" long and about .5" - .75" wide. The bark on this tree is not smooth, but is not like an oak either (which is the only other non-fruit tree I can easily identify, that isn't a type of cedar or christmas tree).

This "thing" messes with my allergies something awful when in bloom, so any help is appreciated.