Ghosts can attach themselves to certain objects. Generally, this only happens if the object was significant to the being in its lifetime. It sounds like you might have just had a ghost in both places you moved to; however, there is most definitely a possibility that the hauntings were linked to the mirror. It sounds, though, like different things were happening in the different places, and most of the time, if you have a ghost connected to an object (i.e. the mirror), the same basic things will happen wherever the object is taken.
Have fun with your new ghosts! ^.^ Haha. Really, though. Keep in mind that ghosts cannot hurt you, and, aside from making your heart beat faster out of fright or giving you a chill every once in awhile, they pose no danger.
Poltergeists are another matter. It doesn't sound like you have one, though. Even then, you'd have to have an extremely powerful poltergeist to actually physically harm you.
Don't smash the mirror. That would be plain rude. I doubt the spirit means any ill-will, and smashing its "home" (I loosely use the term "home") would not be very polite, and that would absolutely make it turn against you. If you really have the desire to get rid of the spirit, get rid of the mirror.