Buffy probably used a combination of kick boxing, taekwondo, karate, and a little muay thai.
Pretty sure I read somewhere that Sarah Michelle Gellar had previous taekwondo experience prior to her 'Buffy' role.
Faith used a lot of the "spinning back fist" move, but her stance and the general way in which she fought a lot (S3 "Graduation Day Pt. 1) she's using a lot of flashy boxing techniques, dancing around her opponents strikes a lot and the spinning back fist, for example.
Spike tends to use a lot of muay thai, styles relying heavily on the kick, knee and clinch techniques it looks like.
Its kind of ironic, but Buffy herself doesn't really have much fighting technique. A lot of her moves would be impractical in a real fight, not all of them though. But still a surprising amount. A lot of the fight scenes were choreographed more for entertainment value than real practicality, which would explain this.
I always wanted to know why the rest of the Scooby Gang never bothered with taking a few fighting classes. Having known Buffy since high school, having been fighting by her side, you'd think they would have thought a little training useful. Its not like you have to be physically super powered to pick some technique up.
Gunn, Wesley - and even Cordelia, a bit - put time into developing fighting skills. Not so much Xander or Willow, even though they could have used it plenty of times.