Add negative stars. If someone gets down voted enough their stars turn red to signify negative stars and people should ignore them instantly. Also maybe add some good stuff for people that always get up voted. Like a little badge to make people trust them.
Why does everyone hate me nowadays?