Flaw in the Rating System?

Discussion in 'Bug Reporting' started by Whystler, Sep 20, 2008.

  1. Whystler
    Whystler Member

    I think there's a flaw in the rating system, especially when there is the incentive to "game it" during contests.

    When I say "gaming the system" I mean the ability to win or benefit from a system by using it in a way that it is not intended.

    For example, there are plenty of good products out there that deserve a 4 or a 5 rating. And there are a few duds that are either for testing or by people who are learning that deserve a 2 or a 3. With the stakes of the contest so high (getting your expensive object printed, $250 in printing coupons, exposure of your work at conferences...) the pressure is on contestants to review the competing products lower than you normally would.

    I like the rating system in general. But I think maybe a different voting system should be in place for the contests. I have always like voting systems where you can choose more than one (ie. your top 3 or top 10 favourite items), and then the finalists are voted upon in the end.

    What do you think?

  2. joris
    joris Member

    Currently we are using the regular rating system and trusting people.

    I do agree that we could come up with systems that are more airtight.

    What, we would like to do most is encourage enough people to vote so that any person trying to game the system does not exert enough influence to change a rating of a model.

    We would also really like to keep on assuming that everyone will keep their ratings as honest as possible. Is that naive?

    I will check and run some numbers of people's average ratings and things to see if there is a noticeable difference in the ratings of contest models.
  3. Whystler
    Whystler Member
    I guess my experience in IMVU is coming through here. Gaming the system in IMVU is rampant. I've been involved in testing that system on various levels, and I know that before a feature is even released, there is much discussion about how the feature can be gamed (so that people can get more credits/money).

    Perhaps if you have a smaller testing base of users, gaming the system is not an issue here. It will be an issue in the future though, when you have a larger base of users if you are running contests.

    Everyone does seem to be pretty supportive and honest here though at the moment :)

  4. Christel
    Christel Member
    Ideally, contest submissions should be anonymous. Another solution would be to make ratings NOT anonymous.
    But I agree with Joris that for now it's all just for fun and in good spirit.

    Personally I didn't manage to come up with anything as magnificent as I'd hoped for - due to the usual lack of time and talent ;) - and I have no problems giving high ratings to other entries. I do tend to go for aesthetics more than functionalism though. Maybe that's not entirely in line with the purpose of the contest :)
  5. Whystler
    Whystler Member
    I like this idea * A LOT*. It really helps the transparency of rating.

    I have also seen rating systems where you can add an accompanying short message. This is good for the person whose item is rated because then, rating becomes a learning experience.