Ozyman wrote:

I actually have been waiting to rank boards until I have played them at least 3 or 4 times, because I don't feel qualified to rank them.  If there was some kind of weighting system, I would be more willing to rank after only playing once, because I would know my vote would not get counted as highly.

Remember, you can edit your review any time and as often as you like after you initially post it.

Ozyman wrote:

Alpha wrote:

I support the weighted rating system as well and now all we have to do is get more people to leave a rating for boards.  I glad that this was discussed.

I actually have been waiting to rank boards until I have played them at least 3 or 4 times, because I don't feel qualified to rank them.  If there was some kind of weighting system, I would be more willing to rank after only playing once, because I would know my vote would not get counted as highly.

We are similar, there are many boards that I have played once or even a few time, didn't like and have not played again.  Since I know I probably missed something about the board (and I know I didn't read the description), I feel guilty leaving a rating.  There are other times where I don't like the style of the board (i.e. basic continents, very long empire building, strategy only, etc.) and so I don't play again and don't feel that I should rate a board based on the fact that I don't like the style.

As discussed ages ago the site has been calculating a parallel Bayesian ratings system for a while now. Now we have a fair amount of ratings data (over 1000) the data should be valid.

The Bayesian rating is a formula used by statisticians and web developers to obtain a more accurate rating from votes provided by the users. The formula is:

W = (v / (v+m) ) * R + (m / (v + m)) * C

Where:

W = Weighted Rating
v = Number of votes
m = Minimum number of votes
R = The average score
C = The average vote across the entire dataset.

I have set m = 10 currently.

To compare the figures, here is a list of the top 20 board ranked by average Rating (i.e. the current system)

#	Board Name	Average Rating	Bayesian Rating	Number of Ratings
1	Capital Clue	10.00	8.24	1
2	WarGear 2210	9.63	8.46	8
3	War of the Titans	9.50	8.09	4
4	Gotham	9.25	8.02	4
5	Resident Wargear	9.10	8.31	10
6	Infection	9.08	8.40	13
7	Mario 3 -- World 1	9.00	8.13	7
8	Wargear: The Gathering	9.00	8.13	7
9	Another Brick in the Wall	9.00	8.01	5
10	Wargate SG-1	9.00	8.01	5
11	Cumberdale's Bomb Factory	9.00	7.86	3
12	Ribbon World	9.00	7.86	3
13	Salem's Dusk: Classic	9.00	7.86	3
14	Chain Game	9.00	7.77	2
15	Lord Of Destruction	9.00	7.77	2
16	Titans of Italy	9.00	7.66	1
17	Castles	8.96	8.56	26
18	Spy vs Spy	8.93	8.34	14
19	Mobs of New York	8.83	8.01	6
20	Holy Land	8.67	7.79	3

Note how the top rated board only received a single vote of 10 so jumped straight to the top of the rankings. Here is the same chart but ordered by the Bayesian average:

#	Board Name	Average Rating	Bayesian Rating	Number of Ratings
1	Castles	8.96	8.56	26
2	WarGear 2210	9.63	8.46	8
3	Infection	9.08	8.40	13
4	Spy vs Spy	8.93	8.34	14
5	Resident Wargear	9.10	8.31	10
6	Capital Clue	10.00	8.24	1
7	Gauntlet	8.65	8.23	17
8	Mario 3 -- World 1	9.00	8.13	7
9	Wargear: The Gathering	9.00	8.13	7
10	War of the Titans	9.50	8.09	4
11	WarGear Warfare	8.15	8.06	60
12	Hordes of Africa	8.40	8.05	15
13	Gotham	9.25	8.02	4
14	Another Brick in the Wall	9.00	8.01	5
15	Wargate SG-1	9.00	8.01	5
16	Mobs of New York	8.83	8.01	6
17	Vertigo	8.50	8.01	10
18	GearStorm	8.36	7.96	11
19	Clue	8.38	7.90	8
20	Fallout	8.50	7.89	6

You can see the adjustments that the Bayesian average makes - dropping the board with the single vote down the ratings and promoting the ones with more votes. This will become more important when the board list changes go live (still a while off!) as sorting by player rating will be one of the default views.

What do you think of this type of weighting?

What's Capital Clue?

Is there a minimum number of Ratings?  The Capital Clue being #6 with only 1 Rating to be #6 seems off to me.

Perhaps WarGear Warfare should somehow be higher?  60 ratings is a lot, especially with a decent average.

If this was used for the top rated board lists it would probably be filtered with a minimum number of ratings required. This was really just to illustrate how using a Bayesian average will affect the rating for the most popular boards. I think it makes the system fairer.

Actually I am not sure how Capital Clue got in the list, it's in Dev but somehow it got a rating. It was probably a copy of 'Clue' that inherited the original's rating. So ignore that one (even though it is a good example of the system working!)

tom wrote:

This was really just to illustrate how using a Bayesian average will affect the rating for the most popular boards. I think it makes the system fairer.

Oh ya, I forgot to mention that the Bayesian list looks great.  Definitely seems more reflective of overall ratings.

The Bayesian definitely looks better than a straight average, but it's different sort of weighting than we have been talking about. It "normalizes" for # of votes, and the weighting we have been talking about "normalizes" for experience on a map. Ideally, I think having both combined would be for the best, but I'm not sure how the math would be done. If I had to choose one, I think I'd rather have the experience weighting, combined with some small minimum # of reviews before a board shows up on the rankings.

I think in the long run the problem that bayesian weighting corrects is largely self-correcting. If a board rockets to the top based upon a few very high rankings, that will encourage others to play & review it, and if it is not worthy of it's high rank, they will be more likely to register their dissatisfaction.

Weighting based upon experience should help niche boards or boards that are more unusual or complex from being sunk by a few voters who didn't understand how to play.

Agree with Ozy.  Would be nice to somehow add weighting to the Bayesian.  I still like my suggestion about taking out a few outliers as well, but there aren't that many boards that have been reviewed the 15+ times that you'd probably want to have.

As usual I am interested in how this would actually work in practice :) Would you take the number of games played at the time of the review? So if I rated a board after one game but then subsequently played 100 games on it, would my weighting still be as if I had only played one game or would my rating weighting steadily increase as I completed the games?

Also for the mathematicians, how would this formula be calculated - e.g. given the following stats:

Player 1 rated a 9 (played 4 games at time of rating)
Player 2 rated a 3 (played 1 game at time of rating)
Player 3 rated a 8 (played 10 games at time of rating)
Player 4 rated a 10 (played 7 games at time of rating)

tom wrote:

As usual I am interested in how this would actually work in practice :) Would you take the number of games played at the time of the review? So if I rated a board after one game but then subsequently played 100 games on it, would my weighting still be as if I had only played one game or would my rating weighting steadily increase as I completed the games?

Also for the mathematicians, how would this formula be calculated - e.g. given the following stats:

Player 1 rated a 9 (played 4 games at time of rating)
Player 2 rated a 3 (played 1 game at time of rating)
Player 3 rated a 8 (played 10 games at time of rating)
Player 4 rated a 10 (played 7 games at time of rating)

In practice:  You are currently using:

W = (v / (v+m) ) * R + (m / (v + m)) * C

Where:

W = Weighted Rating
v = Number of votes
m = Minimum number of votes
R = The average score
C = The average vote across the entire dataset.

Using Oxy's suggestion above, at the time of the rating use the number of games, so with your example:
Player 1, rates 9 with 4 games, count this as 1.75 votes worth 1.75*9=15.75
Player 2, rates 3 with 1 game, count this as 1 vote worth 3
Player 3, rates 8 with 10 games, count this as 2 votes worth 2*8=16
Player 4, rates 10 with 7 games, count this as 2 votes worth 2*10=20
so basic weighted rating = 54.75/6.75 = 8.11 (vs. unweighted 7.5)
Bayesian
  W is what it is
  v = number of votes taking with multiplicity (1-1 game, 1.25-2games, 1.5-3 games, 1.75-4 games, 2-5+ games)
  m would need to be adjusted to 20
  R - still the same but taking with adjusted weight
  C - same as before but with weight taking into account.

Two things, since the rating can be updated at a later time, any update to the rating should update that rating as if it just occurred (i.e. weight changes when updated if more games played).

Second, I would propose using the weighting:
rating worth                games played
1 vote                           < 2
1.25 votes                    < 4
1.5 votes                      < 6
1.75 votes                    < 8
2 votes                         <10

The only problem that I can see is that I might play a board once and really love, leave my review elevating the ego of the designer and promoting the board, however I have only played one game so even though I play 50 subsequent games, my rating is only worth 1 vote.  To alleviate this, there may need to be, You have now played x games on y, do you still agree with your review (given below), yes or no.  This may need to be a private message.  I am not sure how big of an issue this is, but pointing it out for what it's worth.

i like that alpha

Thanks Alpha. I'm not sure about the prompting to check your review, I think that might be a bit annoying.

I like the proposed system because it gives an incentive to play more games on the board, whether or not you actually like it.  Trust the the players on this site, if informed of how the rating system works, will be more willing/likely to playe more games before leaving a review...thus promoting a more thorough review process.

 

agreed!

tom wrote:

Thanks Alpha. I'm not sure about the prompting to check your review, I think that might be a bit annoying.

if it was that funny popup that went away after you clicked it, i could live with that - assuming the check that made the indicator popup was at 10/25/50/100 at the most.

I think the normalization formula does a pretty solid job of sorting out the outliers and accounting for the number of reviews.

I do see some small value in weighting votes, though. So here is an alternative to basing it on the number of games a player has finished on a particular map:

Each account gets 10 'favorites'. A player can designate up to 10 maps as his favorite maps. Then, you can either do something like count the rating a player has left on their favorite maps twice (or three or more times!) or, just total up the number of times a map has been favorited by different players and use that number as a factor in the overall rating formula.

The "favorites" system has the added benefit of being dual use in that then players have a handy list of their 10 favorite boards to quickly find them to start games.

I think your vote should get weighted according to how many games you had played on it when you made the vote.  So if you vote after one game, you have to go back and re-vote after playing it some more to update your weighting (even if you give it the same score).  I don't think you have to bother with a prompt - the proposed weighting is not that big an effect anyway, so no big loss if most don't bother to update.  If that newsletter idea ever comes to fruition, I think a reminder to review your ratings would be good for that.

 

Cramchakle wrote:

The "favorites" system has the added benefit of being dual use in that then players have a handy list of their 10 favorite boards to quickly find them to start games.

 

Can you sort the boards you have rated by your rating?  That would also give you a sort of 'favorites' list.

There is already the option to Favorite a board which adds the board to your Favorites list.  I assume there is currently no restriction on the number of boards you can Favorite, and I would be opposed to having only a set number.

I think just having a Board Filter by "Most Favorited" would be sufficient, not sure it really has to tie into the Rating.  If it did tie into rating then I would think that would make the Rating system much more fluid and less transparent (unless Board Favorites were shown on Player Profiles, which would be cool too).

2 small suggestions -

1) Automatically include in the review the number of games a player has played on the board when they did their review.

e.g. "This review was made after completing X games on this board."

 

2) Include in a player's profile the boards that player has favorited.  Similarly could also include in the board description "Favorited by X people" with a link to a pop-up saying who those people were.

I have a feeling these weighted rankings are just going to skew the board ratings higher than they should be. The main part that is being missed is that if I like a board and rate it a 9, I'm likely to play it a lot more times and if I don't like a board and rate it a 2.

I can play a board once, figure out the perfect strategy, and decide I don't like the board. Maybe its due to design/gameplay/strategy....why should I have to play the board, a board I don't like, a dozen more times to make my rating count more.