Well, someone had to start this thread so we wouldn’t go on hi-jacking others.  I’m  thinking that we can write/edit a document as we go.  Here is a quick off the cuff list of considerations:

 Overview: The general function of the board.

This should include information that directs what the board does and does not do.

Passing criteria:

Adequate board description

Clear borders

Border functionality

Reasonable/appropriate playability, rules, dice mods, bonus structures, etc.

Clear rules regarding legality/obscenity.

Overlapping themes with existing maps.

Image Quality standards.

Guidelines/Expectations regarding the role of reviewers in supporting the designer of a board that is “not quite ready for prime-time”

Perhaps stating a clear distinction between “recommended” and “required” changes to the board should be the norm.

Example: fix border from T4 to T7 (required). 

Example: Font used in the key could be larger (recommended).

Should reviewers shower designers with non-required recommendations that would improve game-play, graphics. (this could potentially get out of hand, though in my opinion sometimes there's not enough of it.

Process:

Which reviewer passes the board?  Why shouldn’t there be a requirement for more than one reviewer to pass the board? (which could take multiple games in the case of a dueling game)

Does a game need to be finished in order for it to pass?

Rating criteria:

Beginning, Intermediate, Hard?

Are these determined by vote?  ..Consensus?  Strictly quantitative criteria?

Very nice start.

I think the passing criteria, could be wrapped up as guidelines to the board creator, divided into 'must' and 'should'.  i.e. all of your passing criteria are 'must' guidelines.  A 'should' guideline would be something like creating custom cards.  (basically, see RFC 2119 (http://www.ietf.org/rfc/rfc2119.txt)).

So Guidelines:

• Must have a board description that explains the theme and/or advanced features.
  
• Must have clear borders (non-obvious borders should be indicated on a  legend or in the board description)
  
• Must have border functionality  (what does this mean?)
  
• Must have reasonable/appropriate playability, rules, dice mods, bonus structures, etc.  (can we define this better?)
  
• Must not be obscene.  (I'll know it when I see it)
• Must not be illegal.
• Must not be nearly identical to an existing map. 
  
• Must meet Image Quality standards.  (no "anti-aliased" fills)

• Should have custom cards.

Some things like the Image Quality standards, could be expanded as shoulds - Should have borders that are not the same color as a player color, Should have adequate sized territories so that territory ownership & unit count is clear, etc.

M57 wrote:

Which reviewer passes the board?  Why shouldn’t there be a requirement for more than one reviewer to pass the board? (which could take multiple games in the case of a dueling game)

 

+1 for needing 3 people to pass a board

Great questions and great start!  I'll chip some thoughts in later too!

Existing replies from the Review Board thread: http://www.wargear.net/forum/showthread/1601/Review_Board

Yertle wrote:

M57 wrote: 

This has been discussed in another thread, but we really should have a Review Board Guidelines page so that reviewers AND designers are all on the same page.

but any guidelines specified on this page are implied, not specified.

As it stands right now, the board review process has an ad hoc feel,...

It shouldn't be that hard.

I really wish it was more set too and would love a more defined process.

I do want to see user developed boards and I think that's a huge part of the fun of WarGear, but to prevent what happened on WF with just a massive board list would be sooooo nice.  Perhaps the great Boards tab that is here helps negate the problem that WF had, but in a sense fewer can be better too IMO.

One other thing that I'd like to see is a quicker Board Review process.  I don't see the need for a board to be in a Submitted status for more than a month, which probably means the Review game wouldn't necessarily be completed prior to Pass/Fail (Fail could continue longer if the designer is working on the board).

There's a board in Review right now that the game is completed, but IMO it doesn't pass the Review (as defined by the Help), but to say it doesn't pass when some other boards doesn't feel right either. 

I agree that it all feels very wishy-washy.  I don't have the answers, and don't even know if I really have a super strong opinion as to go more lenient or more restrictive (I would probably say quality over quantity though), but would be nice to be more defined.

Threads from the past:

http://www.wargear.net/forum/showthread/229p1/Board_Review_Guidelines Where the Help page came from.

http://www.wargear.net/forum/showthread/112p1/New_review_system_now_live I'm not really sure much about what passes and what fails is in the thread, more so just about how the review process should be handled...I think.

http://www.wargear.net/forum/showthread/645p1/Review_criteria_-_time_to_review Attempt to redefine the Help.

http://www.wargear.net/forum/showthread/949p1/Board_Review_Graphics How much do graphics play a role.

 

*edit* This may be considered hijacking the thread, so if this is better with another new thread :) then we can go there.

Alpha wrote:

Yertle wrote:

I agree that it all feels very wishy-washy.  I don't have the answers, and don't even know if I really have a super strong opinion as to go more lenient or more restrictive (I would probably say quality over quantity though), but would be nice to be more defined.

I personally vote for more restrictive.  I think that in the interest of both the designers and the players, a more restrictive review process is better.  (I am really stating that quality over quantity is what matters).

There are many boards which have been released prematurely in my opinion.  These are boards which I think where based on a great idea and could have been boards which rivaled the top in terms of game-play, but were released and now there is no reason to continue working on them.

The boards I have currently I can discuss, but will not critique other designer's boards.

Five - This board was a fantastic idea and played very well.  At the time it was initially released Mongrel and I felt that it was winnable by black with perfect play.  (This turned out to be correct, although it took around 200 games before the conditional perfect play was found.)  Since then, we released a "better" version which is much more even between black and white.  The first release led to some fantastic times on Wargear as many players tried to figure out how to win as black, but also led to many sore players and bad reviews (both official and unofficial).

Seven - This board was thoroughly play tested prior to being released and the majority of the test games turned out fantastic (this was with a very some sample of players who were all in love with the idea).  After this board was released, it became apparent that although many games were good and some were great, there were a few that were bad (irrecoverably one-sided from the start).  The scenarios which have been released since then play out much more evenly.

Prion4 - This is the first board I ever developed (back in 2003), but I worked on finalizing the graphics, borders, and bonuses for months in 2010.  At some point, I decided that I had enough and released it.  It passed review, but had a terrible end-game which could (and most of the time did) stalemate when two or three players were left and would not end.  The card-scale fixes have reasonably fixed this problem.

Day at the Races - Again this was a thoroughly play tested board prior to being released and has a sometimes stalemate type mid-game, but played pretty much how I expected.  The default version of this board plays well, but the random scenario is often lopsided and could be changed (but I am not in favor of major changes this long after release).

From the design and development of the boards above (and many I have left in design phase), I feel that scenarios should be reviewed the same as any board and I feel that three reviewers should have to sign-off on a board before the board goes live.  In reality, I suggest this not because I think that fewer boards should be released, but because I think more players (the review team) should give their input into the boards that are released on the site prior to the release.  This extended process will be beneficial for the site, so players in general (boards are not prematurely released) and for the designers (their designs should ultimately improve with more input).

As for the last part, I believe the input should be just that, suggestions, not mandates.  There should be open discussions between the designers and the reviewers during these review games.  Recently, there seems to be a laissez-faire attitude in review games where not much is discussed and the games are briefly okay-ed and released.  I would like to see more happening these final development games or more happening prior to review.  The increased discussion should in my mind lead to better boards and better designers.

I have thoroughly hijacked this thread if Yertle did not, but I feel that those who are willing to join the review process should also be willing to discuss it.  I became part of the review process because I liked the site as much as I do and do not want to see it change from the great site it is.

Ozyman wrote:

I pretty much agree with Alpha, now that we have decent # of boards on the site, it would be better to shift the focus to quality over quantity.  At the same time, we don't want to discourage map makers too much (especially beginning ones).  Figuring out how to fix a board with stalemate problems, or how to tweak it so all seats have an even chance, can require many many games.  I don't know what the answer is here, maybe some sort of in-between stage between beta & a full release.  Or maybe if we had beta tournaments... I'm not sure.

I've mentioned this before, but I really think some sort of statistic available to see how many games have been voted to terminate would be useful.  I think the biggest game-play problem boards face is a stale-mate (usually via crab-game).  I think a fairly strict rule could even be helpful - something like if > 20% of all games are terminated on a scenario, that scenario is pulled from active play until the designer & council review it.  The reason I set this % fairly low is that I think in many games instead of voting to end, players get frustrated and either suicide, or quit playing and get booted.

If that is too strict, even including this percentage for view would allow players to avoid boards with a high % or allow designers to go back and review their boards that have a high %.

font in the first two posts are white for me (IE7, work mode) - i have to highlight them to be able to read them.

I started to compose a pre-amble of sorts for the page.  Let the editing begin..

This page sets guidelines for members of the Review Board.  It is highly recommended reading for both prospective board members and board designers alike.

When a designer submits a board for review, all board members are automatically invited to a "Review" game for that board.  After the requisite number of players has joined (a number set by the designer that offers optimal play), a normal game then begins that includes the designer as a player.  As the game is being played, board members are expected to make sure that all criteria are met before passing the board.

WarGear board members are encouraged to be both critical of the board and supportive of the designer. For instance, while it is expected that submitted boards have been previously play-tested by the designer, it is not uncommon for reviewers to find a few technical errors that can be relatively easily fixed by the designer in the course of play.  There is however, a line that can be drawn by the board and they can "fail" the board, requesting that the designer resubmit the board after fixing the problems.

It is also common (and encouraged) for reviewers to point out non-critical flaws in a board.  In many cases, reviewers make suggestions that are meant to improve game-play, but are not essential for the board's passing.  Sometimes these suggestions are taken to heart by the designer, who may even chose to end the process to make significant changes to the board before resubmitting.

With this in mind there is a conventional nomenclature that designers are expected to use as follows. When a critical problem is pointed out by board members (one that must be fixed for the board to pass),  by convention they use the word "please" when requesting the fix.

Example:  "Please fix border from T15 to T17."   This is a required fix. 

Corollary: The lack of the word "please" implies that the request is a non-binding suggestion.

Example: "That player-color of gray looks an awful lot like the neutral color gray to my aged eyes." This is a non-binding suggestion.  Follow it with "Please Fix" and it becomes required.

Example: "I recommend that you change the dice mods attacking from the castle to +2" This is also non-binding.

Example: "You should make all borders 2 pixels wide" This is still only a recommendation.

M57 wrote:

 ...by convention they use the word "please" when requesting the fix.

Example:  "Please fix border from T15 to T17."   This is a required fix. 

Corollary: The lack of the word "please" implies that the request is a non-binding suggestion.

Example: "That player-color of gray looks an awful lot like the neutral color gray to my aged eyes." This is a non-binding suggestion.  Follow it with "Please Fix" and it becomes required.

Example: "I recommend that you change the dice mods attacking from the castle to +2" This is also non-binding.

Example: "You should make all borders 2 pixels wide" This is still only a recommendation.

Come On! No one's gonna pan my somewhat over-the-top rebuttal to Ozyman's link? ..or is it actually a good idea?  ..or do we want to go with must/required, suggested/recommended? ..or none of the above?

haven't read the thread yet. will try to today.

haha.  I thought your post was a bit weird, but meh.  I don't care too much.  I prefer the precise terms defined by the RFC, but that's because that is the vocabulary I am familiar with.  As long as we have some way to define between must, should & may, I don't care what words we use.

All of these things sound fine to me.  I think the biggest thing that detracts some people from building boards is the review process itself.  It doesn't seemed to be defined.  I have a buddy who has tried to submit a board, but it was rejected and he wasn't sure why.  As long as the builders of the boards know what areas they need to work on the review process should be smooth.  Also I think that a full game should be played before a board gets approved, but that's more of a personal thing.

Beastlymaster wrote:

All of these things sound fine to me.  I think the biggest thing that detracts some people from building boards is the review process itself.  It doesn't seemed to be defined.  I have a buddy who has tried to submit a board, but it was rejected and he wasn't sure why.  As long as the builders of the boards know what areas they need to work on the review process should be smooth.  Also I think that a full game should be played before a board gets approved, but that's more of a personal thing.

Perhaps a rejection should come with a written description of the reason for the rejection, preferably with suggestions that would help the designer to get it to pass on the next go round.

For instance, next to the rejection button is a field to be filled out by the rejecting board member - that gets sent to the designers mail-box.

Beastlymaster wrote:

All of these things sound fine to me.  I think the biggest thing that detracts some people from building boards is the review process itself.  It doesn't seemed to be defined.  I have a buddy who has tried to submit a board, but it was rejected and he wasn't sure why.  As long as the builders of the boards know what areas they need to work on the review process should be smooth.  Also I think that a full game should be played before a board gets approved, but that's more of a personal thing.

Was it actually rejected, or did everyone assume everyone else was going to be the one to pass it and no one did - 

I like the idea of three needed for a pass and three needed for a fail. 

Each fail comes with a reason for failing.  This can be easy like: inappropriate _____ (this is a family site), images don't work (fill is not solid, board is solid and territories cannot be seen), etc.

Both side of this issue have problems.  Serious issues only require one fail, minor issues may not warrant a fail, but could discourage a designer from trying again (some boards come to mind).

I think what is really needed is a mandate that the review board is trying to fulfill.  Are we trying to keep bad boards from being on the site?  If so, what exactly is a bad board?  If not, then what are doing?

The answers to these questions will really drive the need for a more lax review process or a more stringent one.  I have decided (today) that I really don't care which way it is, but I would like to know what the site feels the responsibly of the review team is and what it is not.  This way I will know what I should be doing.

Currently, I feel that each reviewer has their own idea of their responsibility and just as in life some reviewers/judges are harsher than others.  In some ways this is good and in some ways not.

Alpha wrote:

Currently, I feel that each reviewer has their own idea of their responsibility and just as in life some reviewers/judges are harsher than others.  In some ways this is good and in some ways not.

This is why, no matter whichever way we go, the guidelines should stress the importance of being supportive of the designer.  So far (and I've been on both ends of the process) my experience has been that this has been the case.  Members that have reviewed my boards have been very helpful and patient, but nevertheless this should be explicitly stated in the guidelines

WarGear board members are encouraged to be both critical of the board and supportive of the designer.

The potential problem of some reviewers having stricter standards than others is somewhat mitigated when there's a 3 min rule, but designers are still at the mercy of the draw.  I.e., what if a bunch of tough reviewers joins the review game?

I was thinking of issues like overlapping themes and borderline image quality, etc. and how unfairly subjective these types of decisions are to make for any sub-group of board members , and I thought I'd throw a radical idea out there. It involves a pseudo-democratic mechanism for the review process..

Pass or Fail Criteria: A four (or possibly three) vote margin for 24 hours.

Eligible voters:  All board members AND the designer, regardless of whether or not they are playing in the review game (this is, of course, contingent on their being able to watch the game and post to its chat).

Process: At any time during the review game, eligible voters are allowed to cast their yea or nay vote. One of those votes must be from the designer. Votes are revokable.

Advantages:

• As before, all reviewers don't have to participate, but no reviewers are shut out of the process.
• The decision is made by a reasonable majority of "concerned" reviewers.  Also, the designer can float more than one review game (also in the case of a board with multiple scenarios).
• When the board supports very few playing reviewers and/or the designer only floats one game, e.g., a dueling (two-player board), all can participate in the dialog and decision.
• Games don't necessarily have to be completed to pass (though we may by convention prefer games to be completed).  And there are built-in safeguards - For instance, if a reviewer ill-advisedly and/or prematurely passes a board, another reviewer can cast an equalizing fail vote until the game is completed.
• The 24 hour time period coupled with the 3-vote margin criteria enables quick passage of boards that are known to have been thoroughly play-tested, but prevents their premature passage.

I actually really like this idea.  I think the 24 hour period might be overkill.  As long as you trust that the reviewers are all reasonable people, then if you get 4 fail votes in a row (or 4 pass votes in a row), that is probably a pretty good indication of the quality of the board. 

This could be done with little extra work by Tom, by creating a private forum that is used to hold the votes and can only be seen by the review board.  One thread per board submission.  Only problem with this, is board designers can't post their to vote, but I think we can just assume it is a vote to pass.

I also think anyone who votes should have to write something to justify their vote.  Could be just a few words, but they have to say something.

Feeling very republican: I think all this front end review standards stuff will just increase animosity and further politicize the process.  Furthermore, there is no amount of testing and reviewing that can guarantee flawless board design. So my proposition is at the release side, with some sort of retirement system.

Simply:

1.) Review team checks borders, ensures no T-n-A goes live (on this site anyway), and that the map is up to graphical standards (e.g., no T-n-A). Suggestions can be made by the review team, and implemented by designer. Objections from a majority is the only way to fail a board. So new designers won't be as discouraged to submit, feeling at the mercy of the review team.

2.) Once a board is released it gets 30 (or 60, or 90) days for a trial period. Non terminated, non designer started games in tournaments, open tables, and private games all count towards a board's "staying live credits". Could be a simple formula: Let x be the number of non terminated, non designer started games in tournaments, open tables, and private games. Then the number of days past the release date a board stays live is

30+x

3.) Once a board is retired, nothing really changes, it just goes on a separate page. Games can be opened, championship points can change hands, and so on. The land of misfit boards. A board can jump from retirement if 10 non terminated, non designer started games are opened in as many days, at which point it goes live for

10+y

where y is the the number of non terminated, non designer started games past the resurrection.

4.) If a board reaches 500 (or 750 or 1000 or...) plays, it goes into a hall of fame and can never be retired.

Essentially, this brings the whole site into the review process, allows for enough plays to spot/report any glaring design flaws, and supplies the uniform "tough love" we wish to instill in the review team, without, you know, pages of discussion and rules that few will follow.

This will actually allow more crap to pass, but puts a diaper in place to catch and sift said crap.

Mongrel, I think your market-driven ideas re: retiring boards are interesting, but I have concerns regarding a few issues.

Scaling: Consider that participation in the site may grow or shrink.  Hard number ceilings and floors become very arbitrary.

With designer started games: if players join there is interest.   What does it matter if the designer started them?  In fact, under your rules a designer might become torn about starting games on their own map because the people who join those games are potentially the people who would/might otherwise originate them.

Re: my idea.  I don't foresee the process getting any more political than it already is('nt).  In fact, I think my proposed system is designed to make the process less political.  The problem you described can happen with both your system and the existing system with a subset of reviewers making the decision.  Consider that specific reviewers (who have particular leanings, e.g., prurient or modest ) could make sure that they review certain types of boards, etc.

Ozyman, The reason I was thinking the designer should have a vote (required for passage) is that sometimes in the course of the review process a reviewer makes a really good non-binding suggestion.  In these cases, it is not uncommon for the designer to request putting off release of the board while (s)he implements the idea.  In fact, there are times when the designer may want to consider pulling the board from the process as these ideas present themselves.

I'm for formalizing what's in place, and the list of suggestions in this thread look fine, but chances are low that in a few games one can detect all possible gambits and player workarounds to cheat a board. So the "playability" category, beyond basics, will be trickiest to quantify.