A Level Playing Field

The Water Consciousness Challenge is designed to ensure that all applications are evaluated fairly and judged by the same set of standards. Below is a detailed explanation of how each complete application will be scored to create a level playing field that reflects a spirit of transparency and equal opportunity. For additional information, please see the scoring process and meet the judges.

The Scoring Algorithm

Once you have completed your application, a minimum of five judges will be assigned to score your proposed project. Judges will offer both scores and comments for each of four distinct traits. Each of the four traits will be scored on a 0-5 point scale, in increments of 0.1. Those scores will combine to produce your total score. Examples of possible scores for a trait are… 0.4, 3.7, 5.0, etc.

The most straightforward way to ensure that everyone is treated by the same set of standards would be to have the same judges score every application; unfortunately, due to the number of applications submitted, that is not possible.

Since the same judges will not score every application, the question of fairness needs to be carefully explained. One judge scoring an application may take a more critical view, giving every team a range of scores only between 1.0 and 2.0, as an example; meanwhile, another judge may be more generous and give every team a range of scores between 4.0 and 5.0.

For illustrative purposes, let’s look at the scores from two hypothetical judges:

The first judge is far more generous in scoring than the second judge, who gives much lower scores. If your application was rated by the first judge, it would earn a much higher total score than if it was assigned to the second judge.

We have a way to address this issue. We ensure that no matter which judges are assigned to you, your application will be treated fairly. To do this, we utilize a mathematical technique relying on two measures of distribution, the mean and the standard deviation.

The mean takes all the scores assigned by a judge, adds them up, and divides them by the number of scores assigned, giving an average score.

Formally, we denote the mean like this: 

The standard deviation measures the “spread” of a judge’s scores. As an example, imagine that two judges both give the same mean (average) score, but one gives many zeros and fives, while the other gives more ones and fours. It wouldn't be fair to you if we didn’t consider this difference.

Formally, we denote the standard deviation like this: 

To ensure that the judging process is fair, we rescale all the scores to match the judging population. In order to do this, we measure the mean and the standard deviation of all scores across all judges. Then, we change the mean score and the standard deviation of each judge to match.

We rescale the standard deviation like this: 

Then, we rescale mean like this:

We are pleased to answer any questions you have about the scoring process. Please feel free to ask any questions on the discussion board. Please register today to begin developing your application.

We are pleased to answer any questions you have about the scoring process. Please feel free to ask any questions on the discussion board. Please register today to begin developing your application.

An initiative of the Arizona Community Foundation, Republic Media, and Morrison Institute for Public Policy to create the Arizona of tomorrow. Made possible through the support of the Tashman Fund and the Lodestar Foundation.

mangakakalot