Imagine going to the grocery store with nine other friends and selecting some fruit to buy. No one picked the bananas as they were too ripe, three picked apples, four picked navel oranges, two picked another type of orange and one picked a grapefruit. You were all selecting “the best” fruit but came out of the store with different selections. Some of this might be attributable to bias (e.g., I like apples more than oranges), and in cases where the same type was selected, everyone evaluated the pile of relatively similar fruit but ultimately selected one that looked the best to them.
Now imagine fruit are proposals and you and nine colleagues are selecting the “best” one for funding. We draw on our knowledge, personal experiences, and internally recognized biases, and of course we are influenced by unconscious biases. How do we choose well? What is the magnitude of “noise” in selection? Can we recognize bias and account for it? Are there fairer or more equitable ways of selecting proposals to fund? Encouraged by Gordon and Betty Moore’s vision, these problems and more are central questions in philanthropy and driving us to learn what we can about the science of decision making.
Just as we provide resources to researchers to explore the natural world, we can and should deepen our understanding of making decisions about who or what to fund (or which fruit really is ripe). Make no mistake, these are difficult questions with fuzzy outcomes but exploring this space also teaches us what or who we might be missing or how we might make better decisions in the future.
The first was an experiment to see how gender impacted selection. In our experiment, we found that proposals submitted by women actually received higher ratings when the gender was known than when it was hidden. This doesn’t mean gender bias is gone, but it does point out that we should question our assumptions and when possible, take data to explore issues and gain a better understanding of how decisions and strategies impact our actions. We continue to use dual anonymous (double-blind) applications, but this result motivates us to question commonly held assumptions and run simple experiments when we can to gain better information. Experiments we are currently contemplating include probing the number of reviews needed to reduce scoring variation, or evaluating how certain information in a typical CV (e.g., institution where a researcher received their degree, amount of current or past funding) might impact reviewer’s scores.
Our second example was a comparison between scores given by the same reviewers on preproposals and on full proposals. We found essentially no correlation between these scores – a correlation we might have expected. The lesson we took is that noise in these scoring methods is large – perhaps much larger than the signal. Subsequently, we made a decision to invite applicants from a broader range of preproposal scores to submit full proposals.
If you are not convinced that noise dominates these decisions, we invite you to read the book by Daniel Kahneman, Olivier Sibony and Cass Sunstein, “Noise: A Flaw in Human Judgement” to get a taste of how an individual’s life experiences impact their decision making. For our part, we have decided to make decisions partly by random selection. We make an initial determination that separates proposals within submissions to the initiative into three categories: fund, maybe fund, and do not fund. We select all proposals in the first category, and then randomly select our remaining awards from the second category. This selection by partial randomization minimizes the effect of noise in the selection process and partially ameliorates unwanted bias.
In addition to this, we are also gathering data from funded and unfunded applicants to better understand funding impacts.
We believe as funders, we have a responsibility to assess our actions and to learn from what we are doing (decisions) and how we are doing it (design).
The Moore Foundation embraces this philosophy and it was stated clearly by Gordon Moore in the Statement of Founder’s Intent, “…science and the type of rigorous inquiry that guides science are keys to achieving the outcomes we want. Scientific methodology should be a cornerstone of nearly all of the Foundation’s efforts.” It is this principle that both keeps us up at night wondering if we are choosing the right “fruit” and propels us forward to ask difficult questions about our selections. We will let you know what else we learn – stay tuned!
Theodore Hodapp, Ph.D. is the program director for the Experimental Physics Investigators Initiative at the Gordon and Betty Moore Foundation.
Message sent
Thank you for sharing.