When I joined the foundation last year, my primary responsibility was leading the 2020 Emergent Phenomena in Quantum Systems Flexible Funding competition. I didn’t have much experience running an open grant call; however, I had spent the previous ten years designing and running complex experiments to study the physical properties of quantum materials and high-energy-density physics.
I ran the grant call as if planning an experiment. The main difference was that instead of characterizing how matter behaves in extreme conditions, I intended to explore how to make our grantmaking selection processes more equitable. This goal is related to the prevailing issue of lack of diversity in the field of physics, and in particular, within the quantum materials community. Gender, racial, career stage, and other inequities persist within the field of physics, while studies show this lack of inclusion hampers innovation.
Changes in one grant call will not solve the lack of representation that stems from historical barriers to inclusion. Nonetheless, as grantmakers, we influence the fields that we are involved in through our decision processes. We have an impact through the projects we fund but also on how we choose such projects and grantees. Reflecting on our internal processes with a spirit of experimentation and a dose of humility to identify where we can minimize implicit biases, will make our impact more equitable.
Fundamental research taught me that, more often than not, scientific progress is a slow, iterative process that demands much trial-and-error, thorough data analysis, questioning our assumptions, lots of tolerance for frustration (experiments find many ways not to work out as intended!), and persistence. Many of these traits from the scientific method are present in the adaptive management approach to philanthropy at the Moore Foundation. I think they are a good framework to make steady progress when facing complex issues.
To hide or not to hide that is one question.
First, I researched and gathered information. I wanted to learn how other institutions ran open calls designed to minimize implicit bias in the grant selection process. To my surprise, I found less than a handful of examples.
One that attracted my attention for its thoroughness and abundance of data was the review process implemented for the Hubble Space Telescope time-allocation. In 2018, they decided to implement a dual-anonymous review system to address the disparity between lower success rates for proposals led by female principal investigators compared to male principal investigators.
It is standard that the identity of reviewers participating in a review process remains hidden from applicants. It is rare to keep the identity of the applicants hidden from the reviewers. In fact, it can spark intense debates to choose the latter. For some, it is relevant to know the identity and track record of the applicants to assess their capacity to carry forth the proposed project. Others think that ideas from a broader set of researchers might have a better chance to be selected by hiding information like the applicants’ identity and their institution.
A fair compromise that the Hubble Telescope implemented was to add a final, non-anonymized post-evaluation round that served to cross-check the team's expertise. The switch to the new review system significantly reduced not only the gender but other biases like career stage and institution, further enriching the talent pool.
“Wayfarer, there is no path; you make the path as you go.”
One advantage of the Hubble Telescope evaluation process is that they had several years of data records of the time-allocation, and they could analyze the effect of the dual-anonymous system. We did not have that benefit for this call for Flexible Funding. It was the first time these grants were run as an open competition and using a dual-anonymous review system.
The words above in the quote belong to a beautiful poem from Antonio Machado, a Spanish poet. They reflect quite well how one feels when exploring new ways of doing things when there are few references to lean on, and Google can't answer your questions. At that edge, one needs to rely on one’s capacity for observation and discernment, even though they are limited and biased, then take the plunge and make decisions.
The goal of the initiative’s Flexible Funding strategy is to identify research ideas with the potential to make breakthroughs in the field of quantum materials. Our team saw this grant call as a valuable and timely opportunity to explore ways to minimize implicit bias in our grant selection processes. To that end, we designed and implemented the following dual-anonymous evaluation process.
The applicants were asked to write the description of their research projects in a way that concealed their identities. The review process had three stages.
- In the first one, nineteen experts reviewed subsets of the applications relevant to their area of expertise. The applicants' identity was not shared with reviewers.
- Upon completing this review stage, program staff invited twenty applicants to submit full proposals. A new set of six reviewers participated in the second review phase. First, the reviewers submitted independent reports on the proposals assigned to them based on their areas of expert knowledge.
- After our team selected nine proposals for the final review stage, each reviewer evaluated them and submitted their final written reviews.
Only after this stage were the applicants’ identities revealed to reviewers. This ensured that the reviewers had all the relevant information about each applicant’s expertise to make fully informed recommendations. As the final step of the review, we organized a video conference with the reviewers where each proposal was extensively discussed.
Four projects were selected for funding based on the reviewers' cumulative input. Each awardee received between $770k and $1.6M over 4 years.
One more tool in our grantmaking toolkit
The dual-anonymous review process raised many questions and a lot of curiosity among the stakeholders at the foundation. We tracked and evaluated the review process by analyzing the surveys we prepared for reviewers once the evaluations were completed and the applicants' demographic data. While we do not have a previous baseline to perform a quantitative comparison, the rate of awarded female principal investigators (33%) was double than that in the initial pool of applicants (17%). Among the four institutions, one is a new grantee in the Emergent Phenomena in Quantum Systems Initiative.
Applicants proposed unique instrumentations and techniques, which raised the question of whether their identities could be revealed too early. Yet, the dual-anonymous review process was effective in concealing their identities. Reviewers were asked in the surveys to guess the applicants' identities. Among the nine finalists, the successful guess rate was above 60% for only two applications. For the remaining seven, the successful guess rate was 20%. In case you are curious to know, all awardees belonged to this last category.
Finally, the feedback we received from reviewers was very encouraging. Several remarked that the dual-anonymous review helped them focus more on the science and evaluate the feasibility on its face value, and they felt it created a level playing field. Many also pointed out the relevance of knowing the identity of the principal investigators before the final discussion as the experience and track record have a legitimate bearing in evaluating the feasibility of a proposal.
The intention of implementing this review process was to explore ways to improve equity in our grantmaking selection processes. When to apply it will depend on several factors like the nature of the grants (e.g., project-based versus people-based) and the resources that one has available, mainly time. Ultimately, I hope this example will inspire others to reflect, explore and tweak their selection processes to serve better the communities they work with and to make greater scientific progress through a fairer selection process.
Amalia Fernandez Panella, Ph.D. is the program officer for the Emergent Phenomena in Quantum Systems Initiative at the Gordon and Betty Moore Foundation.
Message sent
Thank you for sharing.