Partial randomisation trial for Research Incentive Grants

Following the lead of The British Academy, the Carnegie Trust will be introducing a partial randomisation process for selecting applications to its Research Incentive Grants. Peer review will remain central to the assessment process in order to determine whether an application is fundable or not, but the current scoring scale and comments used by Peer Reviewers will be replaced by a series of Yes/No Questions, with a single review per application.

All applications where reviewers have answered YES to all questions will be entered in a randomised selection process. The latter will only include the unique identifier for each application and the amount requested by the applicant to ensure the selection algorithm is completely blind to personal characteristics or professional affiliation. Prof Emma Hart from Edinburgh Napier University has designed a sampling method that will automatically allocate a random number to each application to determine which applications will be funded. As is currently the case, Arts, Humanities and Social Science proposals will be kept separate from Science, Engineering and Technology and the budget allocated to each area will reflect the proportion of applications received in each. This new selection process will be trialled for 4 years. You can learn more by downloading the New guidance for reviewers.

Changes to the application form

In developing the trial, we have also redesigned the application form to help reviewers and applicants. Changes include moving all personal and contact details relating to the applicant to the end of the form, so that the reviewer’s first reading of the proposal is less influenced by who submitted it. This will not make unconscious bias disappear, nor will it stop reviewers from checking the identity of the applicant, but it should encourage reviewers to focus on the content of the proposal from the start.

The new online form consists of a series of questions in which applicants are asked to detail the What, Why and How of their proposed project. No CV or similar career history information is required and we are moving the focus away from previous grant success or publication records. Instead, applicants are asked to answer three open ended questions about the knowledge and skills acquired to date and how the proposed project will contribute to their career and to the development of their research interests and skills. We also acknowledge that many researchers have a diverse career path and experience. Therefore, applicants are encouraged to include in their narrative the knowledge and skills gained in other sectors or non-academic roles that are relevant to the proposed research.

Feedback to applicants

All unsuccessful applicants will receive feedback on their applications. Applicants whose applications are judged not to meet the quality threshold on one or more of the criteria will be informed of this when given the result at the end of the assessment process. Applicants who are entered into the random allocation process but not selected for award will be informed that their applications were fundable and the only reason for being unsuccessful was that there were not enough funds to support everyone entered into the random allocation.

Evaluation of impact of randomisation

During the four-year trial, we will monitor the outcomes of each round to gather information such as the success rate by university and research area and protected characteristics of applicants to enable us to compare the data to that collated in previous rounds. This will also allow us to see the impact of the changes on the distribution of funding across the Scottish universities and their researchers. We will also check the number of applications that meet the quality threshold against data from previous rounds of funding that relied on a scoring scale in which 60-70% of applications were deemed to meet the quality bar for funding so we can re-calibrate or adjust the assessment criteria if issues arise.

We believe that this new approach will have welcome benefits by reducing the burden on researchers looking for funding as well as on reviewers assessing proposals. The experimental approach acknowledges that there is already a chance element in evaluating proposals and making funding decisions and that subjectivity is inherent to deciding the quality of a research proposal. We hope our trial will help diminish potential bias and ensuresthat all applications that meet our quality threshold or funding criteria for support are treated equally. Reducing the turnaround time to inform applicants of funding decisions offers a further advantage.