Committees prefer a clear story: http://labinitio.com

The Journal Nature reported yesterday that scientists have complained that there is a mismatch between expert referees’ evaluation scores of research grant applications and funding decisions.  Different interviewees claimed that this mismatch either does or does not indicate either a flaw in the system or mistakes by referees or by committees.

There may be flaws in the system and mistakes probably happen, but there is a more obvious reason that referees’ scores should not be expected to predict funding decisions. Referees and committees do different jobs which impose different constraints on the way a grant application is written. Very few proposals are written in a way that satisfies both sets of constraints, and so, for the majority of proposals, there is no reason to expect a close match between the referees’ score and the committee’s score. Before I explain the constraints and how to meet them, I’ll clarify the story and explain its relevance.

The story was based on 302 grant applications to the Medical Research Council. It states that some applications that received high scores from the referees were triaged (rejected without being discussed by the committee). Of the proposals discussed by the committee, the story states that the group of applications that were successful and the group that were rejected had ‘a nearly identical spread of scores’. It’s worth noting that the story focuses on a statistic of the scores that is not particularly informative (spread) and does not mention any other statistics. It is relatively common that the referees include both friends and foes of the applicant, which can cause the spread of scores on a single application to range from the lowest to the highest possible. Consequently, nothing that the story says about this rather small data sample indicates a general failure of referees’ scores to predict committee decisions.

However, whatever we may try to conclude from this small dataset, most funding agencies (the EPSRC is an exception in the UK) ask referees and committees to do very different jobs. These jobs depend on different aspects of the way the proposal is written.

Referees work alone and each one works on a single application. The referees are expected to be experts in the same research field as the applicant and their evaluations are typically seen as coming from within that field. Their main task is to test the detailed rationale of the proposal. Are the research questions important? Is the research approach feasible? Does the research team have the ability to carry out the project? What is their standing in the field? Has their previous work made a significant contribution? Is the approach novel? Are other teams likely to get the answers first?

To get a good score from a referee, a grant application must contain relevant detail. Evidence that the questions are important and relevant must be cited. Any evidence to the contrary must be dealt with effectively. The research approach must be described clearly and in sufficient detail to convince a knowledgeable sceptic that the project is feasible, can be carried out with the resources requested, and will lead to the promised outcomes.

The committee works as a collective and takes a view across all the applications before them, typically about 100 for a single meeting. They must also take a view across all the different research fields that the committee supports. They need it to be clear that the project will deliver an  outcome that will have importance beyond the immediate research area and that the applicant has identified an approach that is likely to be productive and that the research team has the skills to deliver it. They need to know what the research aims are and how they relate to the overall outcome. They need it to be clear that the research objectives will satisfy the research aims and deliver the overall outcome and that the results will have appropriate impact

More importantly, the committee also works very quickly. They have to reach an agreed view about grant applications which most or all of them may not understand completely. For the committee to score an application highly they need it to be possible to understand it on the basis of a single hasty reading – or even a quick skim. The case for support should have a very clear structure that states clearly what will be the research outcome, why it is important, what are the research aims, what are the objectives and what will be done with the results.

Of course the best possible case for support has a clear structure that enables the committee to understand it and appreciate its strong points. It also uses the structure to guide the referees to the relevant detail. In my experience cases of support written in this way get very high scores both from referees and from committees. Unfortunately they are very rare but this blog explains in several different ways how to write them.