I just became aware of the establishment by the European Commission of a Review Panel on the European Research Council’s structures and mechanisms. I learned that the Panel is consulting with representative research organisations in Europe and has asked for inputs, with a deadline of 24 April 2009.
Since I missed this deadline, I shall summarise my thoughts in this letter. One of the tasks of the Panel is listed as “analyse the ERC’s structures and mechanisms against criteria of scientific excellence, autonomy, efficiency and transparency”.
I would like to bring to your attention my personal experience with the workings of the ERC, based on a grant application made in the spring of 2008. It is my belief that the facts to be described below represent food for thought on the subjects to be considered by the panel in their analysis.
I applied for an Advanced Grant in Life Sciences and Medicine in the 2008 round. My application was not successful. This document is meant to describe to you a number of serious deficiencies in the process of reviewing advanced grant applications by the ERC and the particular application submitted by me should serve only as an example.
B. What I found unsatisfactory in the reviewing and awarding of grants in the “ERC Advanced Grant 2008 system”:
1. Transparency. An essential component of any evaluation must be transparency. The heart of this is full disclosure of how the scientific evaluation is performed. The applicant receiving the Step 1 Evaluation Report cannot figure out whether the evaluation of the applicant and of the research proposal were based on the opinion of the panel members or on that of external reviewers, or both.
Just as an example, in the correspondence received by me, there was contradictory information on the identity of the reviewers responsible for the comments. Thus, in the Step 1 Evaluation Report, it was stated that “the panel bases its appraisal on prior individual reviews conducted by ERC panel members”. In a letter to me it was stated that “the panel has based its judgment on remote review followed by a panel discussion and decision”.
A further attempt to clarify this issue led to the following statement by the ERC Secretariat: “If your proposal was rejected at Step 1, in that case, all reviewers are panel members and identified directly by the Scientific Council […] remote referees only come in during the 2nd step.”
It seems that the earlier statement was incorrect and that no use was made in the evaluation of the proposal of external expert reviewers.
My application was discussed in panel LS1 (Molecular and Structural Biology and Biochemistry). The likelihood that members of such a vaguely defined panel can express an expert opinion on the scientific leadership profile of a very diverse group of investigators and on their equally diverse projects is very low. This would be like asking a panel defined as “General Literature” to judge a researcher working on the posthumous reconstruction of the unpublished parts of Robert Musil’s “The man without qualities”.
Please compare this to the NIH system of study sections. These are structured on well-defined, rather narrow areas of specialisation, with a high likelihood to be able to competently review an application. In spite of this, the NIH reviewing system is under constant scrutiny and is subject to criticism (please see paper by F. C. Fang and A. Casadevall, entitled “NIH Peer Review Reform – Change We Need, or Lipstick on a Pig?”, Infection and Immunity 77, 929-931, 2009).
Since Dr. Elias Zerhouni is a member of the Panel, he can confirm the exactitude of these statements.
2. The quality of reviewing. One of the most disturbing aspects of the review procedure, as applied by the ERC, is the quality and quantity of the information transmitted to the applicants. The minimal expectation would be to obtain from the panel a detailed and documented critique of the applicant’s status and of the application itself.
Taking my particular case as an example, the Step 1 Evaluation Report was limited to the equivalent of hardly one page, containing the opinions of five panel members. I am reviewing grant applications and I have received feedback to my own applications for about four decades. I do not recall having ever encountered such poor reviewing. This was expressed in a few sentences, many in careless and incorrect English.
How can one trust the ability of reviewers, who are the authors of these superficial, simplistic and carelessly worded “judgments”, to be competent to decide whether the application is worthy to be retained for the second step?
With all due respect to the panel members, I would not dare to submit a report like that on a M.Sc. thesis, not to mention a Ph.D. thesis. To think that the award of millions of euros of taxpayers’ money is decided based on the opinion of people who have such a poor grasp of the English grammar is more than worrying. I do not believe that such low quality prose would leave the desk of a member of an NIH study section.
3. The new Holy Grails: “Ground breaking” and “interdisciplinary.” It is not fair for me to judge the quality of reviewing by panels other than LS1, but I suspect that the problems that came up with my application are unlikely to be unique. I think that some of the problems are not epiphenomena but are linked to the unrealistic use of the term “groundbreaking”.
All those who are active in day to day “bench top science” know that whether a project is “groundbreaking” can only be known after its completion and, frequently, only many years later (see prions, helicobacter, or X-rays).
The same applies to the requirement for “independent creative thinking” and “capacity to go beyond the state of the art”. This is trendy PR-oriented language, which should not be part of the process of judging grant applications. What discovery was predicted to be groundbreaking before it was actually made?
An instructive example of how difficult it is to predict “groundbreaking” research is the following. In 1981, I was a visiting scientist at the NIH. Around that time, a voluminous book was published by the NIH, entitled “New Initiatives in Immunology”, written by a special Study Group, which included the cream of American Immunology, at that time. Suffice it to say that only a few pages were dedicated to the whole subject of “cytokines”, a field that occupies, at present, a major part of Immunology.
This is a perfect example of our inability to predict what will be important in the future. This fascination with fashion-conscious verbosity is also at the origin of the fact that “solid”, “focused”, and “classical approach” are looked upon as “dirty words”, as if the only ways to discovery are lack of focus, frivolous ideas, unconventional approaches, and “épatez les bourgeois” proposals.
4. What does “scientific leadership” mean? The definition of “scientific leadership”, appearing both in the application form and in the explanation of the panel marks in the Step 1 Evaluation Report, is also questionable. Even the term “leadership” is borrowed from the arena of politics and PR.
What should count is one thing: did the published work of the applicant have a major impact on the field. In other words, would the respective area look different if Dr. X would not have been born? The level of past funding, the number of students, collaborations, prizes, and membership in organising committees, are questionable indicators of the accomplishments of a scientist.
Some of these criteria have little to do with scientific research per se and are merely epiphenomena which grew out of the dominance of “mass science”. There is a clear indication that the ERC was carried away by the present trend of rewarding “scientific contractors” and “entrepreneurs”, as opposed to the individual scientists working in modestly sized groups on a key subject in which they are the recognised experts.
The “objective” bibliometric methods also give a clear advantage to the “contractors” since indexes and citations per paper do not differentiate between the real contributors and the “honorary authors” and offer a clear advantage to scientists pursuing networking intensively. A perusal of the successful Advanced Grants for 2008 shows that the ERC predominantly supported the already powerful and well-funded groups led by the science “entrepreneurs”.
5. The amount of funds: excessive and inequality-generating. This refers to the excessively generous support provided by the ERC Advanced Grant Programmes. The following aspects were, I think, not given sufficient consideration:
– There is an enormous difference between the legitimate financial needs of researchers in the experimental sciences (both physical/chemical and biological) and those in the humanities and social sciences. Normally, experimental sciences require much more funding. However, because nature abhors vacuum, money once given, even if excessive, will be absorbed to the last penny, whether needed or not.
– ERC funding also appears not to take into consideration the varying requirements in different countries, such as the United Kingdom, Israel or Latvia, just as an example. Recipients of an ERC Advanced Grant, in a country like Israel, become so much “wealthier” than scientists of equal or, sometimes, superior stature, who were merely the recipients of a prestigious Israel Science Foundation grant, as to cause the science equivalent of “social injustice” (the matter from which revolutions are born).
– What was meant to be a meritocracy became something resembling an aristocracy and even if one admits that ERC Advanced Grant recipients are an elite group, their supposed “superiority” does not justify the outrageous differences in funding. More injustice is generated when these large sums are given to scientists who lead groups by virtue of an administrative appointment. There are, however, more subtle cases of the ERC failing to make a clear distinction between senior scientists and their groups of students and post-docs and the science ‘bosses’ ruling over groups of fifty or more investigators.
– The sudden availability of millions of euros is inviting uninhibited and frequently unjustifiable spending. The worse infringements are expected to occur with the interdisciplinary grants where the Principal Investigator sees him/herself principally as a manager. It is likely that a considerable part of the money will be spent on “administrative expenses”, a cover for everything from salaries for secretaries to the furnishing and other outfitting of offices.
6. The magic of the “interdisciplinary”. The automatic equalling of “interdisciplinary” with quality has given unwarranted encouragement to the “big operators”, whose main attributes are the ability to network successfully and find the “right person for the job”.
Among the recipients of a large “interdisciplinary” ERC grant of 2008 are persons who lack expertise and, frequently, elementary knowledge, in most if not all the fields that they are expected to coordinate. By this policy, the ERC is guilty of cultivating and rewarding a prototype of “investigator”, who might be the right person to direct the marketing section of “Roche” but not to discover oncogenes.
7. The absolute requirement for “external review by experts”. External review by experts should not be an option but a firm requirement. External review is the standard with most of the granting bodies known to me and the alternative is the “study sections” system adopted by the NIH (also supplemented, when needed, by external reviewers).
As I am sure at least some Panel members are aware, serious problems arose at the NIH, too, due to the limited competence of some of the study section members, who lacked sufficient research experience. Except for the unlikely situation that a majority of panel members are also experts in the area of the submitted application, no application should go through Step 1 without being reviewed by external experts. This review should cover both the applicant and the research project. Many granting bodies are asking applicants to suggest and exclude reviewers.
8. Reform the Redress procedure. The Redress procedure should address not only the formal and legalistic aspects of the application and its review but foremost the problems with the review. The response to my request for Redress was worded in typical legalistic terminology but gave no indication for (and a lot of indicators against) my arguments having been discussed by persons knowledgeable in my field. It also contained the statement, admitted by the ERC officials as being incorrect, that my “proposal was reviewed by independent experts who were relevant specialists in the field”.
As already exposed at point B2, the scientific level of reviewing was sub-standard and revealed a total lack of familiarity with the field. This requires both a top-to-bottom reform of the ERC grant reviewing system and the establishment of a real Redress procedure, which is more than lip service. These requirements are made even more acute by the fact that unsuccessful applicants have to wait for two years before reapplying.
C. Conclusions and suggestions.
My suggestions are:
1. A thorough restructuring of the scientific panels is required. Clearly, panels such as the LS1, are incapable of covering, with the required expertise, the staggering variety of subjects in the applications to be reviewed. Thus, the routine use of expert external reviewers must become a sine qua non condition for judging the applications. After all, this is the method used for judging all papers submitted to peer-reviewed journals and is the standard procedure with most grant-awarding organisations.
2. An equally thorough revision of the concept of “scientific leadership” is urgently required. I do not think that what the ERC had in mind was to make the already rich richer. I have been in life sciences for 44 years and you can guess that I have a certain awareness of who is who in both mine and related fields.
For ethical reasons, I cannot provide you with actual examples, but I do not think that funds should be provided to directors of large research institutes, many, if not most of which, are financed by governmental funds. This would be similar to providing extramural NIH support to Directors of NIH institutes on the Bethesda campus. The same applies to other awardees in the 2008 round of the Advanced Grants. Many of them are administrative (admittedly, also scientific) heads of large, well financed groups which have little need for the ERC funds.
I would like to draw your attention to the fact that this very same issue came up recently in the criticism of the NIH grant system, when it became apparent that, in 2007, 19 investigators captured 165 grants, totaling US $ 160 million (F. C. Fang and A. Casadevall, Infection and Immunity 77, 929-931, 2009).
Part of this is due to using the extent of past funding as a criterion for excellence. As everybody who has been in science for some time knows (but does not always dare to say), grant writing, just as omnipresence on the congress circuit, and the participation in congress organising committees, became a profession of the science “bosses”, to allow them to employ sufficient labourers (frequently foreign), to get ready for the next round of grant submission.
One should separate entrepreneurship from real science. The ERC system, as applied these days, promotes the first and does a disservice to the second. A lot of superb science is done by researchers not in the limelight. These are the people who not only require but also greatly deserve at least part of the support to be provided by the ERC.
3. As discussed at point B5, the ERC Advanced Grants are too large. The 19 NIH-supported science mandarins, mentioned above, have captured an average of 6.3 million euros per head and this is not too far from some of the large ERC grants. In Israel, the Israel Science Foundation (the flagship of Israeli grants) will give a maximum of 75,000 euros per year (for a maximum of four years) for a successful application in life sciences by a senior investigator (all are carefully reviewed by external referees, mostly from overseas). The ERC might give to an investigator of the same stature 700,000 euros per year, for five years. This is unnecessarily generous and generates the science equivalent of social inequality, perpetrating and promoting the dominance of science entrepreneurs.
4. The Redress process has to be reformed. In its present form, it is useless. As detailed under point B9, I put great effort in writing a detailed Request for Redress. It appears that this was never brought to a discussion by experts and I fail to understand the rationale of having a scientist act as a “Redress Coordinator” if scientific issues are not discussed.
I apologise for the length of this letter but I think that we are dealing with issues of critical importance, which might determine whether Europe can generate a fair mechanism of support for science, free not only of politics but also of science politics, which can make sure that the best people and the best science are supported (or, at least, that good scientists and good projects have got a fair chance, too).
Edgar Pick, M.D., Ph.D.
The Julius Friedrich Cohnheim Laboratory for Phagocyte Research
Department of Clinical Microbiology and Immunology
Sackler School of Medicine
Tel Aviv University
Tel Aviv 69978