The challenge: Find the best research

“In our case, the assessment is both easier and more difficult than for others. More difficult because it absolutely cannot be wrong, easier as we have time on our side. A scientific breakthrough leaves an impression that can be assessed retrospectively. It is no accident that it can take 20 years before a discovery is rewarded,” says Thomas Perlmann, professor of molecular developmental biology at the Department of Cell and Molecular Biology, recently appointed secretary of the Nobel Committee and one of those who will decide who receives this year's Nobel Prize in Physiology or Medicine.

There are many possible answers to the question of what makes good research. One is that the question at issue must be relevant and that the study set up in the attempt to answer the question must be designed correctly. The format of the study has an impact on the chances of interpreting the results, for example if there are sufficient patients in order to draw statistically significant conclusions. Another is that the research is original and pioneering, something that is emphasised in the context of the Nobel Prize. Alfred Nobel's will stipulates that the research that is rewarded with the Nobel Prize in Physiology or Medicine has to be a discovery.

“Such a breakthrough will change ways of thinking from their very foundations and there has to be a clear before and after in the field, or an entirely new field has to be opened up,” says Thomas Perlmann.

Good research also has to be ethical and conducted well. This means avoiding, for example, a large number of people for whom the treatment has had no effect leaving the study, that the samples taken are mixed together or contaminated or that straightforward fraud means that the results cannot be trusted. In addition, it can be argued that the research has to be of benefit; in other words, that it is possible to make use of the results.

Accordingly, there are many facts that have an impact on whether the research is good and thus much that can go wrong. The Nobel Committee is able to avoid these pitfalls, primarily by penetrating the research deep inside. In other cases, for example when someone is to be appointed to a post or a funding body is to allocate research funding, other assessment criteria are used for reasons of time and resources. What is measured is primarily citations, which means that a certain scientific article is used as a reference in a subsequent article (please refer to the glossary). The articles cited most frequently are often those that are published in highly ranked journals, with a high impact factor. Accordingly, it is considered important to be published in these journals.

But this can cause problems. Within the research community, we often talk about something called “the rush to publish”, i.e. that is it becoming more important to publish in the right journal than to do good research. This can result in a lack of quality. Thomas Perlmann provides an example:

“Imagine that the editor of the very highly regarded journal Nature says that the article you have submitted was interesting, but that complementary experiments are required in order to get the article published. It is easy then to think that 'I've got my foot in the door of Nature, no silly experiment is going to prevent me from publishing!'”.

This increases the risk that the individual researcher fudges their data and perhaps chooses the experiment that supports the thesis. This sort of sliding involves publishing results that are not as robust as they can be. This puts obstacles in the way of other important quality criteria, namely that it is not possible for the results to be repeated by other researchers. This reproducibility involves other researchers and research groups being able to obtain the same results if they repeat the study, which is the test that shows that the results are reliable.

A lack of reproducibility has been noted a great deal in recent years. For example, there was great excitement a few years ago when researchers at the pharmaceutical company Amgen attempted to reproduce 53 studies that had had a major impact in the field of cancer research. It was only possible to reproduce six of these. Thomas Perlmann also describes how an entire “story” is required in order to be able to publish in the highly ranked journals.

“The article has to contain everything! An exciting hypothesis, a series of experiments in several different model systems – from the cellular level to experiments on humans – and preferably a breakthrough,” he says and continues:

“What happens then is that what is presented sometimes appears worryingly complete, with all the pieces of the puzzle sitting a little too perfectly in their place. Those of us who are looking for real breakthroughs know that these are very rare. There are definitely not scores of these in each issue of the established journals, even though the articles are often presented in this way.”

Critics of the current system have therefore questioned whether the journals in which a researcher publishes and the number of citations the publication receives is actually a good measure of the quality of their research. Pauline Mattsson, researcher at the Department of Learning Informatics, Management and Ethics, has interviewed 34 Nobel laureates in physiology or medicine and collated information about publications, patents and collaboration with industry as part of a current project.

“When we look at all the articles they have published, in half of the cases, their most cited article is also the one for which they were awarded the Nobel Prize. But in the other cases, it was other articles that were the most cited. This may, for example, involve methodology that has been refined and developed,” she says.

Many citations also does not necessarily mean that the research is “better”. The number of citations is also influenced by other factors such as when an article is published. If it was published a long time ago, this means there is a chance it has more citations. The type of article also has an effect, review articles result in more citations, as does the field the article concerns – articles within medicine and biochemistry traditionally reference many articles, while ethics or physics have few references. The size of the field also has a role to play, whether there are 100 or 10,000 researchers in the field. Henrik Schmidt works as a librarian providing services to researchers at Karolinska Institutet's library in Solna. He emphasises the problem of individual measures.

Gert Helgesson, professor of medical ethics at the Stockholm Centre for Healthcare Ethics at Karolinska Institutet, also agrees with much of the criticism levelled against the use of individual indicators as measures of research quality. He emphasises the differences between different field of research.

“In medical ethics, for example, it is extremely rare for our results to be published in the journals that have the highest impact factor, although it does happen. I said to my students recently that it was previously a case of 'publish or perish'. Instead the situation is now 'publish AND perish' – getting published doesn't help if it is not in the best journals,” he says and continues;

“But having said this, I must emphasise that attempting to measure the quality of the research in a formalised and objective way is, in principle, a good idea as it combats nepotism, in other words preferential treatment not based on a person's actual merits. This is something not to forget.”