woah-blog

Someone over on a discord channel I hang out in asked for some thoughts on a blog post made by Adam Mastroianni on the subject of Peer-Review [1,2]. I have some years of experience in academia, during which I have published and reviewed papers, so maybe I would have some thoughts? I have heard of people trying to discuss peer-review before, when I was still in academia, but have never taken the time to look at it. Now I did take the time, and it turns out I did have some thoughts on the subject.

What is missing?

When reading Adam's posts, I find that there are two points which I think are very central to a function peer-review process, which Adam does not really touch upon. These two missing points are:

1. The journal editors
2. A functioning pre-print culture

I think that these points may not have been reflected on by Adam because they are not present in the same way in his field, so I thought it would be interesting to talk about them. I used to work in physics, and even physics is not homogenous on this, other physicists in other areas may not have the same view as me on 1. and 2.. Then I can imagine that Adams subject is even further away. When discussing these two points I will comment on some statements that Adam made that I think relates. I will also discuss two points that Adam does bring up, and that is:

1. Public data
2. Publishing reviews

Because I think there are some details that need mentioning. Lastly I will comment on “The point of peer-review”. But first what is peer-review?

What is peer-review?

As a student, my naive picture of peer-review was basically: An independent scientist have checked that the paper is “correct”. Very fast it becomes clear that that is not the what peer-review does. You start working on your first paper, and you notice, even if you lack the experience to be properly skilled at it yet, that it takes a very long time to check stuff. Doing all the analysis for a paper — calculations, analysing data, interpreting — can take years. Can we reasonably expect a reviewer to reproduce and verify this with high certainty in a month or so? No!

There is no reason to definitely trust a paper that is published. What you do to be certain of a paper is either to: study it yourself or talk to someone you trust that has studied it. Only the people who had to extend the work of a paper as had the chance to independently go though it and verify it, because they spent years working on it. A reviewer? No chance.

For myself I started to form this picture of our field being a form of oral wiki. I sat down with a paper, I did some of the calculations, and I noticed something was wrong. This paper is 20 years only, how could that be? I talked to a collaborator:

Yeah, it is a well known error. The exponent should have a 2 not a 3. It is quite obvious when you study it deeper, they comment on it in this other paper, read that.

This is what emerges: The correct thing may not be written down in the paper, the paper has no errata, not updated pre-print. People just know. Sometimes only a handful of people know, and then you have to find out who it is and talk to them. This is a closer picture of what should be considered “correct” than trusting peer-reviewed papers. I'm not saying it is entirely good, for people who are more introvert or beginners, finding out these things can be hard, creating an unnecessary high threshold for them to start doing their research. What I am saying is that even if papers have mistakes there is a whole other dimension to where the “truth” is stored.

If we are to discuss what function peer-review is supposed to have, I think we must first understand that what peer-review cannot do, which is to certify that a paper is “correct”. What function does and should peer-review have then? I would argue that is has still a few. But before we get there, I want to discuss some of the points I mentioned earlier.

1. The Journal Editors

Adam talks about unreasonable reviewers in a few places, like here [1]:

If you try to publish a paper showing that, say, watching puppy videos makes people donate more to charity, and Reviewer 2 says “I will only be impressed if this works for cat videos as well,” you are under extreme pressure to make a cat video study work.

and here:

Remember that it used to be obviously true that the Earth is the center of the universe, and if scientific journals had existed in Copernicus’ time, geocentrist reviewers would have rejected his paper and patted themselves on the back for preventing the spread of misinformation.

I do agree with Adam that such reviewers exist, I have had a lot of experience with them, and it is a potential worry. However, I do not think that is an argument against peer-review in anyway. I have had reviewers reject my paper several times, and I have always managed to publish that paper in the same journal anyway (except once). We did not, which Adam brings up “just submit the same paper to another journal”. How did we manage that? Me and my collaborators talked to the editor.

I did not publish crap. I knew what was in my papers were correct, so if a paper got rejected the reviewer had to give a good argument. There was a common main reason my papers were rejected: my papers criticized the current mainstream understanding of an issue. The general worry was that my papers would give the subject a bad name, and you could see that in some reviewers: they had a preconceived idea of what the paper was, why it was wrong, and hence did not properly read it. So when they rejected the paper, we made the argument to the editor in charge of our review that the reviewer did not take our paper seriously and made it clear that we request a new reviewer.

The editor in charge should act as a moderator for a discussion between reviewer and authors. You can make your case to the editor why the reviewer is being unreasonable! We always won that fight (for the one exception, see [3]). For my papers it was an ideological battle, but this goes for any submission to a journal that is met with an unreasonable reviewer: talk to the editor!

I do not share Adam's worry on the “geocentrist”-analogy, nor the cat-person reviewer, because I would just talk to the editor. If in Adam's field you cannot talk to the editor, or editors do not act independent moderators, then the journal is corrupt. I would not say it is a necessary issue of the peer-review process in itself, it is a problem of the journals. Or perhaps it is the culture that has emerged in the field? If you cannot have a reasonable, moderated, dialogue, then yeah, peer-review must seem more like a problem.

2. Pre-print culture

Adam only mentions pre-prints once in [1]. I find it strange that there is no proper discussion of it, since it appears that they still have pre-prints. But when he says [1]

Well, last month I published a paper, by which I mean I uploaded a PDF to the internet.

and continues later with

Total strangers emailed me thoughtful reviews. Tenured professors sent me ideas.

I'm sorry, but this is just good pre-print culture! What we do each morning (basically) is that we go to the pre-print server feed for today and we look through the papers that appeared last night/over the weekend. You read the titles, perhaps the abstract if they relate to something you have done, if they seem to comment on your work you open them up. “Does this-and-that look correct? No? Maybe I'll send them an e-mail. Do they cite me? I should correct them on that.” When you upload a paper on the pre-print server, you know that this is what people do, so you always wait a few weeks/a month to send it off to a journal, because it is very likely that someone will send you some comment, praise, or whatever. That is what you want! A good pre-print culture.

If that is what the problem is, then the question to ask is: what is the reason for there to be a bad pre-print culture in your field? Is the pre-print server feed too flooded with garbage papers? Does the server need moderation? Do you need new sub-field feeds? If this is the problem — people do not give spontaneous comments on “uploaded pdfs”, or people to not make jokes and are creative in their writing — it is not directly a problem with peer-review, it is a problem of the pre-print culture.

Pre-prints is also where we add jokes and come up with funny titles. The journal may ask us to remove and change them later. But no one is using the published versions anyway, so the jokes often remains. Pre-prints also have the feature that they are much easier to update. In my field all versions of a paper is saved on the pre-print server (arxiv), and you add a comment to each paper, like: “5 pages + appendix”, “published version, 4 pages, 2 figures”. It is very easy to produce a new version and papers are therefore often updated after publication, often correcting minor errors.

3. Public data

Earlier in this text, I mentioned that analysis and everything else that is necessary to produce a paper can take a long time. To add to this, sometime it is because the datasets are very large. Gigabytes and even terabytes are not unusual. When Adam writes:

Why don’t reviewers catch basic errors and blatant fraud? One reason is that they almost never look at the data behind the papers they review, which is exactly where the errors and fraud are most likely to be. In fact, most journals don’t require you to make your data public at all.

I do realise that our fields probably look very different. I can imagine that in many fields this is borderline impossible to just start doing even if you you think it is the right thing to do. The journal won't accept a TB of data from you, where can you host it publicly? I agree completely that it is something we should be doing to a larger extent, but the infrastructure does not exist. That is the first thing, the second: What is the reviewer suppose to do? Rewrite the analysis tools to make sure that the paper gets the correct results from that data? For a single person working independently? That can take years! What I was saying before about papers being “correct” was that you need someone independent to work on it to makes sure it is correct or not, and the reviewer will not be able to do that.

While I generally agree that data should be made public, there are also non-technical issues associated with it. A collaboration generating a bunch of data may not want to be associated with fraudulent paper that happens to use that data, even if they are not part of the collaboration. The some collaborations are therefore very strict with who can work on the data. And I can understand that. My field is not very political, but for fields that are, the far-right does this all the time: take statistics and data out of context to build support their false world-view. Some research groups have decided that you do not want to end up in a book of some anti-vaxxer that has “proven” that 5G causes Covid, by “using” your data.

4. Publishing reviews

This is an interesting idea, and Adam mentions it:

[O]nce a paper gets published, we shred the reviews. A few journals publish reviews; most don't. Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place.

One thing that is very easy to find out for us is “how the authors edited their paper in response”: the pre-print versions! I have done that many times: Download the original version tex-file, download published version make a word-by-word diff. It is a very useful thing to do if you want to really get to know a paper.

That aside, I think it is an experiment we should do. My worry is though that it makes it very easy to post-review de-anonymize the reviewer. Some fields are very niche, perhaps even below 10 people active, often work with each other, and they are almost always selected as reviewers for each other. You will only have a few candidates, if you have ever talked to them or read one of their papers, now it can be quite obvious who it is.

But maybe that is a good idea to do, such that reviewers have some extra responsibility for what they write? Sure, maybe. But, I would be worried of how people would self-censor, especially younger people. If a student get to review their supervisors paper? Would they be as critical as they should in their review? If it is another professor, will they not be invited to a conference in the future? Fail to find the next postdoc? Be blocked from a tenure track position? I think this is a serious risk to take.

The point of peer-review

Through all this, peer-review seems to serve little to no purpose. But I would still say that it does. It is, as it looks today, a necessary purpose, even though it is a bad solution to that problem. Bureaucracy.

In this text I have described how there is some form of collective understanding of which papers are “correct” and can be trusted, which papers have errors and where those errors are made obvious. When a university is going to hire you, or when you apply for funding, then someone who does not know the field, the papers, the people, have to figure out who to trust. We try to solve it with published papers in various ways: number of cites, cites per paper, h-index. All these numbers have some blind-spot and brilliant people may end up not being able to continue their careers just because of some dumb metric, when it is obvious that they should. But it is a crude solution to a problem that exists everywhere: who should get a welfare check, and who is “just a leech”? How does the university know that the candidate selected to be hired is not just there because of nepotism? Peer-reviewed publications.

While I disagree with Adam on many of his points (mostly, possibly, because they are not applicable to my field), the discussion he has spawned is a very healthy one to have. There is a problem to be solved here, and it is solved in a very bad way.

After all this, I still ask myself: What is the actual critique against peer-review? The “number of published papers”-is-a-bad-metric critique is not even a critique of peer-review itself, in the same way as I have accused Adam's point for not being critique of peer-review itself. As I have argued: If reviewers do not catch all errors, that is not a problem, since an healthy amount of scepticism will get you there in the end; If reviewers are unreasonable, that is not a problem because you can talk to your editor; If you cannot be creative with your writing, do not get response on pre-prints, then you need a better pre-print culture.

The critique is: It costs money and time, to what end? Adam's titles are strong, “A WHOLE LOTTA MONEY FOR NOTHIN’”, but what are the conclusions?

All we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal.

(my emphasis) I'm still confused: Is it actually nothing? Did it actually hurt?

Footnotes

[1] https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review

[2] https://experimentalhistory.substack.com/p/the-dance-of-the-naked-emperors

[3] The exception was when we were rejected by the editor after the reviews, because while the paper was correct, the journal was too prestigious for our paper. The editor instead automatically cleared the paper for a more standard journal at the same publisher immediately. Disappointing, but reasonable.