How to peer review

[article index] [] [@mattmight] [+mattmight] [rss]

I've been asking my graduate students to perform peer reviews of scientific work in areas where they know more than I do.

They've done an excellent job, but I've also spent time repeating advice on how to write a good peer review.

I've compiled that advice for graduate students writing their first reviews.


Peer review distills a paper to its essence.

The key principle to writing a good peer review is easy to understand, but more difficult to implement than one would expect:

Review the manuscript as you would want your own reviewed.

Modern life is fraught with agents that blunt empathy.

Peer reviewers must mitigate these agents.

Read on for my tips.

Avoid decision fatigue

I've written a few hundred peer reviews.

The dominant factor in whether I wrote a high- or low-quality review was decision fatigue.

Decision fatigue impacts far more than peer review. As humans, we make decisions all day long. Each decision draws from a finite (but replenishable) cognitive resource.

Reviewing an entire scientific manuscript drains that resource.

The book Willpower is a compelling survey of the research in this regard:

I recommend it to all grad students.

The book presents advice that we can adapt in the context of peer review:

  • Review one paper per day.
  • Review in the morning, before decision fatigue sets in.
  • Never review on an empty stomach.
  • Never review when you're tired.
  • If you must review papers in succession, rest between reviews.

I've never observed all of these rules for all papers in a single venue.

There have been times when it was mathematically impossible to review only one paper per day.

But, I do my best to stick to these principles. It shows when I do.

Recognize conflicts

You can have several kinds of conflict of interest.

Many venues use the NSF's proposal-reviewing conflict of interest policy as the basis for their conflict policy.

The policy conservatively eliminates many conflicts of close affiliation: family, advisor, advisees, recent paper authors or collaborators or those at the same, recent or potential institutions.

There are other conflicts not covered, for which you should refuse review.

If you're not sure whether you have a conflict, disclose that to the editor or program chair so that they can decide.

Friends

If you're a "close personal friend" of an author, decline review.

There's debate about the what "close" means, so ask yourself the following:

Would you review the paper impartially?

If the answer is no, she is too close.

And, the meaning of close can fluctuate over time.

Is she nearing tenure? Is he about to go on the academic job market?

In such cases, you might be more tempted to "help."

Foes

It is equally important to recuse yourself if you have strong personal objections to an individual or work.

Even if you think you can review the manuscript fairly, decline to review, since a negative recommendation would be seen as tainted.

Scholarly conflict

Every so often, you'll receive a manuscript to review in which you discover that someone else has independently solved the same problem as you.

These manuscripts cause heartburn.

I recommend informing the program chair or editor and recusing yourself from discussion of such manuscripts, since it is unlikely that you will be able to review it without some emotion.

If the paper is accepted but your work provides a better solution, you may be able to publish it.

If the paper is rejected, you may still want to walk away from publishing your work, since there could be accusations that you benefited from seeing their work.

If you decide to pursue publication, you can approach the authors directly or through the program chair or editor.

Explain that you recused yourself from discussion once you recognized they had solved the same problem.

Provide them with all of your work to date, and offer to combine your insights for a future submission.

Promise to respect whatever decision the authors make.

Pigeon-hole conflict

There is a perception among some graduate students that if a venue will accept N papers, then giving a favorable review to a competing submission reduces that to N-1.

This is simply not true.

Papers are judged on the standards of the venue, and this tends to yield a similar rate of acceptances every year.

If you feel you can't review a manuscript objectively because you have a competing submission to the same venue, then decline to review.

Don't think that tanking a competing submission will improve your odds.

It will not.

And, if anyone suspects your review was tainted by a perceived pigeon-hole conflict, it could do measurable harm to your reputation.

Financial conflict

If, for some reason, you have a financial stake in the acceptance or rejection of a paper, then you should decline to review.

This is not common, but if it happens, you need to disclose it to the editor or program chair.

Don't review like a grad student

Reviews written by graduate students are among the most negative.

Grad students are often the on the receiving end of negative reviews, because they are just learning how to write papers.

This colors their perception of what a typical review should be.

Not all reviews are negative.

I recommend that grad students focus on listing the positive aspects of a paper before moving on to criticism.

Acknowledge visceral reactions

Record your initial reaction to a paper.

You may form an accept/reject opinion from just the title and the abstract.

That gut instinct frames the way a reviewer reads a paper.

Confirmation bias sets in, and reviewers will look for evidence that supports their initial predisposition.

If your initial reaction was strong, write down why it set your expectations.

Acknowledging your gut instinct will help you avoid confirmation bias, and it will also give the authors feedback when framing the paper for a resubmission.

Summarize

Write the summary last.

Your summary should allow the authors to quickly estimate your understanding of the paper.

Include:

  • your interpretation of the core idea;
  • what you view the contributions to be;
  • general areas in need of improvement, e.g. presentation, proofs, experimentation; and
  • your recommendation for acceptance or rejection.

Hold the snark

Peer reviews are not the right venue to show off your wit.

Save your dismissive one-liners for confidential discussions with the program committee or the editor.

Putting them in the review itself will only cause harm.

It will cause the authors to view the review as flawed, and any thoughtful feedback you have given will be completely ignored.

Respond as you read

I recommend recording your thoughts as you read the paper.

Such a stream of consciousness tells authors how the paper "flows."

It can be mined to find the point at which the tide turned definitively for or against the paper in the reviewer's mind.

Report as you find:

  • Items you like / don't like.
  • Findings that surprised or disappointed you.
  • Concepts that were well or poorly presented.
  • Anything that confused you.
  • Anywhere you feel an example would have helped.
  • Anything that contradicted your running expectations.
  • Anything unclear, even if later text clarifies it.

Offer constructive criticism

When you criticize an aspect of a manuscript, offer advice on how to fix it.

For example, if the presentation is poor, identify why:

  • Is the grammar poor?
  • Are transitions stunted or awkward?
  • Are concepts, definitions and theorems convoluted?
  • Are the diagrams unclear?
  • Does the author assume too much of the reader's knowledge?
  • Are explanations inadequate?
  • Are more examples necessary?

If there are technical mistakes, can you correct them?

If the experimental results are flawed, can you suggest a way to correct the experimental design or to properly interpret the results?

If you recommend rejection, what specifically could have changed your mind to argue for acceptance?

Anonymize but don't hide

It's important to frequently change your style when you review.

Don't use a standard personal format which might become identifiable.

My advice when writing a review is to avoid using less common conventions, and to avoid your personal style:

  • Use American spelling.
  • Avoid the Oxford comma.
  • Avoid logical quoting.
  • Vary your choice for bullets: use numbers, +, -, o, and *.
  • Avoid all emoticons, especially culturally distinct ones.
  • Vary indentation, spacing and formatting between reviews.

Also, be cautious when citing your own work in a review.

Finally, don't hide behind the veil of anonymity to lob attacks.

Anonymity in peer review is meant as a shield, not a sword.

Keep it impersonal

My colleague Miriah Meyer added that it is important to focus on the work and the paper under review rather than the people doing the work.

Don't let the track record of the authors sway your opinion in either direction.

When commenting, avoid referencing the authors personally.

All comments should address the paper itself and the work it describes.

Empathize

Once you've completed your review, ask yourself if you would be satisfied with the quality had you received the same for your own work.

If the answer is no, revise.

Related pages


[article index] [] [@mattmight] [+mattmight] [rss]