1 The first pass
The first pass is a quick scan to get a bird’s-eye view of
the paper. You can also decide whether you need to do any
more passes. This pass should take about five to ten minutes
and consists of the following steps:
- Carefully read the title, abstract, and introduction
- Read the section and sub-section headings, but ignore everything else
- Read the conclusions
- Glance over the references, mentally ticking off the ones you’ve already read
At the end of the first pass, you should be able to answer
the five Cs:
- Category: What type of paper is this? A measurement paper? An analysis of an existing system? A description of a research prototype?
- Context: Which other papers is it related to? Which theoretical bases were used to analyze the problem?
- Correctness: Do the assumptions appear to be valid?
- Contributions: What are the paper’s main contributions?
- Clarity: Is the paper well written?
Using this information, you may choose not to read fur-
ther. This could be because the paper doesn’t interest you,
or you don’t know enough about the area to understand the
paper, or that the authors make invalid assumptions. The
first pass is adequate for papers that aren’t in your research
area, but may someday prove relevant.
Incidentally, when you write a paper, you can expect most
reviewers (and readers) to make only one pass over it. Take
care to choose coherent section and sub-section titles and
to write concise and comprehensive abstracts. If a reviewer
cannot understand the gist after one pass, the paper will
likely be rejected; if a reader cannot understand the high-
lights of the paper after five minutes, the paper will likely
never be read.
2 The second pass
In the second pass, read the paper with greater care, but
ignore details such as proofs. It helps to jot down the key
points, or to make comments in the margins, as you read.
- Look carefully at the figures, diagrams and other illustrations in the paper. Pay special attention to graphs.Are the axes properly labeled? Are results shown with error bars, so that conclusions are statistically significant? Common mistakes like these will separaterushed, shoddy work from the truly excellent
- Remember to mark relevant unread references for further reading (this is a good way to learn more about the background of the paper).
The second pass should take up to an hour. After this
pass, you should be able to grasp the content of the paper.
You should be able to summarize the main thrust of the pa-
per, with supporting evidence, to someone else. This level of
detail is appropriate for a paper in which you are interested,
but does not lie in your research speciality.
Sometimes you won’t understand a paper even at the end
of the second pass. This may be because the subject matter
is new to you, with unfamiliar terminology and acronyms.
Or the authors may use a proof or experimental technique
that you don’t understand, so that the bulk of the pa-
per is incomprehensible. The paper may be poorly written
with unsubstantiated assertions and numerous forward ref-
erences. Or it could just be that it’s late at night and you’re
tired. You can now choose to: (a) set the paper aside, hoping
you don’t need to understand the material to be successful
in your career, (b) return to the paper later, perhaps after
reading background material or (c) persevere and go on to
the third pass.
3 The third pass
To fully understand a paper, particularly if you are re-
viewer, requires a third pass. The key to the third pass
is to attempt to virtually re-implement the paper: that is,
making the same assumptions as the authors, re-create the
work. By comparing this re-creation with the actual paper,
you can easily identify not only a paper’s innovations, but
also its hidden failings and assumptions.
This pass requires great attention to detail. You should
identify and challenge every assumption in every statement.
Moreover, you should think about how you yourself would
present a particular idea. This comparison of the actual
with the virtual lends a sharp insight into the proof and
presentation techniques in the paper and you can very likely
add this to your repertoire of tools. During this pass, you
should also jot down ideas for future work.
This pass can take about four or five hours for beginners,
and about an hour for an experienced reader. At the end
of this pass, you should be able to reconstruct the entire
structure of the paper from memory, as well as be able to
identify its strong and weak points. In particular, you should
be able to pinpoint implicit assumptions, missing citations
to relevant work, and potential issues with experimental or
analytical techniques.
