Hints on how to write an SOSP paper

After reviewing between 50-100 papers per year for the last few years, I’ve noticed some common problems in the papers that I review. While spotting these issues makes it faster for me to review papers, I thought I would summarize them here. Note that these are generally aimed at systems papers but likely apply to other areas as well.

Generally speaking, these are the bare minimum requirements for a systems paper and I frequently see papers rejected for missing one or more of these points. I break them down into roughly three sections. A useful exercise is to summarize each of these parts of your paper (i.e., the why, the how and the results), which can then be turned into the conclusion.

The Why: Intro, Background and Motivation

• The goal of a systems paper is generally to state a thesis and demonstrate that thesis through a software (or hardware) prototype and series of experiments. For example, “this paper proposes that X is better for applications Y running in environment Z”. While there might be a bigger picture thesis or point, when it comes down to brass tacks, this is the core of a good paper.

• First, clearly understand and state your target environment (Z) and application (Y). Many people miss this point and it makes the rest of the paper extremely difficult to understand!!

• Clearly motivate Y and Z. Why is application Y important? Why should is it ok to assume that Y commonly runs in Z?

• Clearly state why X is novel. In very few cases is it only X that is new (unless you have some new super smart algorithm). Instead, many systems papers are about a new application Y (the hot one now being ML workloads) or Z (being some new hardware, like NVRam, or deployment environment, like mobile devices). This is your chance to teach the community about an exciting new Y or Z!

• Bonus: What is the lesson to be learned from X? This is an extra level that many papers miss. Is there a broader lesson to be learned from X that could be applied to other Y and Z?

• Clearly state why previous systems do not meet the needs of Y for Z. This is often most easily done by making a table of these previous systems and show that X better meets the needs of Y for Z based on some system aspect. I often have students make this table even if it does not wind up in the paper to make sure that we all understand the related work clearly.

The How: System Design

• Now you can get down to talking about X. Clearly state Y and Z again and use a figure to show where X fits. Do not assume that this is obvious!

• It might be relevant to discuss how X interacts with Y and Z, especially if it is different. For example, if X has a new API or new operations for Y or has new requirements for Z, summarizing these in a table is important.

• If X introduces a lot of new terminology, a table that summarizes and defines the new terms is important. It is easy to miss these definitions in the text.

• Every design choice made in X should be discussed with alternatives. Almost all design choices come with trade-offs. Do not only talk about the benefits of X but why the particular trade-off made by X is right for Y on Z.

• Some problems might be out of scope. Clearly state what X does not handle and justify why. Do not assume that reviewers can guess.

The Results: Implementation and Evaluation

• Explain how you constructed a prototype system to test your hypothesis. This prototype might exactly match X or there might be reasons why it cannot. It likely does not implement all of X thoroughly but the parts of X that are needed to test your hypothesis, so clearly state the parts that are implemented.

• Clearly state the set up for every experiment. So many papers miss this. How many times did you run the experiment? For how long? State every data object size!

• Every experiment should have a point; that is, each subsection and graph should have a concrete conclusion that contributes to demonstrating your hypothesis.

• Clearly state each conclusion 3 times: at the beginning of the section as a hypothesis (e.g., this experiment will show), at the end of the section (e.g., this experiment showed), in the caption next to the graph (e.g., this graph shows). Bonus points for explaining how to parse the graph in the caption so there is no need to dig through a bunch of text, then page back to the graph.

• There are generally 2 types of experiments. The first are somewhat obvious; they compare X to a previous system for Y and Z. This could use a number of metrics (e.g., performance, programmability, security, etc.). There should not be surprises here! Many papers fall down here when the experiments just do not match with hypothesis.

• The second breaks down the performance (or other metric) impact for each design decision and potential alternative design(s). Many papers fail to include these but they are really important for understanding which parts of the system are fundamentally important to your hypothesis.

Bonus part 4: Conclusion

• The conclusion should be a useful exercise in summarizing your paper in 3 sentences. What is your hypothesis? What is your solution for testing this hypothesis? What is the result of this experiment? Now, you are ready to go back and write the abstract as well! In general, the abstract is harder to write because you cannot use terms or concepts that you introduced in paper, so someone with no background can read it. However, the conclusion does not have this limitation, so should be easy to write.