Slow Software: The Case for High-latency Systems Development

Systems research prizes efficiency and performance. For the last decade, I’ve worked on ultra-low-latency systems; the goal of Demikernel and its related nanosecond-scale systems (e.g., Cornflakes, Capybara) is to do more with fewer cycles. AI coding has brought the same level of efficiency to building systems, letting fewer programmers write more code. However, it has also decoupled the speed of development from the importance of the software causing failures with a “wide blast radius” to happen more frequently and be more difficult to fix. These problems will only get worse unless we find a way for slow software to enforce thoughtful design and development for our most critical systems.

Background: Naturally Slow Systems

Until now, systems building has always been time-consuming. First, low-level systems development is tied to hardware, which must be physically built and deployed. Next, deployment of systems software can be slow; it often required software to be physically delivered and installed or for the entire system to be rebooted or replaced. Finally, building large systems using basic tools – usually assembly or C – meant that implementation complexity limited development speed.

These barriers imposed a rough correlation between a system’s development pace and importance. The more software relied on a system, the more effort was required to implement and update it. Likewise, the larger the blast radius of a system failure, the more time naturally was spent building the system and making sure that it did not fail.

In the last 20 years, these barriers have been systematically removed. Hyperscalars have been building datacenters at a faster rate than ever and continuously deploy new hardware. Most software runs on a centralized cloud which enables weekly or even more frequent updates. AI, while not perfect, has made it vastly faster to write systems code without using basic tools. Over time, the importance of a piece of software has been completely decoupled from the time to develop it.

This decoupling lets people treat systems software like any other software. The same push to use AI is being applied equally across the board, to our testing infrastructure, network routing, datacenter management and operating systems. Recent work has highlighted how to use AI to improve system performance without understanding the entire system or implementing the optimizations. Programmers are rapidly making changes without understanding the system or the guarantees that we use to reason about underlying infrastructure.

While this approach works well for a lot of software, it has implications for systems building. As a concrete example, distributed systems depend on consistency guarantees provided by an underlying storage layer. However, storage systems today commonly have complex weak consistency models that are not well documented and may not be correctly understood by AI models without adding the storage system’s entire code base to the context (which might not be available or is constantly changing itself). As a result, the AI model may change some code that violates the consistency assumptions and the programmer may not find it unless she knows where to look.

This attitude has led to predictable side effects. Inevitably, something will break (because neither humans nor AI are perfect). Only then does the difference between systems software and other software become apparent, as the wide blast radius takes down many other systems. As an analogy, we all use real world products that are made quickly and cheaply (e.g., processed food, IKEA). However, we also want some things that are not (e.g., fruits and vegetables, roads and bridges). Software is exactly the same; we do not have a good way to distinguish important infrastructure systems from other software until something goes wrong (e.g., health problems arise, the bridge collapses).

The Slow Software Movement

We need to re-establish some of the coupling between the importance of a software system and the time to develop it. People need time to consider their design decisions and forcing them to move more slowly imposes this time. Thus I propose the slow software movement, akin to the slow food movement. The goal is not to slow down all software development but to identify the important infrastructure software, where the slower speed of development is a feature, not bug.

More concretely, the more dependencies on a piece of software, the more effort should be put into developing it. And the more impactful the failure, the more effort should be put into avoiding failures. Experienced programmers can make this distinction. Mark Russ recently talked about how he knows what code he needs to read and doesn’t, and argued for a mentorship program to impart the same experience on others. However, I believe this doesn’t necessarily need to be entirely learned from experience, but we could build something into the development environment to help.

Not only do new programmers need to spend time thinking about these things and learning, experienced programmers should as well. And it should be clear which software requires more thought and what does not. In lieu of natural impedence, we could perhaps introduce artificial barriers to push programmers to have more skin in the game, so to speak, when they decide to implement something important. They will eventually need to invest time in maintaining the software and making it faster to write up front means more pain later when it breaks, especially for younger programmers.

Note that I am not saying that we should not use AI to write code. It is clearly useful for many tasks that involve tedious or repetitive code. If I am building a new system, I should not need an entire team of engineers to build compilation scripts, testing infrastructure or evaluation frameworks. However, this code has a very small blast radius when it fails, so it should be developed faster. On the other hand, faster systems building has costs and we should make those costs felt up front when writing the code, not only when fixing it later.

Systems Research Opportunities

What does this mean for systems research? Just because we can prototype any systems idea with AI in a way that is fast and cheap doesn’t make it a good idea. In the past, it was extremely expensive to prototype an idea, so researchers thought a lot about it before implementation. Thus, almost any idea that was worth building was worth publishing (e.g., the first SOSPs).

Now, that barrier has been removed because it has become extremely cheap to implement a prototype. Our conferences are clearly being flooded with poorly considered research papers, which now can be implemented, evaluated and written extremely quickly. However, this drive to more efficiently produce research papers has removed the hard parts of systems research. There’s no clear indication that just because an idea can be AI-tested that it will hold up under real world use.

We’re relying on experienced reviewers to guess at whether a systems idea will work in the real world. Unfortunately, this is like trying to guess whether an architectural model of a building will stand up or last decades under less than ideal weather conditions. Even if the reviewers could guess at it, it’s not a sustainable way to do useful research. As a result, we’re merely producing papers but not actually contributing or evaluating useful ideas.

So we need a plan for how to fix this. We didn’t create the problem but we should not lean into it either. Relying only on our most experienced reviewers that have been building and maintaining systems for decades to use their intuition will only burn them out. Instead, we should figure out how to do research to re-establish best practices for systems development. We should actively try to figure out how to judge the practicality and maintainability of systems, along with performance and generality. While I only have a few answers, the systems community has adapted in the past and I trust them to do it again.