The System Paradigm

One of my favorite talks is “Engineering(,) A Path to Science” by Richard Gabriel.¹ He relates an experience he had with incommensurability and a paradigm shift from systems to languages. Gabriel had been involved in engineering Lisp systems, publishing papers, organizing and speaking at conferences, then he went back to school to earn a Master of Fine Arts, upon returning to engineering the landscape had changed.

At first Gabriel thought that while he was gone there had been a shift from an engineering paradigm to a scientific paradigm, but eventually he saw it more focused. It was a microparadigm shift from a system paradigm to a language paradigm.

In the system paradigm, people build systems of interacting components forming a whole; real things doing real stuff. One could observe and experiment with a running system as if it were a living being. With the language paradigm, people define languages with symbols and grammars that convey meaning. Compilers make mathematical transformations that preserve the meaning of the symbols.

Good design for the system paradigm is the result of an experienced technician putting skills, knowledge, and best practices to use. The system can help with good design, but good design springs from the art and skill of the programmer. Good design in the language paradigm is the result of a language designer carefully crafting the rules of grammar to make bad design ungrammatical; the compiler enforces good design by enshrining good techniques into the rules of the language.

There is much more to enjoy from Gabriel’s talk, and I highly recommend it. But the first time I heard it this idea of system vs. language was an important concept to me. I have a Masters degree in computer science, and I like to read academic papers. I used to call myself an “armchair computer scientist,” but I realized that I liked to think in the system paradigm. I liked to tinker with running programs. I started to self identify as an engineer. I am an engineer.

I also had a better understanding of why I liked to work in Clojure.

Organisms

“Pascal is for building pyramids—imposing, breathtaking, static structures built by armies pushing heavy blocks into place. Lisp is for building organisms—imposing, breathtaking, dynamic structures built by squads fitting fluctuating myriads of simpler organisms into place.”

—Alan Perlis

Another watershed moment for me was reading Structure and Interpretation of Computer Programs. I had been programming for a number of years and had even worked in Clojure for a while. SICP demonstrated to me the beauty of abstraction. It was also the first time I finally understood the power of Lisp macros and “metacircular interpretation.”

After Gabriel’s talk, when I looked at SICP again it was there all along—right there in the foreword by Alan Perlis—“Lisp is for building organisms…” Lisp is for building dynamic systems of organisms. Systems. Living things. Real things doing real stuff.

I have used a REPL connected to a production system to change code. Yes, I have changed code in a running production system. Does that terrify you? Honestly, it kind of terrifies me a little, too! Regardless, sometimes the best way—the only way—to diagnose a particularly nasty bug is to poke a living organism and observe the result. I am more like a doctor diagnosing a patient and less like a detective trying to piece together a sequence of events from the clues left behind.

A stack trace and heap dump are dead. It is far better to poke at a living thing than dissect a lifeless one.

REPL the Ultimate

One common complaint I have heard about Clojure is that you are forced to order the code in your source files. You must define or declare something before you can use it. In the early days of Clojure, Steve Yegge famously complained about this. A firestorm ensued, or at least what seemed like a firestorm to the small, young Clojure community.

Rich responded to Steve in a comment on Hacker News:

“Should Clojure start requiring files and defining semantics for them? (it does not now)”

Wait, what‽ Clojure does have files. You do require them. Has Rich lost his mind?

In this comment Rich explains The Lisp Way. A file of Lisp code is like a transcript of a REPL session. You can literally open a REPL, enter the top-level forms one-by-one, and produce the same effect in that REPL session as you would by requiring the file. The compilation unit of Clojure is not the file, it is the top-level form. The REPL is not this nice debugger bolted on to the side of Clojure, it is Clojure.

Furthermore, the REPL is a tool of the system paradigm, not the language paradigm. Each form you evaluate in the REPL adds another little organism to the larger system of organisms. You can see it in the way that Steve and Rich frame their arguments. Steve mentions C++ as a bad example of a language, and presumably has some good example in mind. He is talking about files and compilation; about the compiler having a broader view of your application than just a form at a REPL.

Rich has in mind a system. Since Common Lisp does what people want (allows a reference to an as yet undefined symbol) it seems possible to do something similar in Clojure. But Rich has thought about it and rejected it, and his argument for not doing it comes from the system paradigm. He does not like how it would affect the REPL experience and the way the system components interact with each other.

One of the hallmarks of incommensurability is using the same terms but meaning different things. Gabriel found this to be true in his experience with incommensurability between the system paradigm and the language paradigm.

In the language paradigm, a class is a static thing. It lives in a file and to change the class you change the file. In a system paradigm, a class is an organism in a running system. The Common Lisp Object System has rules for resolving an instance identity after its class has been modified. Such a thing is simply impossible in the language paradigm.

A Lisp programmer does not define semantics and then say, “hmm… how can I make this work at the REPL?” The REPL is the beginning and the end. REPL the Ultimate. Clojure has what I like to call “REPL semantics.”

When I interview developers for Clojure positions, I pay attention to their view of the REPL. I think someone who reaches for the REPL “gets it” whether consciously or not. They understand that programs are meant to be built incrementally, top-level-form-by-top-level-form. They understand the value of probing a running system.

System Boundaries

Perlis’ organisms are nicely fractal. Pascal is used to construct pyramids which are made up of heavy blocks², not smaller pyramids. Lisp is used to make organisms out of organisms. It’s organisms all the way down. And all the way up.

If immutable data and controlled, explicit state work well for components in memory, would they not work well for a REST endopoint? If referential transparency simplifies reasoning about a call tree of functions, shouldn’t it do the same for reasoning about a constellation of microservices?

If REPL semantics are useful for a single Lisp process, would they also be useful for a distributed system?

I believe that we are building increasingly complex systems, and we will need to simplify, decouple, abstract, and isolate system components as much as possible, but that is not enough.

As we come up against SLAs for 99.99% uptime and zero downtime deploys, we will need to think about how we can gracefully build systems incrementally. What does component A do if component B is down? How does component B communicate to A that it is now using a new data format? How does component A ask B for the old format (because B should continue to support both otherwise the system grinds to a halt)?

Networks go down. Fixes are deployed. New features are rolled out. This is the real world with real things doing real stuff. Thinking of your application as a system—as a living thing—is especially useful when you’re working in a language like Clojure that lends itself to the system paradigm.

Thanks to Vincent Storme, Phil Hagelberg, Rusty Bently, Devin Walters, and Jake McCrary for giving feedback on an earlier draft.

Footnotes: