June 19, 2006

The Soul of A New Machine

by psu

I read The Soul of a New Machine for the first time when I was in high school. It is the best book I have ever read about computers. It is one reason I ended up working in software engineering. If you have not read this book, you should go and buy it now, and read it, and then come back here. Ready? OK.

Since you have now read the book, I don't have to tell you that it chronicles the development of a mini-computer by a company called Data General at the end of the 1970s. This machine was built to compete with a machine called the VAX, which was also developed the last years of the 1970s. It happens that I used both of these types of machines when I was in high school in the early 80s and so the story of the development of one of them seemed interesting on the surface. I picked it up in paperback, and read it in bed in two or three sittings while trying to finish my applications to college.

Back then, the stories about engineers on the project, and the technical aspects of the machine they were working on really pulled me in. The company politics and the maneuverings of business seemed to me to be the filler between the high points of watching this machine actually come to life. I soaked up the stories of late night microcoding, the architectural design, and the months spent debugging the machine.

To this day I am still amazed that a journalist, presumably with a limited technical background, could have such an obvious understanding of the layers of abstraction and complexity in the system that he was describing. One early section of the book walks the reader down through the machine from a single line of BASIC that divides two numbers into the CPU to a divide instruction and further down into the micro-architecture, where there is another software layer to tell the hardware how to actually divide two floating point numbers. Another section of the book contains the single best description of how a machine that uses a virtual address space fetches and executes its next instruction that I have ever read. It's all there, the address translation, the instruction cache, the page fault handler, instruction decode, the microcode sequencer, all the way down to the bare hardware. He even talks about what might happen if the machine recursively page faults forever. No computer architecture book ever explained this basic pipeline in a way that is more understandable. Every new student of computer or software engineering should be required to read the book just for this reason.

I also love The Case of the Missing Nand Gate. I have often wondered how to teach people to debug complicated systems, be they hardware or software. I still don't have an answer to that question, but this chapter of the book would be part of the formula. After yet another beautiful tutorial on how the instruction and data caches of the machine are specified to work, the chapter launches into a case study in finding, isolating, and finally fixing an intermittent bug of the worst kind. Finally, in the end, when the engineers have figured out how to make the machine fail, they capture a picture of the error in one of their logic analyzers, and immediately the fix is clear. What I found amazing about this section in high school was the detective story aspect of the narrative. What I find amazing about the chapter now is how clearly it captures the process of observing a system that is failing and how you trick the machine into failing while you collect the proof. Twenty five years after the book was written, this process remains unchanged.

What has changed in that time is what the book means to me. Having worked in software engineering, both academically and commercially for many of the twenty or so years since I read this book, I now realize how much more there is to the story of this machine. It wasn't apparent to my high school self, who saw the engineers as the heroes and the company above them as an often malevolent force, working to crush their spirit and creativity and take credit for their hard work. I think this is the way young, naive, and/or inexperienced engineers see the world.

I've come to see the book as more concerned with how the project was managed than anything else. My younger self never really realized that Tom West is the hero of the book. Not even the engineers on the team seem to realize this until the very end. In a coda to the main story, one of the engineers finds a specification for a trivial piece of hardware that allows the machine to interface to some kind of third party peripheral. He realizes that no one on the core engineering team could have possibly designed the plug or written the specification. There are hundreds of details like this in the final machine and the book credits West with collecting the support needed from the rest of the company to make sure all of these little problems were found and solved. The book accurately observes that while the central act of creation is in the hands of the engineers, this act would be wasted without the coordinated efforts of dozens of other people working on behalf of the project in other parts of the company. In other words, the romantic myth of the lone engineer taking over the world in his basement is, for the most part, just a romantic myth. Products that ship are built by teams of people led from a shared vision, not a single genius working alone.

Through all of my re-readings of the book, Kidder's writing remains wonderful. He manages to package up all these layers of narrative and psychology while keeping the story moving forward. He captures exactly what I find fascinating about working on computers. Moreover, the book perfectly expresses the pain and torture of managing engineers: needing to trust the team, needing to shield the team from anything that is external to their core task, and, perhaps most importantly, needing to convince the team that the task they have been given is important and groundbreaking in some concrete way, so that they will "sign up" for the job.

I don't really have anything left to say that Kidder doesn't say better. You should go and buy it now, and read it. And then read it again.

Posted by psu at June 19, 2006 06:31 PM | Bookmark This

I, too, read Soul of A New Machine around the time it came out. I didn't go on to become a programmer, but I've dealt extensively with technology evaluation in my career.

It was Kidder's book, more than any other, that helped foster the insight and confidence that, with the right approach, I could understand a complex process or technology at a level that allowed for (mostly) intelligent decision making. It also encouraged a distate for the oversimplification and spoon-feeding that is common where technology and investment meet.

That may sound a bit soppy on my part, but honestly, I've never read anything else that so clearly caputures what is at stake in a technological venture, how the thing itself works at a basic level, and how it is brought into being.

I'm off to find another copy and read it again and hopefully confirm that my memory is correct.

Posted by Will C at June 19, 2006 11:16 PM

I was already out of college when "Soul" was printed. Still it was amusing. I was doing some work with people at DEC at the time. Everybody thought DG was done, after maybe a dead cat bounce. The funny thing was, I thought DEC was done too, but for different reasons. It just toke another decade or so before that played out.

Wired had a 20th anniversary article that is pretty amusing.

There was even a movie option taken out on the book. Ha! That would have made rivetting cinema.

Posted by Amos the Poker Cat at June 22, 2006 07:03 PM

Amazon lists that 173 books cite "Soul". Pretty impressive. I wonder what the top cited book is?

Posted by Amos the Poker Cat at June 22, 2006 07:28 PM

As a software developer since the early 70's, I read this book when soon after it came out, and was fairly impressed.

I was even more impressed when I finally met some of the DG engineers from that era who had fled to a startup in the mid-80's in Branford, CT, a Yale spin-off called Multiflow (we built mini-supercomputers based on the VLIW research done at Yale in the early 80's). One of them, Dave Papworth, was a hardware/software design/debugging genuis so far off the scale (I've known a lot of them, including Bill Gates) that the scales broke.

He and Bob Colwell ended up at Intel when Multiflow imploded in the very early 90's (everyone wanted personal machines, so Sun and SGI won that battle, even if our machines were faster--never bet against the desktop), and they were 2 of the 3-4 key people who designed the Pentium 5 and 6 up in Oregon. See Bob's new book "The Pentium Chronicles: The People, Passion, and Politics Behind Intel's Landmark Chips" for details.

Posted by Chris Ryland at June 24, 2006 06:00 PM

Please help support Tea Leaves by visiting our sponsors.

November October September August July June May April March February January

December November October September August July June May April March February January

December November October September August July June May April March February January