What no one has told me about programming

I had another interesting breakthrough yesterday with regard to how I think about programming, or rather creating applications using programming. I’ve learned over the past couple years of creating an application that seemed simple at the outset (a simple number-crunching program!) that the really hard parts of programming are not the parts that people typically write about, and the really challenging parts are things that are obvious, but nobody talks about those things, perhaps because they are so challenging. This brings me to another question about how I should handle those really hard parts.
Yesterday I toyed again with the idea of learning C++, and decided against it yet again. I’d heard that C++ had a number of tools that are good for numerical programs, like vectors, and I recently heard of a new matrix library called Eigen. However, I’ve avoided C++ because I still believe object-oriented programming is one of those bad habits people pick up in programming classes, and it didn’t seem to offer any advantages over C. C is good; I mean C is The Right Thing.

There was still something else nagging me, however. When you read the books with the trivial examples that you could do easily with a pocket calculator, they don’t match up with the way the language is designed. Even books on Haskell don’t seem to be saying as much as they should about what’s really important in designing an application. That was the realization: what you’re supposed to do with a programming language is build an application. The “program” part (that is, the algorithm) is really immaterial. After exploring many programming languages, I have found that with few exceptions there are very few that really differ in their offerings for completing algorithms. You can write the same algorithm in almost any language and have it perform pretty well on most hardware these days. So what’s missing? What are all the manuals full of? Why does every programming language have a preoccupation with strings?

Let me use an analogy: I play the banjo, and took over a year of lessons, read tons of books and have probably spent over 3,000 hours playing and practicing, and even after getting in a really good band, having great people to jam with, and practicing really well, there was still something about playing that was so difficult. I just kept saying “I don’t know what to play,” or “I can’t make the notes fit there!” After I started graduate school and my second son was born, I needed to shift back to listening and if playing, playing a quieter instrument, so I started doing things I’d never done with my banjo using a guitar: playing scales, picking out melodies, and listening very carefully to my favorite guitar players. Listening to Steve Stevens, Jerry Garcia, David Gilmour and Kurt Cobain, I noticed something: these guys don’t play notes, they play phrases.

Why had absolutely no one mentioned playing phrases to me? Was I not listening? Did no one just say “Melodies, counter-melodies, rhythms, etc., i.e. music (dude!) is composed of phrases. You can construct phrases in many ways, but the key is punctuation.” When I learned to play the banjo, I learned the punctuation marks (licks). I learned how to move my fingers, and I learned chord formations. But I never learned the fundamental thing about music is phrasing. After I figured this out my brother told me how a famous drummer sat him down at a workshop and pointed his finger saying “One thing is important: phrasing.” Luckily this was when my brother was fifteen. Since I’m not a pro like him, I can understand why I didn’t get that opportunity, but still come on! This is hugely important. Why did nobody mention it?

And why has nobody mentioned, in any programming book that I’ve ever found that the crucial thing — the hard thing — about designing a program is the user interface. There are books about user interface, certainly, but they are concerned with superficialities of user interface, like what color the frame should be. Who cares? The difficult part is deciding how your program should interact with its user. Eric Raymond does spend a whole chapter on this, but he doesn’t start with it. I’d like to read a book that starts with “You can figure out all that stuff about your algorithms: you have the equations, you have the data structures, you know what it’s going to do; spend time thinking about how a user would get it to do that well.”

So my realization yesterday is that the reason the C standard library is full of string functions, the reason Lisp programmers are so concerned with reading program text and the reason that there are so many programming languages and libraries and plugins is that the really hard part is between the user and the algorithm. My inclination is to say that the simplest interface is best. The simplest interface would be “stick something in and see what comes out.” That’s called Unix. Even in Unix you can’t just do that: you have to mediate somehow between the algorithm in a world of numbers, and the user who lives in a world of text. This is easiest on Unix, but it’s still not easy.

There are other schools of thought: your user interface should be a pane full of buttons and pretty colors to dazzle your user into thinking they’re doing something useful, or a monolithic shell that does everything with the computer. I don’t really buy either of those things, because I know how to use stream editing and Make to tie things together. However, sometimes I need a program that I don’t have to re-run all the time. I would like something in between: something where I can run a simulation, look at the results, then tweak it a little and run it again, then set it up into batch mode to produce a huge pile of results that I can analyze. There’s no reason that all has to be in one huge program, it could be several, but the point is that the algorithm contained in there would be the same for all those steps. There are languages like this, such as R, Octave and Scilab. However, I don’t like programming in any of their languages. Maybe I can come to like it since they make the hard parts easy.

The approach I should take with my next program is “How do I write a language for running a simulation?”