The Rise of the Stupid Software

[andrewducker]

There's an ancient, but still thoughtful and correct, article called The Rise of the Stupid Networks which details why stupid networks are better than smart ones.

It basically says:

If you build intelligence into your networks you are optimising them for a single purpose at the expense of their flexibility. Any gains you make from this optimisation will be overtaken by general gains in the medium term, leaving you behind in the long term as unexpected uses are found for more general networks.

The examples in the article are the phone networks (optimised for voice) and the data networks (not as good with voice at the time, nowadays just as good at voice and far more useful in ways that were never originally imagined), but I realised that it's extendable beyond what we normally think of as networks.

I was reading Clay Shirky's article Given enough eyeballs are features shallow too? which says that if enough people use a piece of software, one of them will have an amazing idea that will make the software indispensable, and that thinking of the idea is actually frequently harder than implementing it (true in many situations - take Bayesian filtering as an example, someone published an article on using Bayesian filtering to spot spam, 6 months later there are at least a dozen different completed projects which implement this).

Putting these two ideas together, I realised that what we need is stupid software - software that's incredible simple, that basically gets the data from A to B, but allows other people to slot their ideas on top of it. Some of the first programs to do this were games - Doom had editors that allowed people to modify some parts of it, but Quake allowed people to completely modify the game beyond recognition. Nowadays many erstwhile game companies sell what are basically engines and tech demos (see Quake 3 for a perfect example) for others to build masterpieces on top of.

But what really clinched it for me was the decision by the Mozilla creators to take their massive, monobloc web-browser/mail client/chat program/kitchen sink and split it into much smaller programs that focus on a single task. Not only that, but they simplifed the interfaces significantly and hid everything possible from the users except for what they needed to see. Which isn't the important bit. The important bit is that they made ie supremely easy to write extensions. This means that you've got a stupid program (actually, a very smart program, but it needs to be to pretend to be as basic as web browsers seem to be) that acts as a lowest common denominator for anyone to add their cool idea on top of.

In these days of increasingly open source software, the programs that get the most support will be the ones that allow programmers to scratch their itch the easiest, the ones that are designed from the ground to allow you to build on them. The ones which are, if I may, that are the stupidest.

Comments

[kpollock.livejournal.com]

Maintenance nightmare!

I agree that the ideas are way harder than the implementation - I am good at the latter, but wish I was good at the former too.

[andrewducker]

It's not a maintenance nightmare at all.

It's all modular. So, as long as the underlying application works, all the external developer has to test is their extension to it.

[kpollock.livejournal.com]

And how often has this sort of thing been promised and what like is it when actually delivered? Same as anything else. the modules either don't work as advertised, have outright bugs which nobody seems to fix (or want to) or don't expose what you need.

I've got a lot of experience with building on top of other apps that let you build extensions and one or other or both of the above problems always arise at some point and you end up having to compromise your app heavily (whilst cursing the programmers).

You have a better chance of getting it right if you do it yourself (or at least I do).

Not to say that there isn't soem solid stuff that can be built upon, just that it is very few and far between and you still get the 'not exposed though I damn well know it's in there' problem. Flexibilty comes at the expense of conceptual (and often) technical complexity - which can lead you back to doing it yourself.

[andrewducker]

It can be like that. But take, for instance, MS Office. One of the reasons it continues to be successful is that it allows people to build things on top of it (horrible things, to be sure).

Winamp is such a successful music player because it is so modifiable. There are hundreds of mods for the Quake series, not to mention Unreal engine and numerous other games.

Mozilla Firebird's not even hit version 1 yet and it has over 80 extensions.

Most of the professional graphics/rendering/animation packages are sold on the basis of the plugins that are written for them.

Certainly, there can be problems, but I think that in general it's the extensible programs that last longer.

[kpollock.livejournal.com]

Oh they are maybe better than non-extensible ones, but often shit sells irregardless of actual quality or utility.

Pinning your hopes on basing your own software on such things is taking the (often annoying long) slip road onto the fast lane to disappointment.

With most 'extensible' apps, if you want to go beyond minor tweaking and gimmicks and customising then it's a hard hard thing to do.

Example: I built on top of InstallShield and it was a nasty lonely business (seems nearly nobody does it. It's not because of better research because there isn't the info to suggest anyone has done such research). It epitomised the cycle in that it seems to be a quicker way to create what I wanted than learining the WindowsInstaller database + associated gubbins) but due to bugs and crap coding and non-exposure it was actually became a pain in the butt and my app was crippled in several (fairly important) ways that it woudlnt' have been if the damn doftware just worked. I ended up telling their support what was wrong with their program (e.g. default names that were too big for the database field to which they were written. Oh and tbere was no way to name it yourself, you were stuck with the default)

Office is the same.

Counterexample: the Neverwinter Nights programming interface is pretty good to work with, according to Sean, and seems very powerful for creating games of that genre. I haven't used it.

Just about any 3rd party tool/control/module etc. has been the same - I come smacking hard up against the limitations/errors every time.

Maybe I just push the edges to hard too often. I have to come to this conclusion when I search the net and come up with nobody (or maybe a couple of people at most) trying to do what I am doing. Or maybe I just try to do weird things, I don't know.

Most software is written by people, and even the best of us cock it up frequently. If you can't fix it and they won't fix it (or at least not quickly), then you really would have been better off writing it yourself.

[andrewducker]

Oh, a lot of software is badly written, full of bugs and badly extensible (Installshield is renowned for being an appalling piece of shit).

Office is very powerful in some ways, but could be a lot better.

I've hit numerous bugs when using external components, but they've also saved me a lot of time.

I think the problems happen when people hack extensibility on top of a program - building it in in the first place seems like a uch better way to work.

[kpollock.livejournal.com]

Too true.

I try to do that with my stuff, but the appropriateness (in terms of effort/time) is variable. In my curetn situation, overly generic, flexible extensible stuff is frequently OTT, and usually makes for code it takes other people too long to understand.

I have a massive over-engineering tendancy, which I have to curb.

[andrewducker]

Yeah, I was helped by reading some of the Extreme Programming stuff. Which basically said "Write it to work _now_, chances are the extra functionality will never be needed. For the 5% of times that you need to make it more general, do it at the time it's needed, and you'll still save time overall."

[kpollock.livejournal.com]

Which kind of contradicts the "design extensibility in" that we agreed earlier was important. Allow for it, but don't actually do it, I suppose is the best way to put it.

I think that the best thing that I have read about extreme programming/analysis etc. was about how programmers/architects are at different levels of sophistication and that methods should be/need to be different for each. More casual the more sophsticated, absically. For a beginner (or an unsophisticated programmer/whatever) casual methods lead to LESS productivity. I can't recall the name of the book (and amazon's ability to look at past orders is down right now!). Some of the ideas were OK, some of them unworkable, but that point was great. Exactly what I have been thinking for years (having wanted and tried and seen the successes and failures of more rigid development systems).

[andrewducker]

Aah, Extreme programming says "write to fit the spec, but only just." If the spec involves the ability to be extended, then you put that in.

Extreme programming looked great, so long as you were doing RAD. Great for user interfaces and suchlike, not so good for batch processes.

[kpollock.livejournal.com]

Agile system development or something like that, was the book. Agile is about moderate extreme programming if that makes any sense.

[andrewducker]

I've heard extreme programming referred to as a subset of agile programming before.

I must say I like the idea of pair programming. So long as I wasn't working with idiots I can see it working well.

[diffrentcolours]

The earliest example of stupid software is the original UNIX command line utilities. They each take data from STDIN and stick it on STDOUT, and you can join them together with pipes. So you get something like:

grep 'foo' file.txt | sort | head

Which gets the first 10 lines of file.txt containing "foo", sorted alphabetically. Each program is incredibly stupid, but together they're doing something fairly complex.

[andrewducker]

Yeah, CLIs are great for that kind of thing.

I prefer object/component based interfaces for programming, myself, but they really are the archetypal stupid/extensible program.