The Wheel of Reinvention. Or, the Eight Steps to get to Step One.

The wheel of reinvention is a term that I most closely associate with the history of computer graphics, but the idea is pretty universal.  Technology is cyclical, no matter the specific field. All software needs to be a bit more flexible than you originally thought, and it eventually spawns more software because the solution to the problem of a computer program you don’t like is almost always another computer program that is even less carefully constructed.

Technology is Cyclical from 30 Rock
Maybe Liz Lemon’s boyfriend on 30 Rock was right, after all.

This is basically my take on the Configuration Complexity Clock. I’m not the first person to write about the topic. And since it’s practically a Natural law, like a gravitational pull, I certainly won’t be the last to deal with it, notice it, or write about it. But it is something that popped up on my radar again recently, so I wanted to put it in my own words. Let’s look at the whole cycle of terrible crippling success after terrible success…

Step 1.  Success

Success!  You’ve written a cool new program that does exactly what it’s supposed to do!

Neat.  Before you started, you talked to all the stakeholders to understand the scope of the project. You got copious assurances that it would only ever need to do this one thing, so you made a straightforward implementation, and it works perfectly.

Step 2.  Flexibility

More Success!  Your app has caught on, and more people are using it, and new use cases have arisen.

The assurances you got were from people who lacked imagination.  Or maybe, they were from people that have been replaced.  The company merged with another, or something else has changed and now the assurances you got that your code would not need to change are invalid.  So, you add a little flexibility to your application — for example, the ability to invoke it with some new command line parameters that weren’t part of the original design.  Now, you can override the database that it tries to connect to when you are testing locally.

Step 3.  Configurability

Even more Success!  Your little program has caught on and people are using it for more serious stuff.

It’s fine to invoke your program with a command line parameter every now and then, but now you need a way to set up a particular machine as the permanent development environment, and another machine as the prod environment so the two application instances always talk to different database instances.  You can’t depend on other developers always remembering to invoke it with one of the flexible but increasingly verbose and obscure command line parameters.  But you really only have one or two or just a few things to configure. You stash a text file somewhere with a few lines and you do a little string munging to pick out the relevant values.  You don’t even need to add a new library as a dependency or anything that would complicate the build.

Step 4.  Standardize config file format

Yet further Success!  Or, uh…  Success?  Your proposal for adding a simple config file proved so popular that people have been extending the fields and options that can be set in the various environments.

Now the config files are almost incomprehensible. You wanted to avoid adding a “complicated” dependency early on, so each new config option has come with a new bit of ad-hoc string munging code to parse it. QwobbulateTheDatabase has to be set to “True” or “False” but fremulateTheFiles has to be set to 1 or “off” because the fields don’t share parser code.  The names of the Fremulate options are case insensitive, but the Qwobbulate options have to be in strict PascalCase because of the preferences of a developer who no longer works here.  Obviously this is unsustainable, and everybody knows that you need to abandon the ad-hoc parser and just add support for JSON.  Or maybe it is actually XML, or YAML that will supposedly solve all of your problems, depending on the specific shop.  So, you get to rip out a bunch of custom code, and replace it with a few calls into a well tested JSON library.  Everybody knows how to write it properly without memorizing wacky rules.

Step 5.  De-standardize the config file format to make it more flexible

Success!  Dammit, the continued success of this program is going to kill you.

The config files have flourished in the age of JSON (or whatever), and keeping them in sync between prod and dev has now become a full time job for somebody.  He used to be a developer but now he’s basically a manual diff driver that spends every day bringing change tickets to the review board to keep the two environments in sync.
Clearly you need some sort of templates or expressions to extend the JSON, so you can write a single config file that is valid in both environments.  Perhaps you’ll add a little bit of custom written front end parse code on top of the JSON, so you can add stuff like // style comments to make it easier to keep track of things, and a few blocks that say "dbserver" : "${ENVIRONMENT}" or even a few things like if environment == "dev" {foo()} if somebody on the team is feeling especially ambitious.

Step 6.  Embed a programming language (by accident)

Whelp, it’s another success, you poor bastard.  Your app has flourished.  It has more users.  Your idea to add a simple template or expression language has succeeded beyond your wildest dreams and/or nightmares, and it is now Turing Complete.

Of course, no off-the-shelf JSON linters can handle the new format, but it seems worth it because you’ve got 99% a well known format, and the flexibility to cover all your environments with one config file.  Saltstack is a well known example of this sort of Jinja on YAML idiom.

    {% if grains['os'] == 'FreeBSD' %}
    - name: /etc/motd
    {% elif grains['os'] == 'Debian' %}
    - name: /etc/motd.tail
    {% endif %}
    - source: salt://motd

It’s YAML Jim, but not as we know it.

Step 7.  Embed a programming language (intentionally)

Sigh. Conratulations. The last step was another great success that puts you one step closer to your grave.

The natural consequence of reaching this point is that you either extend your bespoke accidental programming language into one that you maintain intentionally. Or you use something like Python as the programming language and just adopt the idiom of writing Python programs as the config file for your programs.

7.1 – The CMake Way

CMake is my least favorite example of a “simple text file” that accidentally became a programming language. (And an example of what happens when you skip right over Step 4.) CMake scripts are still vestigially known as “CMakeLists.txt” because the original intention was that it would literally be a text file with a list of the C++ source files that needed to be compiled. A simple “Step 3” configuration file for your build system. At this point, after over 20 years of iterations, so much depends on it that the flexibility can’t be abandoned, so the maintainers have been forced to treat it as a programming language even though it only became one by accident.

For example, this Qt blog post actually recommends using cmake -P as a portable cross platform shell for general purpose scripting. In that example, it’s only used to download the actual QMake build system generator because CMake is no longer even being used for the build. CMake is so far removed from invoking a compiler that it is just a build system generator downloader.

As a result of this kind of thinking, CMake has moved further away from being a config file, and into supporting things like self-modifying code and arbitrary functional programming where CMake can eval CMake at CMake-runtime which used to just be considered a configure step for a build. The thing that started life as a config file naturally also supports its own config files to modify the behavior of what are now CMake programs with things like CMakePresets.json and CMakeUserPresets.json.

cmake_language(EVAL CODE "
  if (${condition})
    message(STATUS TRUE)
    message(STATUS FALSE)

I swear to you, that kind of metaprogramming actually started life as a text file that was just a list of .cpp files that needed to be compiled. It was a slow and unplanned evolution. But now it has grown into a full-fledged programming language. Even if it is one that makes you want to peel off your own corneas so you don’t have to look at it.

7.2 – The Buildbot Way

The other option is to go the route of something like Buildbot. One doesn’t so much install a program called Buildbot and configure it to set up a CI environment, as one installs the Buildbot library and writes their own bespoke CI software in Python that happens to call into Buildbot services.

def has_C_files(change):
    for name in change.files:
        if name.endswith(".c"):
            return True
    return False

That example code is from the Buildbot docs, but it’s just valid Python. Could have come from any Python tutorial. Rather than growing a customer language, they went the route of using a language off the shelf. Consequently, totally normal tools like pylint work just fine with Buildbot configs, which is more than you can say about Saltstack’s YAML.

Something like Maya is similar in some ways. Maya is a 3D animation program written in C++, but designed as an open architecture so that studios can modify the UI and functionality with plugins and scripts. For many years, it was a “Step 6” application with an intentional custom programming language called MEL. Some years later, they followed the general trend in the VFX industry and embedded Python.

Nuke is compositing software written in C++. Nuke started out with embedded TCL. TCL is a standard programming language, but obscure enough to most users that it was effectively Nuke-specific. Nuke also added an embedded Python runtime, effectively moving from Step 6 to Step 7. At startup, Nuke loads “config files” like and that are just Python scripts that are allowed to do arbitrary things in arbitrary ways, and behave completely differently depending on where and how they are run.

Step 8.  Success!

Finally, when you get to step 8 you are done and you can sleep. You made it to the end. You wrote a Python script that handles all of the flexibility you need in your application. Right?

A rustic looking “cozy sleeping porch.” The log cabin aesthetic makes it look like you won’t have to deal with any computer problems here. You can just sleep and be cozy. Of course, you’ll never actually get to go here. This place is a lie. Step 8 is a lie. Finishing is a lie. Success is a lie.

Now you’ve got something like a Python script for Maya, or a Buildbot config that does exactly what you need, and nothing more. Actually, what have you got? — A simple computer program written in an existing programming language, with not too much variation in what it does… Hm, that was basically the definition of step one in this whole project. If you’ve made it to Step 8, you’ve found yourself back at step one.

Unfortunately, you have to think of this 8 step process as being in Octal rather than Decimal. You rush forward. Again and again, trying to get to step++ to keep up with the needs of the users or the needs of the business. You try to get clarity on scope. You try not to boil the ocean. But no matter how hard you work, you always wind up on something more like (step % 8). You never actually get to rest at step 8 because step 8 is always step 1 of the next thing. The config file is now a computer program that needs to be maintained and expanded. If I was leaning harder into the modulus metaphor, I guess I actually needed to wedge a step 0 into the start of the blog post, but I think you get the broader point anyway.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s