Label Maker

(2014-ish)

What has been my best moment in IT?

There have been several moments -- how about the overnight datacenter maintenance window when I discovered that I'd bricked the DC's main router redundant pair -- by following the documentation for doing something else, thank you very much -- and THEN discovering that management had let the support contract lapse? Or perhaps the day a customer's core switch started failing and I had to provision a replacement -- including about fifty network spaces, each with their own IPs, VLANs, and access lists, ALL from memory? Or kickstarting a Solaris NIS server over a VPN from another country? Or any of a dozen times where I, as the unix guy, had to diagnose -- AND FIX -- a Windows networking problem in order to prove to the higher-ups that it WASN'T a unix problem? Heck, one of my companies had a door pass made up for me that had my name as "Super xdroop".

Nah, that's all hero administration, when everything's in flames and you walk in and pull a miracle out of your ass. That's part of the job; that's what you signed up for. Praising that is like praising the bus driver for taking your bus fare.

No, I'mma picking something a little more subtle. Something that the vast majority of stakeholders never knew was even wrong.

So it's the 1990s, and I'm riding herd over a flock of unix computers in a big, multiplatform development shop. AIX, Solaris, Tru-64, HP-UX, even... shudder... ultrix... we had it all, and multiple versions of most of them. The product in question is huge, as in a full build from source took a minimum of 4h on the fastest Sunfire E2 computers we had at the time, and over 24h on some of the slowest.

Because insanity like CVS, SVN, and git have not been invented yet, we're doing source code control using perl scripts wrapped around a RCS core. Nasty? No, not really, it actually did some things quite nicely, and it was a step up over the SCCS thing we'd used before. One of the quirks of the sales process was that every time we sold the software, we'd fork the repository at that version. That way each customer could get custom fixes, or a custom collection of fixes, without affecting the versions that the other customers did. The way the repository did this was by using RCS labels. Since multiple labels could point to the same version number, each release could be a label, with that label in each RCS file pointing to a version. Logic in the tool took care of whether or not a new commit to a label caused a version fork in the file, or merely pushed the current branch along.

I say "current branch" because while initially the main release was in the main branch, development got too... alpha at one point and the decision was made to revert the main development line to a more stable starting point. So everything was a branch. And many things were branches of branches of branches.

It also meant that making a fork was easy. You defined your new branch as a child of an existing label, and the fork tool just walked each RCS file looking for the existing label and then stamping a new label on with the same value. It was really fast.

I didn't write that tool, by the way, I was merely a youngling at the time. But in my multi-year-stay I ended up owning the whole system and being responsible for the whole pipeline of systems.

Anyways, time moves on and so do I. But the MSP I work at gets a call because this previous employer has a situation that they think I can help them with.

Said previous employer has been sold a couple of times, and what's left of this project belongs to a dozen or so survivors, most of whom are management. And they decide they want to license the software to another company in Korea or someplace, right? And as part of this, they want me to come in and show them how to build the software, since I apparently built the youngest build node some six or so years previously.

Fortunately my notes are good (always keep notes, kids!) so I'm ready to go.

Only problem is that when I get there, someone's had a really big oopsie with the repository. Most of the RCS files have lost all their labels, and most of what's there is merely conjecture, hacked together with what is probably a really good but totally wrong series of guesses. I corner the most likely suspect, and he confesses to having tried his hand at "fixing" one of the branch repositories for whatever reason is lost to time. Also, none of the dozen or so survivors are sysadmins, so the backups are, as you'd expect, being managed by Murphy and are shit.

Fortunately, this only happened a day or so ago, so we're probably OK. This is because the one thing these guys did do after firing all their sysadmins was to keep anything they didn't understand running. And one of those things was my old workstation, because that was the system that kicked off all the builds every night and checked them for completeness in the morning.

However, one of the OTHER things my workstation did every night was walk the whole respository and record the label values for each label and file, and then keep THAT record under a local source control. So not only could I restore the labels back to what they were very quickly, I could do so for any arbitrary date in the previous six years.

Five minutes of perl, ten minutes of RCS label stamping, and then one careful series of commands to make the edit this guy wanted to make, and we were back in business.

Management, corporate, and the Koreans never knew, and the guilty party bought me lunch. And we agreed never to speak of it again.

I went home and felt really good that day. For all the times I've felt resentment for one of my systems being pulled down in favor of something new and shiny, I think back to that time when I found one of my old systems doing exactly what I'd told it to do six years previously.