The move of the blog contents took 5 minutes.

Releasing the sub-domain blog.dominicsayers.com from the clutches of wordpress.com and getting SiteGround to set it up took 48 hours.

Lessons learned:

1. If you’re thinking of choosing a new hosting provider, try to get a look at the hosting control panel first. I’ve now had three faults with SiteGround that were simply because their tools don’t work.

2. I chose to keep my Name Server management away from SiteGround because their tools (a) didn’t work and (b) didn’t let me view or edit my DNS records explicitly. Don’t nanny me if I don’t want to be nannied. Because my name servers are not under their control, this broke some of their other tools. How about testing this stuff before releasing it, people?

3. SiteGround tech support are helpful and responsive.

Other things I’ve noticed: some of the WordPress plug-in writers have stopped maintaining their plug-ins. Is WordPress a dying platform? Is blogging a dying art?

Internet Explorer 6 was a major advance on its predecessors as an application development platform. What’s more it was effectively ubiquitous – Microsoft gave it away with Windows and everybody had Windows.

In the corporate environment, this meant that IT departments could write client applications for the browser instead of using Visual Basic or worse, Java. These browser-based applications needed no roll-out programme and no desktop provisioning – both expensive headaches.

Whoopee! said corporate IT departments, and rushed to develop applications for the new platform.

Roll the clock forward to today and the folly of that approach is clear. Corporates are now stuck using those same client applications because either they can’t afford to develop new ones, or they bought a third-party application that is still business-critical, or the developers have left the company and they simply dare not touch the code. And those applications only work on Internet Explorer 6 because it turned out it wasn’t a de facto standard after all, it was a proprietary cul de sac.

So I sit every day at a desk in a bank, trying to use Internet Explorer 6 in a world where most people have given up trying to retain compatibility with this out-dated platform. And around the world there are millions like me whose employers or clients are saddled with this dinosaur because of an expensive mistake made years ago.

Why do I believe we are in danger of making the same mistake again?

The number of applications developed for the iPhone is astonishing. People develop for this platform because it is better than its predecessors. And although it is not quite ubiquitous, it certainly has a market share that makes the potential audience for an iPhone application very attractive indeed.

This has not escaped the notice of corporate IT departments. I imagine client applications for the company CRM system are being developed right now for the mobile salesforce of thousands of companies.

And so history repeats itself. The iPhone is a proprietary platform whose future is not yet clear. It could set a de facto standard for future mobile platforms. Maybe. I think it more likely that Android and Windows Phone 7 will go their own way. And corporates will have bought into applications for a platform that will go away before the business requirement does.

By all means buy or write applications for the iPhone, but please factor in the cost of replacing those applications in a few years. I don’t want to be carrying round a five-year-old iPhone in 2015 just because my company depends on an application that only runs on that old thing.

Dominic,

I’m sorry if this email is, in any way (from the subject down to the signature) offensive, bothering, time wasting or anything like that to you. The sole purpose of this email is to better understand some things about your kind of career/profession, which is also mine. Furthermore, I’m not asking for anything besides information — and from your site I infer that you like to share your thoughts, so…

First, let me introduce myself. I’m a consultant working for a multi-national company. My specialty is System Architecture, and while the title is vague, what I actually do is design systems through the use of UML diagrams, a lot of documenting, etc. Usually I spend even more time implementing stuff, from C/C++ to bash shells to C# to COBOL even. I’m very fond of technology, and I chose my career when I was very young — about 10 years old. Back then my desire was to work with videogames (like most children with such love for gadgets and computers, I guess) but since I live in Brazil, I was forced to abandon that path some years ago.

I stumbled upon your website when looking for the maximum theoretical limit for the length of an email address. I needed that piece of information to define an entity in my database, and I wanted a number that was right from theory. When reading the article on the site, however, something struck me as odd. Your profile, on the left portion of the page, read “IT director”. But I was reading a purely technical article! That’s incredible. Here, both at my company and, as far as I can tell, my country, techies doesn’t go too far up companies hierarchies. So I decided that you were not, in fact, what I call “techie”. You are probably more, a guy with both systems expertise and managing skills. Still, such a profile doesn’t make much sense to me.

In my company, after as little as 5 years, you can get a managing position. And when that happens, it’s no longer expected that the worker learns how to implement specific things. In fact, the only thing people seem to be able to do afterwards is learn what a particular technology is capable of. For example, managers here know about SOA, about .Net, about ESBs, about Clouds. They know what they are and how they can help business. However, they don’t know how those stuff works. You, however, seems to know deeply at least about PHP, a good deal about REGEXes, etc. These skills, if not used, are forgotten, so I’m guessing you have a tight connection with technology. I’m sure you don’t look at source codes or solve complex math problems for your analysts, but I think you get in touch with them and probably study deep technology during breaks.

I assumed a lot of stuff up there but, if I’m right about most of it, my question is, how do you do it? How do you balance being a good Technologist with being a manager? Is there some kind of synergy between them? Does your bosses took your tech skills into knowledge as you progressed through your career? Or maybe I’m completly wrong and you don’t manage people, being just the big fish when it comes to technology (highly improbable, but still…)…?

I’m know that, from what I’ve written there’s a good chance that you take me for a fool. These questions have been storming my mind for some months now, and I’ve always wanted to ask someone with more time in the market than myself…

Thanks a lot for reading,

X

Some things I took away:

1. Joel’s approach of talking about a real-life incident is exactly the strategy I’m using to implement Problem Management at my organisation. Nobody wants to sit through Powerpoint slides with process flow charts on them, but get them talking about a real problem and they are hooked. Thanks for the confirmation that my approach is not totally mad.

2. He talks about root cause as the process by which you decide where to target your long-term solution. This is exactly the pragmatic approach to root cause analysis I was trying to describe in my last post.

I wasn’t aware of Sakichi Toyoda’s Five Whys maxim, but it makes some sort of sense. The first couple of questions you ask are going to be about the immediate symptoms of a problem. Beyond 5 questions and you are in danger of looking too deeply at the mysteries of the universe. You could well end up trying to implement too general a solution, at great expense, which fixes problems you don’t actually have.

So my amended principle is this: a root cause is that which we can alter to effect a permanent fix to a problem such that the solution addresses more than just the immediate symptoms of the problem.

The important points again are that the solution domain is limited to the space we can directly affect ourselves and that the solution should be a permanent or long-term fix, not a palliative for the symptoms.

ITIL doesn’t define “root cause” in any useful way. The official ITIL definition is “the underlying or original cause of an Incident or Problem” (ITIL v3 glossary).

CMMI is more helpful: “A root cause is a source of a defect such that if it is removed, the defect is decreased or removed.” (CMMI v1.1, Causal Analysis and Resolution process area).

There are two effective parts to a useful definition: a root cause is that which we can alter to effect a permanent fix to a problem. So the important points are (i) addressing the root cause fixes the problem and (ii) it is something we can directly affect.

Examples:

1.       Database fails because it is full. The root cause is that we failed to monitor its usage effectively to take preventative action in time. So the action we take to fix the Incident (increase the database size) is different to the action we take to fix the Problem (start monitoring capacity usage effectively).

2.       Database fails because a virus infects the database software. Here we are not interested in the coding error that allowed the virus to strike, we are only interested in addressing our technology choice or patching regime. In other words, the root cause for the vendor (software bug) is different to the root cause for us (technology management).

Wikipedia is very helpful on this and has good articles on root cause and root cause analysis. The temptation is to get too philosophical about causation when a useful definition is circumscribed by your own sphere of influence.

Unless you’re a TV presenter, it’s quite easy to understand that risk is nearly always present but is mostly manageable. Often this is inescapable – no amount of effort will reduce the risk to zero. The question is, should we ever try? Should zero risk be the goal of any undertaking?

It depends, of course. You will struggle to make a business profit without taking risks. That’s accepted. But what if you are developing the control system for a passenger aircraft? What is the acceptable risk in that undertaking? One fault in your output could have fatal consequences. Should you pursue zero risk in that sort of project? Well, aircraft control systems are complex and there is a limit to how thoroughly they can be tested (although that limit is quite high). You can take action to ensure that many sources of error are eliminated systematically, but I don’t believe you can ever get to 100% certainty that an aircraft control system is safe. At some point you have to install your system on a plane and watch it take off.

The things you can do to make that experience less nerve-racking are ordinary bits of good engineering practice. Lessons learned from centuries of building stuff from wood, stone, iron, steel, concrete and glass can be translated into software projects fairly seamlessly. Transparent project management, repeatable processes, eliminating unnecessary manual tasks, identifying valuable metrics, keep it as simple as possible, etc. The important point is to understand where the risk arises and mitigate it whenever it is sensible to do so.

And that’s the point I’ve been trying to get to. There are many sources of risk and not all are equal. If a risk can be mitigated cost effectively then you should do it. The key word is “cost-effective” and the trick is to get the cost-benefit sums right. This can lead you to some surprising results. The point my colleague was making was that if you have too few errors after a code release, you may be spending too much on mitigating your project risks. You could have delivered the release earlier or more cheaply. How do you compare the value of expediting the release with the downside of delivering buggy software?

The savings are relatively clear: if you deliver your software earlier or with fewer resources then there are direct and measurable cost savings. The other factor is opportunity cost – the sooner you deliver your software, the sooner it can start earning money for you or your customer. That loss of revenue (or loss of downstream cost savings) can be very significant and is certainly a factor in your calculations.

How do you measure the downside? This is less clear because the errors are not predictable. One disastrous error could wipe out all the supposed benefits of the entire project. You could analyse the maximum loss from a disastrous error then try to estimate how likely it is to occur. For a trading system this should be doable – what is the effect of mispricing a trade by an order of magnitude? The maximum loss could be terrifying, but you can take steps to reduce the likelihood of it happening: (i) release code frequently so each release is only an incremental change from the previous one, (ii) pilot your new version with a restricted set of expert users, (iii) build sanity checks into the software to raise an alert when an unlikely event happens.

I guess you are comparing the direct savings of your aggressive delivery schedule with the premium you would pay to insure yourself against the loss due to delivering bad software. The insurance premium will be lower if you can make the event less likely. You should take the risk if the savings exceed the premium. That’s about as clear as I can make it.

So anything you can do to mitigate risk with only a small cost, you should do. Definitely. Errors that arise from easily avoidable risk are just plain stupid and unprofessional. Doing things manually that you could easily automate, writing code in a bad style, using new technology just because it’s new – these are commonplace sources of error which can be eliminated by adopting simple hygiene measures.

On the other hand, taking expensive traders off a desk to spend time testing your new software in a test environment may not be such a good investment. It prevents them earning money (so it is not popular with them) and you are unlikely to get their full attention. So long as they understand and accept the risk of using software they have not tested personally then that is a business decision for them to make. This is an acceptable, managed risk.

So we get used to this picture of risk being all or nothing, black or white, safe or unsafe. Look at this report celebrating 30 years of the UK’s Health & Safety Executive (HSE):

“HSC’s annual report for 1977/78 states: ‘Our overriding concern is… to stimulate awareness of the risks and encourage the joint participation of workers and management in efforts to eliminate them.’ In 2004, the mission for HSC and HSE is to work with LAs ‘to protect people’s health and safety by ensuring that risks in the changing workplace are properly controlled’. The style may be different and the message broader but the core objective is essentially the same.”

Really? Is eliminating risk really the same as controlling it? I would say absolutely not, and the 2004 mission statement is a huge improvement on the 1977 version. The fact that the HSE’s own self-congratulatory pamphlet fails to point out the difference is an assumption by them that their target audience can’t understand the difference.

So we are led to a situation where an acceptance of risk is regarded as a Bad Thing. Gamblers and traders and other disreputable characters embrace risk. If the man in the street contemplates taking a financial risk, the regulator insists on apocalyptic warnings being read to him. If a popular investment turns out not to have been such a good idea after all, the newspapers blame the financial services industry or the government or the company selling the instrument, never giving the actual small investors credit for knowing what risk they were accepting when they bought it.

I think it’s a damaging mindset and it seems to be rooted in 1970s thinking: that’s when the HSE was invented, that’s when comprehensive schools started eliminating competition from sports periods, that’s when licenses became necessary for bingo and gaming machines. Hopefully that sort of thinking has had its day and we can start taking a more mature attitude to everyday risk and the way we communicate it.

I’m reminded of a scientist I once heard on the radio, who perceptively complained that although arts programmes are allowed to assume some prior knowledge on the part of the viewer or listener, science programmes have to explain everything from scratch every time. In other words, you can talk about Baudrillard as if everybody’s read him but woe betide you if you mention a probability distribution without explaining in laborious detail what it is. Go ahead and quote Descartes on philosophy but don’t dare mention cartesian geometry.

Anyway, I was trying to get to a point where I could start writing about risk management of projects but I seem to have come to a natural break anyway. More later.

1. The Tipping Point: How Little Things Can Make a Big Difference, Malcolm Gladwell (job done)
2. Blink: The Power of Thinking Without Thinking, Malcolm Gladwell
3. The Wisdom of Crowds, James Surowiecki
4. The Cathedral & the Bazaar, Eric S. Raymond
5. The Mythical Man Month, Frederick P. Brooks (purchased but not opened yet)
6. Liar’s Poker, Michael Lewis (now finished. Yes, good stories but not sure what it signifies today. Enjoyable nonetheless)
7. The Long Tail: How Endless Choice Is Creating Unlimited Demand, Chris Anderson
8. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail, Clayton Christensen
9. The Elegant Solution: Toyota’s Formula for Mastering Innovation, Matthew May

Oh dear, the list is getting longer faster than I can read. To add to my woes Mike Masnik has now published the bibliography for his collection of essays on the economics of ideas and content. I suspect that I should have read more of his list than I actually have.

So a way of making the process of decommissioning an IT system more deterministic would be a good thing. Before I start trying to turn a system off I’d like to know what my chances of succeeding are.

Stakeholders

One thing I think you need to establish at the start is who the stakeholders are. For systems that have been around for a long time this can be surprisingly hard. These systems are often part of the furniture to the extent that users don’t even realise they are using the system, it’s just how they naturally do their jobs. An old system may have no formal governance because it was introduced at a time when the control environment was not as far-reaching as it is today. All this can make it hard to find out who is affected by the decision to decommission the system.

What you would have ideally is a clear ownership of the system by the business, with a named owner and some sort of representation from each part of the business that uses the system. From the technical side you would have a named IT owner who understands the dependencies that other systems have on your target system.

If these things don’t exist for your target system then you are probably going to have to invent them. Because the first thing you need is a statement of the system’s functional value from its users – if you can’t get that then you should probably just turn it off anyway and see who screams. Armed with a statement of value and an analysis of functional dependencies from your IT owner, you at least know what work needs to take place to make conditions acceptable to turn off the system.

The needs of the many outweigh the needs of the few

A timeless story of innovation is that a functional gap has been plugged by a tactical system. The system is highly specific to the functional niche it inhabits and over time it comes to do everything the users want and they have learned to operate it efficiently. Then along comes Strategy, lumbering through the IT undergrowth like a tyrannosaurus. Whole swathes of tactical applications are replaced by one strategic system because it consolidates costs and organisation into something that senior management can understand.

The strategic system is specified on the basis of functionality that is required by the user. Inevitably you are proposing to replace a tactical system that does everything the user wants with a strategic system that only does what they need. The cost distribution of the new system will have been worked out “fairly” by accountants. Which often means that your longest-standing users end up paying more for a system that does less.

It can be a hard sell. Fortunately you have senior management on your side and if the users don’t like it they can do their complaining from the deserted outpost they are transferred to.

Data retention

Your users are going to ask for all the data in the system to be retained indefinitely, and made available on line for ad hoc queries. Ignore them, they always say this. In my experience of having acceded to user requests like this, the data is never looked at after the end of the first month of the new system. Suggest a rational data retention policy and a straightforward migration of any applicable to data to the replacement system (if any). Be aware of legal and regulatory requirements.

Execution

I think the steps towards making a decision to decommission a system are these:

1. Establish a competent governance framework for the system.
2. Get a statement of the functional value of the system from its business owners
3. Get an analysis of the dependencies on the system from its IT owner
4. Get a signed agreement that certain named projects will replace any valuable functionality
5. Agree a rational data retention and migration policy
6. Get a signed agreement that the system can be turned off subject to the aforementioned project deliverables and data policy.
7. Plan the decommissioning like any other change using your best practice Change Management procedures.

Turning IT systems off is a good thing. I think it signifies that an IT organisation is mature but not senile. If understand your system landscape well enough to be able to say with confidence that this system is no longer required then you are doing something right. If you can actually execute that decision then it also reflects well on your relationship with your customers.

This isn’t quite the same as open source, of course. They are not suggesting that the code should be publically changeable.

Wired makes the valid point that scrutiny of the source code ought to be (and quite possibly is) a constitutional right.

Also I didn’t know that Australia had already done this two years ago.