The gods are against me - confirmation bias and capacity management

Usually the feeling that the gods and the forces of nature habitually conspire against us is a product of confirmation bias - we forget all the times that woes come as single spies, because the times they come in battalions are so much more memorable.

It's important to be aware that this is not always the case. You are not paranoid when the bastards really are out to get you.

In particular, in many circumstances, maybe even the case of a washing machine, the underlying problem can be one of capacity - capacity problems are difficult to detect because they are intermittent at first, and then, finally, and spectacularly, catastrophic.

There are, in fact, a few cognitive biases involved in producing such things as 'Murphy's law' and 'Sod's Law'. We find things more important if they happen to us. We like to have a reason for things happening, and though the theory that the world is against us is an unlikely one, it is, at least, a theory, so we prefer it to accepting that happenstance is usually a good reason for coincidences.

We also are very poor at judging the probability of things happening. Often, what seems a very unlikely event, is, when you consider the size of the population, and the time over which it could happen, actually something that's almost certain to happen somewhere at least a few times a decade.

How can we then distinguish those events that signal a preventable catastrophe from those that are merely isolated events?

Unfortunately, the simple answer is, that we can't. The reason that our brains are so inclined to so many fallacies is because we live in an uncertain world, and a collection of heuristics that work fairly well, most of the time, is worth having, and using, even though it also leads us into such errors.

The more complicated answer is that events that are connected to one, or a small number, of related causes, that are a consequence of a mismatch between demand and capacity, have some characteristics that allows you to spot them against the camouflage of background noise.

These are that capacity related problems cause events that are:

- Intermittent.
- Apparently unrelated, but often coincident with a specific time of day, week or month.
- Progressive. Strange things happen once or twice a week, but then more often, once or twice a day
- Responsive to intervention. You may try to fix a symptom, and find they go away for a while
- More serious over time. Before the final catastrophe, you'll have one or two more serious events than usual

You'll notice that these characteristics fit a number of naturally occurring events - avalanches, earthquakes and volcanoes being examples. That's not an accident, these events are also capacity related - stresses build up over time, with minor event cascades (there are often a series of small earthquakes before a volcanic eruption, for example).

What can we do about this unpredictability?

When you see the relationship with natural events, you'll see what we actually do. Firstly, we need to anticipate where such a problem might occur, then see how serious it is (we're less concerned with volcanoes under the sea, far from any land, than volcanoes near towns, for example), and then put monitoring in place.

We need to design the monitoring carefully, to make sure that the metrics we use make sense, are connected with the likely capacity problem, and are measuring the system itself.

Then we need to measure the trends. Not just trends that are obviously leading to a catastrophe, but all trends. Then we need to correlate these trends with each other, project where they are tending towards, and find out what is causing the trends. Then we can put measures in place to reverse the trend, or, if that isn't possible, increase the capacity we have to deal with it, or, if that isn't possible, find a way to mitigate the risk of a meltdown.

Measuring trends is a more subtle matter than it might seem. It's often not the most obvious trend, in the main demand, that's the danger. Smaller, deviations at periods of quiet demand, or on the shoulders of a demand peak, are often the warnings.

The analysis required to detect such off-peak trends isn't that difficult to do from a mathematical point of view, but it does mean that you need to design your thresholds in a more sophisticated way than simply a maximum or minimum, based on a percentage of historical demand.

Open Source Project Proposal: Ada Transport Level Security (TLS) module [Draft]

The Problem

The security of web-sites underpins much of the world's on-line economy. Breaches to it, and potential flaws in implementations of it, are a substantial risk to many organisations, in many countries.

There has already been a famous vulnerability found, and repaired, in the OpenSSL implementation, but there are many closed-source implementations that may still have similar, or more severe, vulnerabilities, or that may be compromised in other ways.

One of the reasons for these vulnerabilities has been the implementation of the solutions to TLS in languages such as c, which is an inherently insecure language, and a language that it is difficult to prove, verify or to correct.

The Proposal

To establish a team, or Ada and security experts, to produce a TLS solution, written in Ada for, in the first instance, servers. This solution would provide an API that could be used with, for example, Apache.

Once this solution had been tested, proved and deployed successfully, the solution would be extended to the client side, so that browsers, such as Firefox could use it.

Funding

The proposal depends on the team being paid for the work, and for enhancements also to be paid.

Ideally this would first come from a grant, or grants. Bodies that might wish to provide funds for such grants could include OpenSSL (www.openssl.org), the EU (https://www.enisa.europa.eu ), the Digital Governments Initiative, D5 London (UK, South Korea, Estonia, Israel and New Zealand https://www.gov.uk/government/topical-events/d5-london-2014-leading-digital-governments ), the British Banking Association (BBA), and many others.

Long term funding would come from income. The produce would have dual-licensing. Free open source to individuals, and open source projects, such as Mozilla, but commercial licensing to organisations such as Apple.

The Requirements

The project, to be a success must comply with these requirements 

• Satisfy TLS 1.2 and 1.3
• Be designed to provide general transport layer security
• Be compatible with existing TLS apis
• Ensure highly secure design
• Establish a method to verify a server is running a particular version
• Ensure code is easy to maintain
• Use Ada not just as the language, but as an example of good, secure, reliable and fast open source Ada

Provisional timeline

Funding applications: May-August 2016

Team Recruitment: September 2016

Design: September-October 2016

Coding: November-December-January 2016

Testing: February-March 2017

Beta with customers: April-May 2017

Full Release: September 2017

Next Steps

Please comment on this blog if you have any suggestions for improvements to this draft, or write to peter.brooks@service-governance.org

Good corporate citizenship - and 'The Myth of Maximizing Shareholder Value' - and Service Governance

Here's an important article, on governance, The Myth of Maximizing Shareholder Value  - unfortunately the page doesn't allow replies, so I've put the points in this short blog entry.

Governance thinking, even in the US, is moving. When we are providing consultancy to organisations, we need to be aware of this shift, and, as discussed in 'Collaborative Consultancy' able to make judgements about our ethical accountability to the organisation, its stakeholders, and to ourselves.

Some of the ideas, being based on US law, are not directly applicable everywhere, but the overall argument is, and it's crucial to the future.

The article stops short of a full description of the solution - which is fair enough, as it's seeking to illustrate the problem.

Outside the US, governance thinking has understood this for some time. The law in the UK, South Africa, and other places that have accepted the thinking found in the Cadbury Report, and the King Commission, is that Corporations are required to be good corporate citizens. Their duty is indeed not to maximise profit for shareholders, rather, their duty is to deliver value to all their stakeholders (and, of course, shareholders are a stakeholder, and returns are important to them).

Corporate governance, requiring that corporations deliver value to their stakeholders is a powerful principle, particularly when enforced through a 'comply or explain' method (not ticking boxes on a pro-forma 'have you complied with X' sheet).

What it means is that corporations have to understand who their stakeholders are - the inhabitants of Bhopal were stakeholders in Union Carbide, as they found out, most horribly. If Union Carbide had known that they were stakeholders, and known that it had a corporate duty, to make sure that there was no negligence at that site that could lead to such a disaster, then history would have been very different.

They then have to understand how their vision, mission and charter can deliver value appropriately to all their stakeholders.

Part of the difficulty, particularly for those who have only been aware of profit as a value, is understanding what stakeholder value is, and how to govern it.

A method, Service Governance, using existing best practice frameworks as a basis, exists to help identify stakeholder value, and govern that value, using the paradigm of a 'service' and governing the organisation through a service portfolio, optimising the value / cost ratio, for stakeholder value.

There's more on Service Governance here:

Adopting Service Governance - Governing Portfolio Value for Sound Corporate Citizenship

There is an example of Service Governance working, a short video, on the web-site www.service-governance.org

Adopting Service Governance - a short introduction (Video)

There are also blogs, discussing service governance here:

AXELOS Blog: Making Service Governance Work - The ITIL Advantagetil-advantage

Corporate Governance issues & Service Governance

Organisational value through Service Governance

Centaurs: Organisational Change Management and horse riding.

Centaurs: Organisational Change Management & horse riding.

Part of the problem with organisational change is perception. People see it as something you do, like driving a car, or riding a bicycle. It isn't, though, like that, it's more like riding a horse. 

If the horse wants to make a dash for home, or throw you into the ditch, that's what it'll do. 

You have to help the horse see things your way, and agree to go where you want it to go, and you have to be aware that horses get tired, and need feeding, because, if you don't feed them, rest them, and give them time to play, they become sullen, resentful, uncooperative and, eventually, die.

It's also best not to walk behind a horse - with organisations it isn't alway obvious where the behind is. [though you might guess]

If you wish to be good at organisational change, you need the equivalent of riding lessons - and, if you've learned to ride a horse, you'll know that riding lessons involve lots, and lots of practice.

You also learn that you can't ride a horse on autopilot. You have to be one the horse and aware of it's every twitch and mood. You have to be fully engaged with the horse - with top riders, the horse and the rider seem to be one creature, with one mind. 

Some believe that that is where the myth of the centaur came from - seeing horses ridden so that they looked like one creature, part horse, and part man.

That's the aim. To be like that, when you work to change an organisation.