Effective post-mortems

One of my tasks as a CTO is running post-mortem meetings after we have an incident or outage. This is an extremely important step toward making progress in system stability and performance. Teams that don't do post-mortems miss the opportunity to get ahead of system issues.

Post-mortem meetings take some finessing, they can easily turn into a blame-game, or he-said she-said. So let me layout how I run post-mortem meetings, and how that makes them most effective.

First, important rules for post-mortems:-
1) Timely to issue (next day is best)
2) All relevant members present (no meeting if someone is missing)
3) Impartial moderator
4) Empty whiteboard to describe incident/issue
5) There is no blame

It is critically important to have "No blame", as you will make no progress otherwise. You need to acknowledge that everyone is working hard, systems can fail and its nobody's fault, and that openly discussing the issues together is the best road to ultimate resolution and system growth.

Now, even if your team knows "no blame", they will probably still be on edge at the start of the meeting. Its natural, failing systems create pressure. The team may also be avoiding dealing with the issue, hoping it will go away, and you may have to bring them back into it. What I find in post-mortems is that teams try too quickly to get to a solution. Don't let them, instead have them focus on a timeline of events.

I've found that starting the meeting with a chronology of events is extremely effective. I ask the team "what happened", and "when", and make them be exact about the when (i.e. 5:14pm), and I transcribe it all to the whiteboard. We include communications and hand-offs in the timeline, and any other information we collected at the time or gleaned later from system logs. Something about the focus on an exact time-line gets everyone to focus as a team. Maybe its because it turns us into detectives examining someone elses problem, not ours (if anyone has a better theory as to why, let me know).

After an hour, we usually have a list of immediate, medium, and long-term actions to take to remedy the issue. From that, the candidate cause/solution usually stands out, and we make sure we have alternate solutions should our candidate be wrong. These are captured on the whiteboard, and we make sure have both mitigations and solutions (making a problem go away is sometimes as good as fixing it).

Email wrong-number

Sometimes technology can have significant and unintended cultural impacts. Email in its infancy was a minefield, in that we never realized it was an inappropriate (and one-sided) tool for expressing emotions. It took us a while, but we finally taught ourselves to reread and pause before hitting the send button. In Japanese, there is a word for "unsay" to take something back, but alas not in English, nor is there a reliable unsend in email.

I've been using a Blackberry for phone and email for a while, and I've noticed an interesting phenomenon. I will call it the "Friendly wrong number". I meant to call Wendy A, but instead called Wendy B, since I dialed from my email address book and they were adjacent. "Hi, Wendy?" "Yes, who's this?" "Its Jon" "Jon? Oh, we haven't spoken for a few years, You still working at...?". Damn, wrong Wendy. But here's the kicker. I know Wendy and can't just say sorry, wrong number. This could get dicey if your boss is "Jamie B", and your best buddy is "Jamie C".

A variation is the call-back from someone you just spoke to, but by mistake. Some new phone feature must be causing that, I am guessing. Another variation is in email, the dreaded auto-fill feature in Outlook.

This makes me wonder what unintended future impacts technology will have. Many of my LinkedIn contacts have photos, and it won't be long before they also show on phone and email messages on my Blackberry. Will the prevalence of GPS maps make us lose our orientation, and we'll need to carry compasses with us where ever we go (Compass, what's a compass?). I guess we'll find out.

Worthless processes

When reviewing a write-up or new process/project proposal, a filter that I will apply is "Does this make a difference?".

Let's say one of your team was asked to write up a plan for "managing execution risk", and they wrote a document describing a process for doing this. After you read the document, you decide that while it all makes logical sense, it really will not make a difference. For example, tracking how many lines of code are written each day makes no difference. Who cares? Its business outcomes that matter.

As a CTO, it is your job recognize tasks that don't make a difference, and tell your team that they can stop doing. If something is worthless, say so. Or go one step further, and ask your team to tell you tasks they believe are not useful or optimal. Think of it as spring cleaning. Some tasks are not negiotable and are in fact important for compliance reasons (ie Sarbanes Oxley). But if its internal driven, reexamine it. Our teams are so busy, they'll appreciate the time savings.

So, what processes/tasks are your team doing that you can throw away?

Birds of a Feather

One of my most rewarding experiences is spending time with other CTOs. Who else knows what its like to do what you do? Nothing like hanging out over chinese food and a few beers with like-minded colleagues that I trust.

Earlier in my career, I felt competitive with my peers, both at work and outside. I felt that it was an "either/or" proposition, either I succeeded/got promoted, or they did. I was comparing myself to them. Eventually I realized that my success (and others) was independent of anyone else. If I was successful, I would be rewarded, regardless of anyone else.

Now I am realizing that younger technology managers I've watched over the years are joining our peer network, and its great to see them advancing as I did. It's such a pleasure to have them as peers, and to see them grow into their leadership roles.

One thing I always experience from peers is "have you thought about...". Its great because (1) I trust and respect their views and, (2) it challenges my thinking on something I hadn't considered. Reminds me of my high school days with a great teacher when I would have a break-through epiphany on a complex concept. There is always something to be learned from your peers.

Technology around the globe

David Goodman, CTO of International Rescue Committee (http://www.theirc.org) blogged about his recent trip to Africa, where he spent time in Kenyan and Ethopian refugee camps and reviewing technology needs (http://ctoinafrica.blogspot.com). One discovery was that the internet band-width was poor, and David's blog got me thinking in general about how technology is different around the world.

I was going to title this posting "global technology", but that implies we use the same technologies around the world, but that's just not true. Yes, there are similarities, but from a CTO's point of view technology management is quite different once you leave the US, whereas I am convinced that managing technology in the 48 continental states is effectively the same.

I am going to skip the obvious differences (currencies, consumer shipping & payment methods, multi-language, multii-byte character sets, telecomm, regulatory issues), and focus on other non-obvious issues relevant to a CTO dealing with technology overseas. Here are just a few:-
1) US companies do more custom/in-house development than other countries
2) No-name PCs are often used, as not all vendors are established in all countries (i.e. Dell recently lost their presence in Israel)
3) Phone text messaging is more prevalent at executive level than Blackberries

Clearly, some differences will change over time as companies adopt US practices. But the point is, you can't just apply best practices from the US overseas. Be careful about decisions like "every application must run in a browser", it won't necessarily work. I believe the best approach is to support local technology management in individual countries/regions, and as a CTO support those individuals. Disagree, or have a different experience? Let me know.

Every CTO should be on Facebook

Every CTO should be on Facebook. Why? Facebook could well be what applications look and work like in the future. I don't know if Facebook itself will persist, but certainly the ideas behind it will.

By now, I assume most of us are using LinkedIn, and finding it to be a valuable business networking tool. While Plaxo is mimicing LinkedIn, LinkedIn is mimicing Facebook. You can now load your picture in Linkedin. And LinkedIn just added "today", "yesterday", "last week". Guess where that comes from? Facebook.

"Facebook?", you say. But that's mainly kids. I am not saying that you will get a social networking benefit from using Facebook, but you will learn tons about how new social network interfaces are evolving. Facebook follows the same connection rules as Facebook, other members can only connect to if you accept. I have some personal rules for using facebook. First, accepting links/friends only from people I've met and know. And most importantly, I limit myself to 5-10 mins at a time, Facebook can be really addictive and time-consuming.

There are 1,000s of developers (or more) developing applications for Facebook, and you can try all of them out. Facebook is truly a technology marvel., if your not checking out Facebook, you may really be missing something.

Video promise

Two bigs trends on the web right now are Social Networking and Video. I've been following internet-based video off-and-on, starting when I did some consulting for a CLEC in 1998. I also looked at IP-based video conferencing for international collaboration. It always seemed to me that video was a promise that never delivers; stiff video conferences and postage stamp-sized choppy video were hard to get excited about.

Well, consumer video (like youtube) is changing all that as it takes off, clearly exploding in use. The combination of high-speed internet and the fact that almost all single-shot digital cameras can also shoot and download video clearly helps. Anyone can shoot a video and download it, and then have youtube or similar deliver it at no cost. Google search results now frequently include videos.

What crystallized the potential of video for me was my own experience playing the guitar. As I hear a song that I would like to learn to play, I now go to youtube to see how other amateurs guitarists have played it. They've shot themselves close-up, with view of guitar fret board, so you can see what chords and strumming patterns they are playing, verses trying to learn the song from listening to the recording. This has been a completely breakthrough for me on new songs, and has improved my playing immeasurably.

I can't help but imagine that there are thousands of other examples, both personal and business, where simple-shot video can be used as a training, business, sales or help tool. Expect to see video really take off in 2008.