Doing Root Cause Analysis

An incident is a terrible thing to waste

Failure is a wonderful teacher. We should use it better to learn how you fail
The root case of an incident is always process, culture or architecture.
A good RCA will wipe out a class of errors, not put a band-aid on the symptom. That is the value of going deep
A good RCA is one that goes deep. It must use a cause and effect fishbone and 5 whys2
Be vocally self critical. RCA’s are not the place to be defensive. Defensiveness during an RCA defeats the purpose of an RCA. This is not about a witch-hunt. It is about preventing a class of errors from happening again
Leaders must be present at an RCA. Decisions need to be made about deep changes. Architecture, process and culture changes should not be delegated to frontline engineers. Give constructive feedback during the RCA.
A good RCA requires you to be boundary-less. Often, we stop at the boundary of what we think we can ‘control’ and make a sub-optimal fix. Fix the problem at the root. If that means fixing it in a different organization, go do that.
Writing a good RCA is hard. Writing a good 6 pager is hard. Writing a good RCA is hard. Seek out the experts and understand what separates an RCA from a great RCA.
Put yourself in the shoes of the customer. When we say ‘the network was unavailable’ or ‘application crashed’ we are abstracting ourselves from the in ability of our customers to run mission critical software and they pain they feel.
Take an End to End perspective. Do not stop at your org boundary. The customer does not know or care about this.
Follow up and ensure completion of the root causes analyzed. Ensure the effectiveness by observing the implemented solutions in operations
Ensure that senior leadership is looking at RCA’s. They can help make the big changes that the org requires to fix root cause.
Share your learning. Don’t let others in the company re-discover the problem (e.g. cookie management)
Recognize a good RCA. Use it to teach others how to do it. Invite them to teach others how to do it well.
Give teams the time to do a proper RCA. It will payback 10x in maturity of the organization
Conduct premortems and FMEA’s as appropriate. (when to do what?)
Collaborate x-team on RCA’s. Just because you are in one business unit does not mean that your RCA stops there

  1. Wikipedia



My dad’s last act of defiance was by way of written instructions to my brother. He was on assisted breathing, and was unable to talk. The instructions were clear: ‘No traditional hindu rituals. Body to be donated for research’.

On the 31st of December, 2014, we bid farewell to a maverick, a true renaissance man and a free thinker. He was my dad. For the next few days, the phone rang off the hook. Colleagues who had worked with him 30 years back. Friends from College. The driver he had 30 years back. People loved him, and loved his genuine desire to help others.

His rebellion started early in life. A particular target of his was hindu rituals. An early act of defiance was his refusal to undergo ‘poonal’ or the sacred thread ceremony for brahmins. My dad’s father was not one who accustomed to anyone – much less his sons – standing upto him.  But my dad was clear and resolute: he yielded when his dad agreed that he could bypass other customs (including the ‘kundi’ which is the traditional way of shaving the hair of brahmins).

This is not to say he was not religious – he always distinguished between his beliefs and the rituals surrounding the belief. He was a fervent believer in Lord Venkateswara, and tried to make it to tirupati as often has he could. In his later years, that became too difficult, so he resorted to finding a religious loophole. He visited the Uppiliappan temple in Kumbakonam, near his native Thanjavur. Uppiliappan is the elder brother of Lord Venkateswara, so he reasoned that the Lord would be happy that he was paying respects to his elder brother.

The village where he grew up did not meet his needs to explore the world. He ended up at DMET, a marine engineering college, and got onboard a ship to see the world. On a trip back, he had his eye on my mom, someone he had known for a long time. He went straight to my mom’s dad, and asked if he could marry her. Sacrilege. Marriage was the decision of the parents, with the kids playing minor roles in it (“minor”. Heh. He would’ve liked that pun). But he prevailed and convinced his father-in-law to be. The down side was he knew his days on the sea were numbered.

He went on to becoming a landlubber. My parents moved to Rishikesh, at the foot of the himalayas. His restless nature got the better of him after a few years, and they hoofed it again to Bombay, then Surat, then Orissa, then West Bengal.. everywhere but his native south.

He loved everything he worked on. He was deeply devoted to his craft, though his craft changed over time. He loved the notion of computers making him more productive. 1983 was a gamechanger to him. He purchased the first IBM compatible PC, then released for the first time in India (HCL Busybee PC, two 5 1/4″ floppy drives). His purchase of the computer caused waves of consternation. The labor unions in the plant started agitating because the ‘computer machine’ was going to take over their jobs. In reality, all it did was spreadsheets and word processing (remember wordstar?)

His devotion to his family and his belief in his sons were out of the ordinary. This expressed itself in little ways sometimes, and really large bets sometimes. Thinking back, some of the things he did were just crazy. Like sending his second son – me – to the US for graduate studies – by taking a loan against his hard earned retirement fund. What would’ve happened if I were unable to make the best use of this opportunity? Looking back, I’m slightly horrified at the chance he took. He gambled his future to support my aspirations. (Maybe he never had the doubts in me that I had in myself)

His integrity shone like a beacon. In a society where slipping money under the table is par for the course, he refused steadfastly to accept any or give any, and lived with the consequences of doing that. He lived his life with single minded integrity. My brother reminisced that the only time he had seen tears in my dad’s eyes was when the Dean of a college demanded a job for his son in order to allow admission into the college. My dad refused, of course. It made him really sad to see corrupt educators. What would students learn?

He loved a good joke, and had a stable of standard ones. When I would visit him, we’d spend hours swapping jokes. In the hospital, I told him one last one: An American, an Englishman and an Indian had to cross a desert. They were allowed to take one thing with them on their trip. The American chose a 12 pack of ice cold beer of course. ‘When it gets hot, I’ll just chug it’. The englishman chose a large watermelon. ‘Nothing like a watermelon to cool you down, old chap!’. The indian, much to the puzzlement of the others, showed up with a car door. He explained ‘When it gets hot, I can roll down the window’.

He was too sick to smile. He gave me a thumbs up sign.

His inability to discipline us was a delight to his three sons. Once he did get mad and raised his voice, asked us to turn to the wall to administer corporal punishment. That consisted of using a ping pong paddle to tap us lightly on our backs. We giggled all the way through the punishment.

He smoked. Smoking in his generation was a rite of passage. We made lots of attempts to get him to quit. None of it worked till one summer I returned from the US with a 6 month supply of nicotine patches. At the age of 60, with 30 years of chain smoking under his belt, he quit. And then on he despised smoking (and often, smokers too). We like to think it added 20 years to his life.

His favorite gizmo was an iPad that I gifted him a few years back. The device stunned him. He treasured it. When he got somewhat better after his stroke, his first demand was for his beloved ipad. The easiest way for me to show my affection for him was to feed his gizmo fascination.

He wrote constantly. he started a family newsletter in the days prior to the internet, and mailed it out to the diaspora. He wrote a book of limericks. He published a monthly quality newsletter. At the age of 79, he was still working and active, often flying twice a month to various location, in his role as a quality consultant and ISO process auditor. We tried to get him to slow down, but his reaction was ‘Do you want me to keep doing what I love or not?’ He had my brother send out the last quality newsletter while he was in the hospital. He acted as an editor of the local newspaper. gratis. He loved what he did, and loved that he made a difference to others. He lived a full, happy life with varied experiences. He lived on his own terms and to his own standards. He marched to the beat of his own drummer.

Farewell, Appa. I can only hope to be half as great as you, and I would feel quite satisfied with that outcome.

Ashtanga quantified by my new Basis Watch

After 4 months of waiting, I finally got my new Watch: The Basis. You can read more about it here. It has a whole bunch of sensors in the form factor of a watch, that makes it really convenient to monitor your body.  It continuously captures heart rate patterns, motion, perspiration and skin temperature. The website contains fascinating data about a lot of stuff – your temperature, prespiration, how many calories you burned during the day, with an hour by hour breakdown. Awesome for data geeks.

Of course, the first thing I wanted to know is my vital signs during ashtanga practice at my awesome shala, Yoga is Youth in Mountain View, CA.

The most interesting insight was how Ashtanga impacts perspiration. I know I sweat a lot during practice, but the closing sequence cools us down, and the final shavasana (corpse) posture relaxes us. But this is what – consistently – the basis band showed. In the chart below, my practice was done around 9 AM. As would be obvious, I started at 7:30. I started the closing sequence at 8:50 – but I really did not stop perspiring till 11:30 – when I went for a shower! So what I’ve been told by teachers is actually true – wait for atleast an hour before having a shower after practice.

Perspiration - asthanga primary sequence completed at 9 AM

In case you think this is an error, here’s the reading from Thursday, when I started at 6:30, and went for a shower at 9. Perspiration continues till I had a shower.


Heart Rate

The heart rate capture is a little bit annoying. As you can see from the chart below, Basis seems to have some trouble capturing the right heart rate when you are perspiring (I think). So suddenly your heart rate drops from 110+ to 30.

Ignoring these losses, the chart shows what I’d expect: A gradual rise in heart rate, and then a slow down as the closing sequence is done.

The peak was reached at around 8:45, of 133 beats/sec. Right when I was doing supta kurmasana , Titibasana and the jumpback, I bet. (check it out on youtube if you don’t know what that is)


I do not buy what Basis is telling me – that essentially, the days I do 2 hours of a really physical practice of ashtanga, I burn essentially the same number of Calories as I do on non-Ashtanga days.

I suspect I know the reason for this. I have noticed that doing Ashtanga does not add significantly to the step count for the day, which presumably is how a major portion of calorie consumption is computed. Here is a monday compared with Sunday



I have not found the skin temperature to be very useful. It clearly increases during practice, and then goes back down after. Just what you’d expect

Finally, the step monitoring is actually very useful – I found it quite motivating to try to slip in some extra walking during the day. The feedback cycle really works.

The strap

I really hate the strap. If you fasten the strap too tight, the sensors will start digging into your skin. I do not know if this is because I am not used to wearing a watch, but I find myself adjusting the strap every 15 minutes because it is uncomfortable. I wish Basis would offer the option of a cloth and a steel strap. Plastic sucks, IMO. 

In Conclusion

Any device that gives feedback on how you’re doing is bound to be interesting, useful and motivating.  I highly recommend trying it out. Oh, it is also waterproof, though I prefer removing it when I have a shower



rails: undefined method `model_name’ for NilClass:Class

I ran into this problem recently. It was pretty hard to correlate the error to  the solution.

hitting my /users/new method had suddenly produced this error

undefined method `model_name' for NilClass:Class
in my view on this line
 <%= form_for(@user) do |f| %>
The obvious thought was that somehow my controller did not define @user – but it did. I put a debug statement in the new controller which did not execute. 
The real reason for the error? A syntax error elsewhere in the file. There was an unbalanced ‘end’ statement in a different method in the users controller.  I would’ve expected a syntax error, but no such luck. Fixing this resolved the problem

Meta Thinking

As usual, I discovered I knew something because someone asked me how I do something they had observed me doing.  This has become an interesting pattern for me: I am unaware of things I am competent at. This is not a good place to be because it means you cannot teach it to others. Being consciously competent, I have found, sometimes requires someone to ask me a question.

Let’s put that aside for a moment. The question I was asked was about thinking strategically. I was really taken aback to see  my version of the answer slip off my tongue effortlessly: It is then that you realize that you are pretty good at something without having known it.

So here’s my framing of how to think strategically:

Imagine that you are in a room with a bunch of people. Someone has opened up an idea and there is a free form discussion in the room about the idea.

So what happens in meetings? Someone says something. And there is a reaction to it. And yet another reaction. And so the discussion continues, usually bound by a quantum of time. 1 hour? Have a relaxed discussion for 45, and demonstrate a sense of urgency in the last 15.

The problem with these discussions is the focus on action-reaction.  Reacting to something results in narrowing of focus, and an examination of a path laid out in front of you – and disregarding other avenues that might exist. Indeed, this is the reason that brainstorms are ineffective. They narrow the focus down prematurely.

Narrowing is appropriate in some instances and very inappropriate in others. The problem is when we do not take the time to distinguish between the two.

Quite generally, I have observed three kinds of people in discussions

  • The Take Charge guy. This person will lead from the front, be fearless in saying what is on their mind, and will be the person to get  things closed out and keep an eye on the time.
  • The timid minority – They will barely contribute to the discussion, and when they do, it is because they are very sure of the facts. Being vulnerable is not a strength. They would go to great extents to avoid being corrected.
  • The Quiet listener – this person usually will not speak for the first 5-10 minutes and take the time to absorb what is going on. But when they do speak, they have a significant impact on the discussion, and often it will be a revelation to others – Why did we not think about it? This is your strategic thinker. They garner respect and everyone waits for him or her to speak up. They are able to do it because they have a slightly different frame of reference from others.

What this person does is apply the meta-thinking principle. Stop, uplevel, understand the bigger picture. Often, what this person finds is that everyone is missing the big picture.

If the room represents a bounds on the limits of the discussion, what this person finds is that there is a door that leads to another room, and well as one that takes you to the courtyard. They first zoom out to this bigger picture, understand it , and then either zoom back down to the smaller level – or help others understand  the bigger context.

This can sometimes work against this person – Strategy and tactics go hand in hand.

“Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat.” – Sun Tzu

Being stuck in the big picture land without the ability to ferret out the path to take is a problem many leaders face – especially those who excel in being able to see the big picture. The true leader is the person who can go broad first and then help narrow down, while keeping the big picture in mind.

My advice: Actively cultive the big picture thinking. Resist the urge to jump directly into conversations till you have understood the big picture. The best way to do this: prepare beforehand and understand the big picture. Become the guy that everyone waits to hear from because they are likely to have a deep thought to share with you that could completely upend the discussion. And then be the person who will help zoom back in and propose how we should solve the problem – and actively narrow the scope of discussion.

The retina macbook pro

My 2007 macbook pro was too old to take the mountain lion upgrade. Fine, I took the plunge and forked over the money for the new retina macbook pro.

You don’t normally use words like ‘breathtaking’ when talking about laptops, but I am tongue tied. This laptop is like nothing you might have used before. I’m tempted to say ‘and ever will’, but I know that’s not true.

If you are like me, you have seen this laptop and walked away saying ‘Meh’. Trust me, once you start using the Retina display, there is no going back. More than pictures, it is text that I still gawk at. Brilliantly clear and sharp. Writing code on it goes from fun to downright pleasurable. The form fact is just perfect. So much thinner and lighter. I struggle with using the previously beautiful apple 27″ monitor. Suddenly, everything else has glaring imperfections.  Going from retina to non-retina is a struggle. I have never heard the fan (even when I am running flash).

It reminds me of what Job’s said -‘ Design is how it works, not how it looks’. This laptop is a real winner. The only problem is all the drool landing on the keyboard


12 TED talks for entrepreneurs

The Shopify blog put together a list of 12 talks for entrepreneurs. Here is my list of the major points of the talks:

Dan Ariely:  Are we in control of our own decisions ?

(read his amazing book, predictably irrational)

We are not always in control of our decision making – and how choices are presented to us influence us heavily – and we are completely unaware of it

Rory Sutherland: life lessons from an ad man

Seth Godin: How to get your ideas to spread

I found this having a lot of interesting overlaps with Seth Godins presentation:  You have to either build a remarkable product (Godin) or figure out a way to make it remarkable to people (Sutherland). It is either about having enough implicit value for people that they talk about it (‘no one talks about a cow by the side of the road – but they will if it is purple’) or creating creating intangible value -potatoes became a staple in German food (the kings garden started growing it and guarded the crop after the citizens refused to eat it)

Simon Sinek: How great leaders inspire action

Entrepreneurs make a mistake by focusing on the ‘what’ and the ‘how’. Great leaders focus on the why. Inspire people. Martin Luther  King said ‘I have a dream’ not ‘I have a plan’.  Apple employees believe that they produce great, beautifully crafted products, and consumers can see the love in it. They buy into the products emotionally

(to be continued..)

The three levels of decision making

A recent conversation with a colleague left me thinking about decision making, and the effectiveness of decision making.

How decisions are made impact the effectiveness of the decision, and it is not obvious that everyone thinks through this before deciding.

Imagine that you are a manager. You can

  • Mandate a decision
  • Make a decision, but consult others before making it
  • Delegate a decision or influence the outcome of a decision

From top to bottom, the difficulty level of decision making increases. From bottom to top, the value of the decision decreases.

Let’s consider a decision you have mandated (‘Everyone checks in code at the end of the day, period!’ you yell in an email). The decision is quick. But no one else on your team has skin in the game, and so there is no incentive for anyone else to actually follow this. In fact, it is quite likely that someone on the team is bitching about it behind your back (‘ What a clueless guy! How can we check in code that does not compile? We will then be held responsible for breaking the build !’)

A better option is to consult the team before deciding. Solicit opinions. Use them to come up with a resolution for the underlying problem you are trying to solve. It will take longer – but the decision has more value, because others have contributed to it.

The hardest kind of decision that you will make is one that you do not make. This happens when you delegate a decision or influence someone else – a peer, for example – into making the right decision. This is hard stuff, and it takes time. Yet it is the most valuable kind of decision – because someone else made the decision. They will be committed to it, more so than you. They will go out of their way to get to the right outcome for that decision. These are the kind of decisions that leaders make, by influencing outcomes that they do not control to get team buy in. You know a good leader when they do not throw their weight around, but ask others to make decisions and influence the outcome

Like many others, I mourned Steve Job’s demise. I chanced upon an article which described how he quit Altair because he wanted to visit India.

The obvious thought after reading this was to indi-fy his picture.

Of course, this work was done on a mac, using pixelmator from the macos app store.

Do Women rule the internet?

An interesting article doing the rounds is one titled ‘Why Women Rule The Internet‘. It is authored by Aileen Lee, who is a partner at Kleiner. The article is just strange. I fail to understand the premise and the conclusion it draws.

The premise of the article is
– Women are the majority of users on social websites
– Women shop a lot on the internet, more so than men
– More women are active on twitter. (Apparently, Twitter is supposed to be a ‘techie insider’ product. I am stunned)
– So Women rule the internet

It is no secret either that women are more active in social aspects of life, from relationships to parenthood. (I hope this does not sound chavunistic). Men’s magazines are about fast cars, girls and more fast cars. Women’s are about relationship, fashion and beauty. So why is it surprising that social web sites – all about relationships – attract more women than men?

How on earth is ‘Women shop a lot on the internet’ insightful? I have no interest in digging up stats on this, but really, is that so different from the experience in malls? Women shop more. The fact that they shop more on the internet – well, duh.

I find it interesting to see how ‘consuming’ is being mistaken for ‘ruling’. Consumers are not rulers: they are the suckers who purchase products and fork over their hard earned money. The producers of web properties are the rulers. They take your money to the bank. Women will rule the internet when they produce the web sites where consumers – men and women – go to consume. It is heartening to see more women entrepreneurs, as alluded to by the story. But to claim they rule the internet is silly. When the next generation of Facebook, Zynga, Groupon and Twitter has female CEO’s, there is a claim to be made. Till then, the only claims that can be made are ‘there are more women consumers of social and shopping web sites than men’ and ‘If you are selling stuff online, make sure you take female consumers into account’.