Playing Sherlock Holmes at Work

I have been an avid detective novel reader since childhood. The way a puzzling mystery can be solved easily by the detective through the power of deduction just fascinates me*. I never realized that can actually be an important trait for being a software engineer.

Couple of days ago, our VP of engineering asked our QA lead why the databases he provisioned all shutdown overnight. Was it because of some undiscovered bug? Was it a longevity issue? Did they test things overnight? Since I was responsible for that feature, our QA lead solicited my help to investigate.

Knowing that things wouldn't "just happen" (against the law of Physics), I decided that the shutdown could only happen if 1) it was issued from outside (by user or through a crown job), 2) it was triggered by some internal policy. I also remembered all the outside traffic was logged into our audit log files (since I wrote that module), so, if we couldn't find any logs about shutdown issued from outside, it would be an internal job (and I could transfer this issue to another engineer to investigate).

Sure enough, we found a few commands issued from outside to shutdown those databases in the log. Where were they from? Fortunately (actually it's by design), the source ip of the mysterious outside entity was logged, so we could tell the culprit's ip address. Our QA engineer, who was helping me investigating this issue, looked at it and immediately said: "It's from VPN".

We dug through our VPN log, trying to see who was the real originator of ip address. Unfortunately, our VPN log was not kept that long so we couldn't find it. Suddenly, I remembered our VPN software always assigned the same ip address to us when we used it. So we could actually just ask people to login to find that out. (We are a small company so it's not an issue to try this route.) By that time our QAs and I had started to suspect our VP of engineer was actually the culprit. Sure enough, his VPN ip indicated the commands were issued from his laptop and he "sheepishly" admitted he forgot he issued those commands before he went to sleep. Mystery solved.

So next time when you are interviewing candidates, ask them what they do for pastime. Their reading detective novels may tell you more about their debugging skill and ways of thinking than your standard interview questions.

P.S. I think every software developer should remember the Holmes rule while debugging: "once you eliminate the impossible, whatever remains, not matter how improbable, must be the truth".

*Naturally, you can tell I am a fan of the classical (so called "golden age") detective novels, which focused mostly on the puzzle solving. The newer, "modern" detective (or crime) novels, though claimed to have better writings, don't interest me that much.