September 7, 2009, 7:58 pm
One of the frustrating things about building and running backend infrastructure services is that things break sometimes — often in unexpected ways. And when they do, you can spend untold hours chasing down various ideas, most of which are dead ends, before eventually solving the problem. The goal in the end, as always, is to have a more stable and resilient service that will serve your users without getting you out of bed in the middle of the night.
What‚Äôs often interesting about these problems is not necessarily the solution ‚Äî it‚Äôs what you learn along the way. If you‚Äôre not debugging and testing alone, you‚Äôve likely got a small team of folks helping you out. And in that process you‚Äôll discover things you‚Äôd never encountered before, including hardware/software interactions that you‚Äôd previously never considered…
