It’s inevitable, really, as we move through our careers we won’t just be joining teams but replacing people and taking over their previous duties. We also have a tendency to blame many things about the infrastructure on the last guy who ran it. In my duties as a sysadmin and technical account manager I’ve had the very recent opportunity to “pick up the pieces” from two very different situations and administrators in a very short period.
Our client has had several admins in a short period of time (a year or so) and as a result of some poor decisions by those guys along with some terrible documentation we’ve found their environment was essentially crumbling from the inside. When I came on I found three systems with already failed drives, all of our documentation was missing several systems (physical and virtual), access to each system was non-standard (at least it was for the Linux systems) and at least a few devices we have no detail or access information for at all.
This situation was hardly a surprise, the company had fired their previous sysadmin several months prior and hired a new guy just a few days before I was assigned to the account. The physical infrastructure was a mess, to say the least. Once we’ve cleaned up the remaining cabling I’ll be posting another article showing the difference, despite their racks having no space for cable management. Their power distribution units were overloaded, some key parts of the infrastructure weren’t redundant at all.
While we’re in the middle of working on this, our own company’s Senior Sysadmin decided it was time to move on in his career and many of his responsibilities were transferred to me. This appears to have been an almost completely different experience — the vast majority of our environment is documented, we don’t have standard passwords but there are standardized access groups between different systems which were formally handed off, and current/planned projects were also handed off in a formally documented manner. I have little doubt that if anything fails either I or my manager has the access details needed to log in, and that our documentation has enough detail that I can begin working on the issue.
Again, this was barely a surprise. This was an admin who kept detailed notes on just about everything. He grew frustrated if any of the techs (or customers, for that matter) broke the cabling convention, much less broke the cable management system he had designed for them. The physical infrastructure is certainly aging, but there are projects in place to replace it and either a good backup that we can restore from if necessary or notes on how the systems are configured that we can rebuild from scratch if it comes to that.
As a relatively new admin gaining much-needed experience, this entire situation has been an eye-opener. The idea of clean-as-you-go (or rather, document as you go) is a grand one. Maintaining a well-documented environment is a key component of being invaluable. Many admins have this notion that if they keep everything a secret then they’ll ensure their own job security — this is true, but only in a limited capacity. If that is your only grasp on job security, that tells me you’re not that good at your job and should probably be replaced anyway. On the other hand, if you have a properly documented environment then 1) you can take vacations and trust you won’t get called because something trivial broke, and 2) the guy who follows you, whether you left voluntarily or by force (or joins your team as it expands) won’t be bad-mouthing you to all of your co-workers and other parts of the local administrative community.