Re-Architecting from the Ground Up

Well, nearly the ground.

I think one of the most enjoyable experiences in my line of work is when I get to design and build an environment from almost nothing, with a handful of requirements and design constraints. We do it with some regularity for customers as part of our support agreements, and I’ve now had the joy of doing this twice for my employer – once as the result of an audit/upgrade process, and once as the result of a merger and associated consolidation effort.

The thing is, there are so many roles involved in operating and maintaining today’s corporate environments, and each one needs to have a decision made, either directly or by association, about what tool or policy will fill that role.

Take User Authentication as an example. There are choices to make, which partly depend on whether your logins are on Windows, Linux, Mac, or something else again. Perhaps the three most common choices for filling this role are 1) Active Directory, 2) LDAP, 3) Local Authentication. For tiny organizations, #3 is quite acceptable. If the organization wants central authentication, then the existing skillset and projected direction of the company will typically dictate whether #1 or #2 is chosen, each having their relative merits for any given situation.

When I first started with my current company, the “corporate standard” (and I use the term loosely) was CentOS 5 (latest available at time of install), with centralized LDAP for authentication and authorization to backend SSH logins as well as several HTTP Basic Auth sites that we use(d). No configuration management whatsoever, and all patching was undertaken manually. There was manually configured Nagios monitoring, There wasn’t even a complete list of systems that the group managed. All of the switches and routers were managed by a separate networking team who configured everything with local authentication, individual user accounts on each device. They at least had a more accurate list of network devices that they managed.

The second iteration that I saw, and later became a part of the continued development for, was an improvement over the previous. We were using CentOS 6 (latest available), at the beginning it was still centralized LDAP without any patch management, but we were using Puppet to distribute and manage configurations. Once I was involved we took three drastic measures (not all my choice, in fairness): First, we implemented Spacewalk (2.3, from memory) to handle system patch management. Second, we automated the systems monitoring by integrating Nagios with Puppet. And third, we implemented FreeIPA as a replacement for LDAP. Patching systems became a breeze instead of a nightmare. Monitoring was as simple as adding a system to Puppet and running the agent — we were operating Puppet on 30 minute checkins, so service checks appearing in Nagios should take no more than an hour. Adding services to be checked by Nagios also meant only a few lines of code for the Puppet manifests, and once pushed into the Production server it would be updated into Nagios within the hour. FreeIPA made our centralized auth environment simpler, and while our implementation had issues and was far from perfect it made administration of users and groups much easier overall.

Now, we’re in the middle of doing it all again. The new environment is approximately thus:

  • User authentication is handled by Active Directory. Old FreeIPA and LDAP environments will be deprecated and decommissioned. (We considered keeping FreeIPA and creating a trust, and decided that at our size it was more complexity than was warranted. Most of the FreeIPA tasks that can’t be handled easily by Active Directory will be handed back to Puppet).
  • System standard is for CentOS 7. (We like CentOS around here, OK?)
  • Configuration management is still Puppet, though now being handled by The Foreman (vs. manually, as it was before). We really like puppet — it works well for us.
  • Patch management is handled by the Katello plugin for The Foreman. (Spacewalk has been deprecated, and there are a few things about it that we just didn’t like or use so there was little-to-no point in keeping it around aside from its ability to patch things. Katello and The Foreman replace many of the functions that Spacewalk did or could have done, and do it ten times better as well).

Like I said in my last post, everything good must end. I remember when I built my first environment backend management system (something like Puppet + Spacewalk) and naively thinking I’d never have to build another design again, we’d just keep updating the one we’d chosen. No dice. Every opportunity one has to rebuild the environment should be taken — approach it carefully, take into account what was learned in the last round, understand what was done wrong and what was done right, what can be improved upon with changes in technology and products available.

I, for one, enjoy the challenge.

Leave a Reply