We have also embarked on a journey to automate our systems configuration as much as possible. We have selected Puppet, a systems configuration management and automation tool, to help in that task.

One of the advantages of Puppet is that one can start small. Once a system is under Puppet's control, it is easy to expand the Puppet configuration over time, and thus automate more and more of it. We started out with a very limited configuration that defined some settings that are common to all our systems -- for instance, making sure the sshd configuration is secure. Puppet makes it easy to differentiate rules based on "facts" about a machine, like the version of the operating system it runs, or whether the system is a physical server or a virtual machine. Each system under Puppet's control gets its own configuration stanza, so it is also possible to do things that are specific to one machine.

Here are some of the things that we now do with Puppet, rather than by hand:

• install appropriate software packages, and standard GNU/FSF configurations for them
• create and manage user accounts
• distribute SSH public keys and SSL certificates
• update xen-tools configurations on virtualization host systems

We create new virtual machines with xen-tools, which pulls in the Puppet packages. We then add the new virtual machine to our Puppet configuration, which pulls in the default GNU/FSF configuration without any additional work on our part. We save a lot of time setting up new machines, and we get peace of mind: all our systems under Puppet's control are guaranteed to have our standard configuration.

We are currently working towards the goal of generating our automated systems monitoring configuration from our Puppet configuration. This requires us to migrate more service configurations to Puppet. With enough of that done, it should be possible for Puppet to know that, for instance, www.gnu.org runs a web server on port 80. With that knowledge, Puppet can instruct our monitoring hosts to check for the availability of that service, all without manual intervention from the sysadmins.

I would like to conclude this article with a brief word of thanks to Bernie Innocenti, who left the FSF for another job in September. We are extremely grateful for his contributions as an FSF sysadmin; he was instrumental in the server consolidation and Puppet setup efforts described here.