Administrate files, not systems
For as long as computer science has allowed us to build large complex networks out of various individual nodes, the problem of how to administrate the systems has always been trouble. There are many theories, approaches, and practices people use to try to overcome this simple problem, but sometimes there are strings attached to managing a large group of machines with minimal time and effort.
At the FSF, we have a very small (but talented!) technical team. To cope with this resource shortage, we subscribe to the paradigm of meta-system-administration. The concept is simple: instead of trying to administer all the machines individually, with the potential of creating unique snowflake issues, we manage a set of configuration files that define all the hosts on the network, actually limiting the amount of snowflakes to make a more maintainable network. This way, we spend our time managing a central store of configuration files and not each machine. In addition, it allows us to rebuild hosts on short notice since each host is fully reproducible.
To achieve this we use Fully Automatic Installer (FAI), and routine maintenance of our network. The FAI project has the added bonus of being written completely in GNU Bash, from the configuration files themselves, to the FAI client and server software itself. This makes debugging simple, since you can debug it as you would any other shell script. In addition, this means that any feature you think FAI is missing can be added by adding functionality to the configuration files, or even the Bash scripts.
We share our FAI configuration files with the entire world, at https://vcs.fsf.org/?p=fai-configs.git so others can use and modify the same configurations the FSF uses. By the same token, other organizations and individuals who also share their configuration files allow us to simply pick and choose the parts we want or need to add to our network.