35 years of freedom and memories from the FSF tech team
The FSF turned thirty-five years old this year, and for this fall's fundraising appeal, we took the time to do some digging into the past to bring you some of the key moments and fun anecdotes of the FSF's past in three parts, one focusing on collected stories centered around each of the three major areas of work at the FSF:
- The licensing and compliance lab handles GNU General Public License (GPL) compliance, fields questions from the public about licensing best practices, provides licensing education, and manages the Respects Your Freedom (RYF) certification program;
- The campaigns team advocates for free software, and helps spread awareness through writing, developing resources, and organizing the community; and
- The intrepid tech team supports the development of free software by providing a wide selection of services that support free software development; they also provide all of the FSF's own infrastructure, as well as the infrastructure for the GNU Project, and other free software projects.
This installment is focused on the history of what has come to be the FSF's tech team. The FSF started in 1985, and this series of articles is by no means an attempt to create a comprehensive linear history. Instead, I spoke to some of the former and current FSF system administrators and asked them for memorable moments from their time working with the FSF. This article lays out those stories and moments in the history of the technical infrastructure of FSF in a very simplified order. The two following articles in this series will highlight key moments in free software licensing and campaigning.
The wide range of FSF tech team responsibilities
The FSF's system administrators have always been exceptionally dedicated and skilled. The work is diverse and in-depth -- the team is involved in every aspect of the FSF's work. The team has steadily increased in size over the years, from employing a single systems administrator in the early years of 2000, to currently having four full-time staff. Throughout, the years there has also always been significant help from volunteers and interns.
Duties include maintenance, servicing, and expanding fully free server space; the technical needs of all of our Web sites; running audio and video for our LibrePlanet conference; and making sure the office and databases run smoothly. The tech team is also always on standby to assist the other FSF teams with new scripts, tools, and platforms to service new ideas, projects, or public activities. But they are not content with just maintaining what we have, so they continuously push forward with new projects, including, most recently, a major upgrade of the fsf.org Web site, and the development of our new software forge.
For thirty-five years, the FSF has been supported by people just like you, who give back because you believe in the idea of free software, and you trust that the FSF can make a difference.
Thank you for supporting the tech team's ability to sustain the community. Can you ask at least one other person to support the FSF as well? If each free software advocate like you inspires just one other, we can continue this work into the future. Please share this email and our appeal page with others, using the hashtag #UserFreedom on social media. If you're able to, anything additional you can contribute will enable the tech team to support the free software community for years to come.
This fall fundraiser, our goal is to reach 500 new associate members before December 31st. If you join today, you can select a special gift in addition to being able to enjoy all of our associate membership benefits, which include the free "as in freedom" videoconferencing server.
The early years of the FSF
For years, the FSF employed only a few people, and work on the GNU Project was done by them, with the help of volunteers, and sometimes with additional work by contractors. Most of the time was spent working on the development of elements of the GNU system. Richard Stallman explains the early work in a 1996 edition of the GNU Bulletin:
We started the GNU Project in 1984 with the aim of bringing such a system into existence. A Unix-like operating system consists of many components; we had to obtain each of the important components somehow. The job was so large that many of the people who sympathized with the goal were discouraged from attempting it, but we decided we would reach the goal no matter how long it took.
We found some components already available as free software -- for example, the X Window System and TeX. Naturally we decided to use them, since the job was big enough even with short cuts. We obtained other components by helping to convince their developers to make them free -- for example, the Berkeley network utilities.
The rest of the components, we had to write. These include Emacs, the GNU C & C++ compilers and libraries, Bash, Ghostscript, Groff, and many others.
In 1991, Linus Torvalds made the Linux kernel available. Together, the Linux kernel and the almost-complete GNU system resulted in a complete Unix-like free operating system suitable for actual use, the first fully free computer operating system -- GNU/Linux. The work continued: developers worked on another kernel called GNU Hurd, as well as on GNU components for the system, and other GNU projects. In 2001, Ututo became the first GNU/Linux distribution endorsed by the FSF. The FSF later developed the Free Distribution Guidelines, which are now used to decide which distributions earn the FSF's endorsement. You can read much more about this part of the FSF's history in the early Bulletin archives.
From pure development to project development
According to former executive director Peter Brown, over time, the organization's focus slowly switched from "pure software development to spreading the philosophy and tackling the growing threats to free software that came as result of its success."
The FSF made some significant strides launching new initiatives over a relatively short period of time, and the tech team delivered on a multitude of projects at a rapid pace. The associate membership program launched in 2001. This involved writing and deploying the AMT (Associate Membership Tool), which was written in Python using Zope, because at the time, there was no free software database system that met the FSF's needs. The FSF migrated to its own fsf.org Web site in 2004 and 2005, built in Plone. Before this time, executive director John Sullivan, who started as a staff member in 2003, remembers, "If you visited fsf.org in your browser, what you saw was the same as gnu.org." And in June 2008, the team finished work on the new GNU Press shop, the FSF's online store, which was based on code from the Satchmo project.
Organizing the infrastructure and making machines free
At the end of the 2000s, a lot of work went into the organization and freeing of servers and desktops from their proprietary BIOS. The sysadmins did some of the Coreboot ports for specific hardware themselves. It was hard to convert the servers because of the many random, non-uniform machines that were in use; some were very old, which meant each type of board needed a separate Coreboot port. Former sysadmin Nicolás "Nico" César remembers a time when they were at the colocation facility having to debug one of the server's BIOS. What was supposed to be standard maintenance turned into a six-hour struggle with this machine's BIOS, connecting different cables in order to get outputs and running into over 1,500 errors -- "this real time debugging for a BIOS, nobody will ask you to fix a BIOS in a server datacenter anywhere else. I learned a lot from debugging that."
Colocation spaces moved several times over the years, and at times, the servers were scattered over several locations. Eventually, new colocation space and power were donated in 2012, as well as the bandwidth, and the tech team spent eight days moving all the servers to the new location. This move allowed them to really start focusing on consolidating the server infrastructure.
Many weekends and nights went into sorting out all the old machines that were at the office and colocation spaces. Volunteers and sysadmins spent days and nights cooped up in the server space, transferring and destroying all the data stored on the different servers and disks that had been collected over the years. Andrew Engelbrecht, current senior systems administrator, was a volunteer at that time, and remembers, "With guidance from Nico, I backed up over 1TB of data from over 100 dying laptop hard drives, old RAID arrays, SATA, IDE, and even SCSI drives. Some drives had bad blocks, so I copied data that I could from their file systems, and used gddrescue to copy disk images as well. All of the data was written to a tape archive. I then wiped the drives clean, so we could hand them off to the recyclers."
Ward Vandewege, former FSF chief technology officer, who started his involvement with the FSF in 2002 -- first as a volunteer, then as an employee from 2005-2013 -- remembers, "Sometime in the late 2000s/early 2010s, I took a machine out of commission at the FSF offices that still ran on a Pentium Pro (a CPU first released in 1995). GNU/Linux had been happily running on that desktop machine for a very, very long time..."
What's a foundation without a database management system?
In 2014, CiviCRM was evaluated by the High Priority Projects (HPP) committee, who then decided that it satisfied the need from the HPP list for a donor database system. The tech team migrated all the FSF data from the AMT to CiviCRM. John Sullivan looks back at the transition as a real achievement and a pivotal moment for free software, "allowing the FSF to advance its mission unfettered." The FSF had an enormous database of contacts at that that time, and CiviCRM had not been used at that scale before. The tech team spent a lot of time and effort bringing the system up to the task of handling the enormous amount of contacts the FSF had collected over the years. Today, CiviCRM is not just workable, but I personally would not know how to do my job without it.
Current FSF chief technology officer Ruben Rodriguez could give me the dates the transition happened instantly, because the CiviCRM project had just finished when he joined the FSF. He had only contracted with the FSF before that time, helping out with issues with Trisquel, a distribution that has been on the FSF's FSDG list since 2009, and which, to this date, is the operating system running on the FSF's desktops, and the one our associate members receive on a special USB bootable member card. Just two weeks into the job, he was out at a family dinner with his wife and his extended family, when he was called to fix a server issue that was affecting the FSF database in CiviCRM. With contact information of members and donors at stake, Ruben had to hack and patch his way through the entire dinner, with his wife's grandparents silently judging him. "It was quite the introduction, but on the plus side, it made me feel right at home in the position," he says.
Streamlining the biannual mailing
Having a modern database system then also permitted a modern fundraiser and Free Software Foundation Bulletin mailing. The Bulletin is one of the oldest publications of the FSF, and it is sent out twice a year. Nico remembers this as a very big endeavor: "Before CiviCRM, there was a different system that had a lot of issues with scalability. We would be printing and trashing many letters before the mailing because of the export process, and we'd be writing scripts to merge huge amounts of letters. There were so many parts to it that there was a lot of waste." After migrating all the information, using the new system, they could create batches for targeting, which helped cut down the waste significantly during the printing and mailing process. The total preparation time for the process of targeting, creating groups, and mailing everything out went from around a month to just under a week.
What took up the most time was the manual folding of the letters. Volunteers and staff with papercuts were a norm at the office during fundraiser periods, and Nico sees it as a personal victory that they ended up getting a machine to fold the letters, saving a lot of physical pain and time. "It looked like Sputnik, this huge machine with a lot of cogs, and it made a lot of noise. It was so big and fun and it was a major improvement." The folding machine is used to this day, and we can't imagine a fundraiser without the clunking sound of the folding machine echoing through the office. Thanks to these optimizations, we currently mail out 12,500 individual items of mail twice a year.
LibrePlanet livestream adventures
One of Ruben's wishes since he started work at the FSF was to record and stream the LibrePlanet conference. When in 2016, Edward Snowden and Daniel Kahn Gillmor agreed to keynote, Ruben got what he asked for. Snowden could not physically be present, so the team was tasked with developing the free software for recording and streaming the session, which resulted in the now tried and tested livestreaming and recording setup using Icecast and Gstreamer.
The tech team prepared for this exciting opportunity for weeks, and they were ready to go in the Massachusetts Institute of Technology (MIT) Stata Center at 7:00am for a 9:45am start. Ruben remembers that morning vividly, especially the colorful sight of the Stata Center's red seats in the auditorium, filled to the brim with over 400 people looking down at him. Ruben sat center stage, hacking away, trying to get the system they worked so hard on to work, despite the Stata Center's issues with Internet, and feeling the pressure rise with every keystroke. He recounts, "if my chair had been pulled away, I would probably have stayed seated in the air because of how tense I was." He managed to get everything working just in time for one of the most memorable keynotes in LibrePlanet's history.
What we couldn't cover in this article
Many projects have been covered so far in this article, and it spans a long range of time, which leaves so much more to tell. I could go on by adding the tales of the upgrade to the current server stack, the launch of the RYF site, the freeing and expansion of more machines, or one of my favorites: the story of how Nico was able to pay for his lunch for months and months with Bitcoin he had bought from the FSF with his salary, feeling he made a really smart move, but running out entirely before later Bitcoin booms. Other memories that came up covered the excitement felt in the office when the team managed to light up an MIT building for the GNU anniversary, the strange mix of humor and devastation when one of our members replaced their full name with a snowman and broke the mailing (we now test with the snowman every fundraiser), and the challenge of spending five days getting a compromised Savannah back online.
As Ward Vandewege concludes, "I always thought of the FSF as an unusual organization -- very small in headcount, but with an outsized influence through the GNU Project. We got a lot done together, nevertheless!"
Thirty-five years of the FSF is worth a moment of celebration and reflection of the work done, before we continue our journey forward to achieve our free software mission and bring #UserFreedom to all computer users. I would like to extend a heartfelt thank you to the team that is always there to give a helping hand and write some code. Thank you FSF tech team members, current and past, and thank you to the volunteers and interns that have made all this work possible, for your tireless commitment to keeping the FSF (and our wider free software community) fully free and independent.
Illustration Copyright © 2020 Free Software Foundation, Inc., by
Raghavendra Kamath, licensed under Creative Commons Attribution
4.0 International license.
Photo 1 Copyright © 2019, Free Software Foundation, Inc., by Michael McMahon. Licensed under Creative Commons Attribution 4.0 International license.
Photo 2 Copyright © 2019, Free Software Foundation, Inc., by Rubén Rodríguez Pérez. Licensed under Creative Commons Attribution 4.0 International license.
Photo 3 Copyright © 2016, Free Software Foundation, Inc., by Kori Feener. Licensed under Creative Commons Attribution 4.0 International license.
Photo 4 Copyright © 2015, Free Software Foundation, Inc., by Nicolás César. Licensed under Creative Commons Attribution 4.0 International license.