Sysadmin adventures: When weather threatens our work
You might be wondering, why in the world would an air conditioner be important during a blizzard? It's all about the servers. Most of our servers are not on site in our office, but those that are vital to the day-to-day work of the FSF are.
For me, snow is a welcome given of living in the Boston area, one that I missed in more than three years living in Guatemala and Mexico. Despite meeting some awesome free software comrades during that time, winter was always calling me back north. Home to me has always been Boston (I skip my youth because, well, New Jersey is worth skipping). I moved here in the fall of 1996 and experienced my first major snowstorm the following April Fools' Day, the biggest joke of the year. The temperature was in the 60s (Fahrenheit, the mid-teens in Celsius) for weeks prior, when suddenly we were slammed with 25 inches (63.5 cm) of snow in a twelve hour period—the biggest storm since the infamous Blizzard of 1978. I have always welcomed the snow in my life. After all, I was born in the blizzard that happened just a few weeks before the record-breaker in 1978, which nearly led to my mother being taken to the hospital by snowmobile. I like the snow and the challenges it brings to life, but I never thought it would affect my work as a sysadmin.
A view from the roof door, with the FSF air conditioning unit on the far end.
The weather throughout February brought high temperatures below freezing and weekly snow storms, amounting to piles of snow taller than me (sixty-six inches, or 1.676 meters). We received more than ninety-five inches (2.43 meters) of snow, cumulatively, and none of it melted in February. The last week of the month brought a new challenge, one I had never considered. Another snowstorm with high winds led to giant drifts of snow.
And then came the temperature warnings in our server closet. It incrementally went from 70 °F (21 °C) to 90 °F (32 °C). The weather was bad enough to severely delay—or shut down entirely—the public transit system, so travel to the office wasn't feasible for any of us in the Boston office. Over IRC, me and Lisa, our senior sysadmin, began identifying which servers we could turn off during the weekend to help lower the temperature in the closet. At this point, we had no idea why the air conditioning, crucial to temperature regulation in the server closet, was not working. It was not until Tuesday that we were able to make it back to the office.
Arriving by bike on Tuesday morning, I was ready to investigate the air conditioning issue. Thinking through the possibilities, I decided to turn the unit back on and see if it would stay on. That lasted about ten minutes before it automatically shut down. This left me with few options: either call the repair folks and hope they could come to the office that day or we could further investigate the issue myself by going to the roof, where ventilation part of the unit resides. Off I went.
This was not my first visit to the roof for an air conditioner issue, so I knew where to find the unit. I put on a sweatshirt and a keffiyeh and headed to the roof. The roof door locks from the inside, so I grabbed a piece of metal to wedge in the door to keep me from being locked out. I pushed the door to the roof and...it did not budge even an inch. Snow was piled up blocking the door. I leaned in, hard, and the door ever so slowly opened, leaving me just enough room to squeeze through.
But as I got out on the roof, the next challenge presented itself: four feet of snow, covering a veritable farm of air conditioning units—and ours was on the far side of the roof, forty snow-covered yards away.
To get there, I had to climb over other air conditioning units, duck below beams, and push through the snow. I finally arrived at our AC unit, and after some digging, I discovered that the snow had blown into the unit's cooling fan, preventing from spinning. I spent most of the morning digging and cleaning out the fan. I trekked back to the office, and the system started up again. Just one of the many quirks of a sysadmin's role here at the FSF.