What's up with lists?
The FSF would like to apologize for the recent problems with lists.gnu.org. lists has been suffering from severe hardware problems recently, causing significant delivery delays. Combined with a failure in the TMDA setup over the last few days, a serious mail loop developed between the FSF mail server and the list server, resulting in over 900,000 messages in lists' mail queue on March 1. The sysadmins identified and eliminated over 800,000 of these duplicate messages - however, by March 2, it became clear that due to contention for disk access, lists would never be able to process the backlog - as the number of messages lists was processing per minute was exceeded by the number of incoming messages. On the evening of the 2nd, the systems administrators acquired a high performance SCSI array for lists use, and transferred the queues to it. As a result of this move, we are now confident that lists will be able to properly work through the backlog, and that list delivery will return to normal (and even improve).
When lists.gnu.org was put into service three years ago, a uniprocessor Athlon XP 2600+ with 1GB of RAM. was more than sufficient to handle our needs. In the last year, however, the demand for lists and the amount of mail it processes has skyrocketed beyond anything we expected. lists currently hosts 2,175 mailing lists, and routes between 200,000 and 250,000 messages per day. It also hosts web-based Mailman admin access for all 2,175 lists at both gnu and nongnu.org, and the archives for these lists. It is a very, very busy machine - its load is regularly over 10, and sometimes as high as 30.
Although the usage of much faster SCSI disks has helped significantly, the systems administrators are continuing to look for a higher-performance machine to house lists temporarily, while the mail system is re-engineered - the Foundation has purchased two new quad Opteron machines for this purpose, with very fast disks.
The FSF Systems Administrators
(Justin Baugh, Joshua Ginsberg, and Ward Vandewege)