Savannah and www.gnu.org downtime

UPDATE (2010-12-01): Savannah is now back online!

• Wed Nov 24 12:59 UTC -- On the evening before Thanksgiving, an IP located in Tbilisi, Georgia started an attack targeting the savannah.nongnu.org website. The perpetrators used SQL injection attacks to download the entire database of usernames and hashed passwords, and we should assume anything else in the Savannah MySQL database.
• Fri Nov 26 14:27 UTC -- At least one Savannah admin account was compromised by brute forcing the password.
• Fri Nov 26 16:02 UTC -- They then added themselves to the www project.
• Fri Nov 26 23:51 UTC -- Initial commit to the CVS repository: a hidden static html file which was automatically checked out a few minutes later by a cron job on the web server.
• Sat Nov 27 00:51 UTC -- Observing the successful probe, they proceeded to deface www.gnu.org.
• Sat Nov 27 01:35 UTC -- The attackers committed a PHP reverse shell script (ironically, GPL licensed) to the www repository and after a few tries found a directory where PHP was functional. That was not supposed to be the case, but it was. They then proceeded to try a ton of root kits on the gnu.org webserver. We don't think they succeeded in getting root, but they may have.
• Sat Nov 27 01:36 UTC -- We were notified of the intrusion.
• Sat Nov 27 01:37 UTC -- The GNU webmasters restored the website to its initial state.
• Sat Nov 27 04:42 UTC -- We implemented an emergency fix to the Savane codebase which blocked the particular script they were exploiting. We kept monitoring their activity to get a better handle on what they were up to, but at this time we were still unaware that they had full admin control on Savannah's web front-end.
• Sat Nov 27 19:05 UTC -- We noticed that the cracking activity had resumed on www.gnu.org through PHP reverse shells running as user www-cvs. Realizing that the problem was much worse than we assumed at first, we immediately isolated the Savannah cluster and the GNU website from the network and start a deeper analysis.
• Sat Nov 27 21:35 UTC -- We then spent most of the Saturday afternoon and evening to restore www.gnu.org and a few secondary domains on a brand new machine. In the process, we cleaned all the bogus commits out of the www CVS repository. The jobs that automate the integration with Savannah are currently disabled.
• Mon Nov 29 15:23 UTC -- This morning we started reconstructing the Savannah machines. Currently the only thing we know for sure is that the attacker basically could impersonate another Savannah user. The web frontend system was otherwise up-to-date, including the kernel, so if the user could gain root access, then he could probably do that on any Debian Lenny system around. Nevertheless, to avoid taking any chance, we decided to reinstall the system entirely.

Looking forward, the Savannah hackers are currently at work to restore the services, starting with the repositories. We'll restore backups from the 24th in read-write. Owners of git and bzr repositories can easily push the lost commits from their local copy. CVS and Subversion users are encouraged to recommit the recent changes; we can provide a read-only copy of the 27th backup on demand for inspection.

At this time we have no projection of when the full site functionality will be restored. All the unsalted MD5 passwords stored on Savannah are to be considered compromised and will have to be reset before the accounts can be re-enabled. The encrypted password scheme will also be upgraded to Crypt-MD5 (/etc/shadow's), and user password strength will be checked.

We are very sorry for the inconvenience caused to Savannah users by this attack, and thank you for your patience while we work to bring back an improved service. We would have liked to share all of this information earlier, but it took some time to reconstruct the incident and assess the extent of the compromise. To monitor the progress in restoring Savannah, check http://savannah.gnu.org and http://identi.ca/group/fsfstatus. We will also post any major updates here.

If you have any questions, or would like to help us with the recovery, subscribe to the the Savannah Hacker's mailing list:

http://lists.gnu.org/mailman/listinfo/savannah-hackers-public

For security-sensitive communication, please contact sysadmin@gnu.org.