Skip site navigation (1) Skip section navigation (2)

FreeBSD.org intrusion announced November 17th 2012

Security Incident on FreeBSD Infrastructure

From: FreeBSD Security Officer <[email protected]>
To: FreeBSD Security <[email protected]>
Bcc: [email protected], [email protected]
Reply-To: [email protected]
Subject: Security Incident on FreeBSD Infrastructure

On Sunday 11th of November, an intrusion was detected on two machines within the FreeBSD.org cluster. The affected machines were taken offline for analysis. Additionally, a large portion of the remaining infrastructure machines were also taken offline as a precaution.

We have found no evidence of any modifications that would put any end user at risk. However, we do urge all users to read the report available at http://www.freebsd.org/news/2012-compromise.html and decide on any required actions themselves. We will continue to update that page as further information becomes known. We do not currently believe users have been affected given current forensic analysis, but we will provide updated information if this changes.

As a result of this event, a number of operational security changes are being made at the FreeBSD Project, in order to further improve our resilience to potential attacks. We plan, therefore, to more rapidly deprecate a number of legacy services, such as cvsup distribution of FreeBSD source, in favour of our more robust Subversion, freebsd-update, and portsnap models.

More information is available at http://www.freebsd.org/news/2012-compromise.html

Saturday November 17th, 2012

Table of Contents

Update: December 29th, 2012

With the exception of systems relating to the building and testing of packages, all FreeBSD.org infrastructure has now been brought back online. A full audit of the third party package build infrastructure code ("pointyhat") and package testing infrastructure ("redports") continues, and neither system will be brought back online until audits are complete.

As a result, FreeBSD 9.1-RELEASE will be published with only minimal i386 and amd64 (x86_64) precompiled package sets available, and with no packages available for other architectures. This package set will be available on the DVD image, and are sufficient to install either the GNOME or the KDE desktop environment. For any other uses, or for any packages not included on the CD, either using the most recently available -stable package collection or compiling ports from the ports tree are recommended. Packages for 9.1-RELEASE will be made available at a later date. Instructions for obtaining and updating the ports tree can be found in the FreeBSD Handbook.

Unless there are any other major changes, this is planned to be the last status update to this page. An email will be sent to the FreeBSD announcements mailing list when the package build infrastructure is online and packages are once again available.

Update: November 27th, 2012

Due to the legacy third-party package build controller head nodes being offlined pending reinstall, we have been unable to build new package sets over the last two weeks. As a result, FreeBSD 9.1-RELEASE has been delayed as it was felt that we should not ship the release without at least a minimal package set available. We are now in a position where we are once again able to build third-party packages for both of our Tier-1 architectures (i386 and amd64), and are planning on releasing it within the next few days with only a slightly limited set of packages. Please note that historically we have also provided packages on a best-effort basis for some of our Tier-2 architectures such as sparc64, ia64 and powerpc. We are not currently expecting to be in a position to build any Tier-2 packages before FreeBSD 9.1 ships, so initially no precompiled packages will be available for these platforms. We may be in a position to provide some packages for these architectures shortly after the release.

A few reports covering this incident on external tech news websites have confused details relating to how this incident was discovered. Over the last few weeks, many of our primary cluster servers have been either physically relocated and/or replaced with new hardware as part of work planned several months in advance. The discovery of this incident was unrelated to this ongoing cluster maintenance. Several service outages in the days surrounding the incident were correctly attributed to ongoing cluster work, and were not related in any way to the compromise. In parallel with the physical upgrades and relocation of servers, we are also reworking the network layout in order to provide better functionality, security, resilience, and to reduce any impact from incidents such as this. Due in a large part to the progress already made here, we were able to have full confidence in many systems and services so quickly after the compromised hosts on the legacy network segment were discovered.

Update: November 22nd, 2012

Although not mentioned in the original report, CTM (another mechanism for retrieving FreeBSD source) uses the master trusted Subversion repository as the source of its data. Additionally, verification of CTM-sourced trees has been completed against the Subversion tree, confirming that there are no differences between the two. Our experimental Git repository has been similarly verified.

Work continues on rebuilding internal infrastructure and reinstating services taken down during the incident. Web interfaces to the old CVS repositories (CVSweb), and to GNATS (our bug-tracking database) have been restored amongst other services, and other internal hosts are being examined and rebuilt where necessary. A full audit of the package building infrastructure is ongoing.

The FreeBSD Project is investing significant effort into looking into both medium and long term infrastructure improvements to increase security of the FreeBSD cluster.

Update: November 18th, 2012

Newer portsnap(8) snapshots are once again available. The generation of these had been suspended as part of the infrastructure lockdown, however all machines involved have either been audited or reinstalled and so we are now confident that these can be made available once more.

The Subversion to CVS exporter is now up and running again. Updates made to the Subversion repository will once again appear in repositories available via csup/CVSup. Please note that the use of these exports are still deprecated, and users are urged to move to one of the supported methods (for example, freebsd-update(8), portsnap(8), or Subversion) in order to obtain updates. Note also that we are still currently unable to guarantee the integrity of past history within the CVS repository, but are confident in the integrity of checkouts from the top-of-tree of each branch.

Please note that due to infrastructure changes, the first update through either portsnap(8) or csup(1) is likely to show changes to a large number of files. This is nothing to worry about.

As mentioned in the original announcement, a package set uploaded in preparation for the upcoming FreeBSD 9.1-RELEASE could not be verified, and so was removed. In order to allow system integrators and end users to verify that packages they may have downloaded are not from this set, we have provided files containing both sha256 and md5 checksums for all removed packages.

November 17th, 2012

Initial details

On Sunday 11th November 2012, two machines within the FreeBSD.org infrastructure were found to have been compromised. These machines were head nodes for the legacy third-party package building infrastructure. It is believed that the compromise may have occurred as early as the 19th September 2012.

The compromise is believed to have occurred due to the leak of an SSH key from a developer who legitimately had access to the machines in question, and was not due to any vulnerability or code exploit within FreeBSD.

To understand the impact of this compromise, you must first understand that the FreeBSD operating system is divided into two parts: the "base" maintained by the FreeBSD community, and a large collection of third-party "packages" distributed by the Project. The kernel, system libraries, compiler, core command-line tools (e.g., SSH client), and daemons (e.g., sshd(8)) are all in the "base". Most information in this advisory refers only to third-party packages distributed by the Project.

No part of the base FreeBSD system has been put at risk. At no point has the intruder modified any part of the FreeBSD base system software in any way. However, the attacker had access sufficient to potentially allow the compromise of third-party packages. No evidence of this has been found during in-depth analysis, however the FreeBSD Project is taking an extremely conservative view on this and is working on the assumption that third-party packages generated and distributed within a specific window could theoretically have been modified.

What is the Impact?

If you are running a system that has had no third-party packages installed or updated on it between the 19th September and 11th November 2012, you have no reason to worry.

The Source, Ports and Documentation Subversion repositories have been audited, and we are confident that no changes have been made to them. Any users relying on them for updates have no reason to worry.

We have verified the state of FreeBSD packages and releases currently available on ftp.FreeBSD.org. All package sets for existing versions of FreeBSD and all available releases have been validated and we can confirm that the currently available packages and releases have not been modified in any way.

A package set for the upcoming FreeBSD 9.1-RELEASE had been uploaded to the FTP distribution sites in preparation for 9.1-RELEASE. We are unable to verify the integrity of this package set, and therefore it has been removed and will be rebuilt. Please note that as these packages were for a future release, the standard pkg_add -r tools to install packages could not have downloaded these packages unless they were requested explicitly.

We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources.

We can confirm that the freebsd-update(8) binary upgrade mechanism is unaffected, as it uses an entirely separate infrastructure. We have also verified that the most recently-available portsnap(8) snapshot matches the ports Subversion repository, and so can be fully trusted. Please note that as a precaution, newer portsnap(8) snapshots are currently not being generated.

What has FreeBSD.org done about this?

As soon as the incident came to light, the FreeBSD Cluster Administration team took the following actions:

  • Power down the compromised machines.
  • Power down all machines on which the attacker may have had access.
  • Audit the SVN and Perforce repositories to:
    • Verify that there had been no server intrusion.
    • Verify that no malicious commits had been made to the repository.
    • Verify that the SVN repository exactly matched a known-clean off-site copy.
  • Verify that all FreeBSD base release media and install files on the master FTP distribution sites are clean.
  • Verify all package sets available have checksums that match known-good copies stored off-site.
  • The package set built for the upcoming 9.1-RELEASE did not have an offsite backup to verify against. These have been deleted, and will be rebuilt before 9.1 is released.
  • All suspect machines are being either reinstalled, retired, or thoroughly audited before being brought back online.

At this time, we recommend:

  • If you use the already-deprecated cvsup/csup distribution mechanisms, you should stop now.
  • If you were using cvsup/csup for ports, you should switch to portsnap(8) right away. Ports developers should be using Subversion already. Further information on preferred mechanisms for obtaining and updating the ports tree can be found at http://www.freebsd.org/doc/handbook/ports-using.html
  • If you were using cvs/anoncvs/cvsup/csup for src, you should consider either freebsd-update(8) for signed binary distribution or Subversion for source. Please see the chapter on updating FreeBSD from source in the handbook. Further details on using Subversion and a list of official mirrors can be found at http://www.freebsd.org/doc/handbook/svn.html
  • If you use portsnap(8), you should portsnap fetch && portsnap extract to the most recent snapshot. The most recent portsnap(8) snapshot has been verified to exactly match the audited Subversion repository. Please note that as a precaution, portsnap(8) updates have been suspended temporarily.
  • Follow best practice security policies to determine how your organization may be affected.
  • Conduct an audit of your system that uses FreeBSD.org provided binary packages. Anything that may have been installed during the affected period should be considered suspect. Although we have no evidence of any tampering of any packages, you may wish to consider rebuilding any affected machine from scratch, or if that is not possible, rebuild your ports/packages.

If you have any further questions about this announcement, please contact the [email protected] mailing list, or for questions where public mailing list distribution is inappropriate, please contact the FreeBSD Security Team.

This page will be updated as further information is known.