Supply Chain Backdoors, xz/liblzma, CVE-2024-3094, and what we currently know

This article offers a concise overview of the latest developments regarding the xz/liblzma vulnerability and its immediate repercussions within the bug bounty landscape. I strongly recommend exploring the referenced sources for a deeper understanding of this issue. Our insights are built upon the foundational work of many peers, who you can find linked in the sources section below.

What happened?

The discovery of a backdoor in the liblzma package, key to the widely-used XZ compression library, highlights a deliberate attack on the software supply chain, particularly within open-source ecosystems.

Andres Freund, a developer at Microsoft noticed failing ssh logins (by the usual automated attempts trying random user/password combinations) using a substantial amount of CPU, along with a longer than usual delay of 500ms on his SSH session, leading to him digging deeper into the issue. Upon profiling SSHd, it was evident that a significant amount of CPU time was being consumed by liblzma, but intriguingly, performance tools were unable to pinpoint any specific function responsible for this.In his own words below—this led to a series of elaborate coincidences, an incredible attention to detail, and insight, that led to him finding one of the most public displays of a motivated attacker to date:

Source: https://mastodon.social/@AndresFreundTec/112180406142695845

This investigation led to the groundbreaking revelation that the upstream XZ repository, along with XZ tarballs, had been compromised with a backdoor. His detailed technical analysis of the discovery is available for further reading, and goes into much more complete detail, here.

Timeline of events

One of the most intriguing parts of this discovery is that the backdoor introduction came from a maintainer of the XZ library, Jia Tan, who had contributed to the project for over two years, becoming a key maintainer and able to merge his own work, with this access he then proceeded to introduce a very elaborate backdoor.

Per https://boehs.org/node/everything-i-know-about-the-xz-backdoor the following timeline events is an excellently documented summary of what we currently know about the introduction of this vulnerability into XZ:

Source data https://boehs.org/node/everything-i-know-about-the-xz-backdoor

The exploit within the xz compression tool involved a series of manipulative contributions and patches by new, suspicious accounts over several years, leading to a backdoor that compromised the tool’s security. Through strategic pressure on project maintainers and subtle code changes, these contributors gained trust and inserted malicious code. This incident highlights vulnerabilities in open-source project management and the critical importance of thorough code review and maintainer support. It serves as a wake-up call for the tech community to reassess and strengthen the security practices surrounding open-source software.

Lasse Collin, the original maintainer of the XZ project, has since responded with a concise update on the XZ project incident, promising more information by the first week of April 2024. Currently, the XZ projects’ Git repositories are hosted on git.tukaani.org, following the removal of the xz.tukaani.org DNS name. This change has temporarily left the XZ projects without a homepage, an issue expected to be resolved shortly.

Notable, is the discovery of a backdoor within the XZ Utils 5.6.0 and 5.6.1 release tarballs, which were created and signed by Jia Tan. It’s important to note that tarballs signed by Collin currently appear to be unaffected, as they were personally created by him. In the wake of these revelations, the GitHub accounts of both Lasse Collin (Larhzu) and Jia Tan have been suspended, and the xz.tukaani.org subdomain, which was hosted on GitHub pages, is currently unavailable. Lasse mentions that plans are being considered to possibly migrate this subdomain back to the main tukaani.org domain.

Collin clarifies that only he has had access to the main tukaani.org website, its Git repositories, and related files, whereas Jia Tan’s access was limited to GitHub-hosted components, specifically the xz.tukaani.org subdomain (due to being hosted on Github pages). This incident has prompted additional scrutiny and discussions within the tech community, with further details available through a breakdown by Andres Freund, an FAQ by thesamesam, and ongoing discussions in Gentoo and Debian bug reports.

Who appears immediately affected by this?

Linux users

Affected by CVE-2024-3094 are systems running XZ Utils versions 5.6.0 and 5.6.1, specifically Linux distributions that have adopted these newer releases, including Fedora 41 and Rawhide, and possibly some macOS versions, which has also received security updates for Safari and macOS. Red Hat Enterprise Linux (RHEL) is not affected. Users of Debian’s testing, unstable, and experimental branches, as well as Kali Linux users who updated within a specific March window, should seek updates or downgrade to a secure version.

Kali Linux users

Kali Linux was impacted by the CVE-2024-3094 vulnerability within the xz-utils package versions 5.6.0 to 5.6.1, posing a threat between March 26th and March 29th, 2024. Users who updated their system in this period need to apply the latest patches. However, installations not updated during these dates remain unaffected.

Bug bounty hunters/customers

In the bug bounty landscape, this hasn’t had any immediate impact – certainly not to the scale of Log4J, and other industry wide events. Given the complexity of the findings, and how quickly it was identified, this does look likely to remain the case for the immediate future.

However, given the variety of operating systems in use by the crowd, and customers alike, it’s prudent to check if you’re vulnerable, using the below.

How can I check if I’m running the vulnerable library?

To protect your system against the CVE-2024-3094 vulnerability in XZ Utils versions 5.6.0 and 5.6.1, you should first determine your system’s current version by executing xz –version in the command line. If your system is running one of the compromised versions, CISA advises downgrading to the secure XZ Utils version 5.4.6 Stable. Here’s a practical example for Linux users:

Check your version: Open a terminal and type

for xz_p in $(type -a xz | awk ‘{print $NF}’ | uniq); do strings “$xz_p” | grep “xz (XZ Utils)” || echo “No match found for $xz_p”; done

(source: https://twitter.com/Kostastsale/status/1773890846250926445).

If your version is 5.6.0 or 5.6.1, proceed to downgrade.

For Debian-based systems, you might use:

sudo apt-get install xz-utils=5.4.6

On RPM-based systems, the command could be:

sudo yum downgrade xz-utils-5.4.6

Ensure to consult your distribution’s package management documentation for the exact command syntax. This precaution helps safeguard your system from unauthorized access attempts through this vulnerability.

While macOS users might not be directly vulnerable to CVE-2024-3094 due to the absence of systemd, it’s still wise to ensure the security of your system. macOS users should run brew upgrade xz to potentially downgrade or update xz to a version deemed safe by Homebrew’s repository, which could revert to the stable version 5.4.6, following CISA’s recommendations for mitigating this vulnerability. (brew update && brew upgrade –greedy && brew cleanup)

This proactive step helps maintain the overall integrity of your system against potential future exploits. This step is part of a comprehensive approach to maintain system security and mitigate risks associated with CVE-2024-3094.

Another way of quickly checking if you are running a vulnerable version, is to use the official detect.sh script from Openwall, link here.
Download it on the system you would like to check.

Run the following commands:
chmod +x detect.sh
./detect.sh

How did we respond at Bugcrowd?

Bugcrowd Internal Security & Engineering Teams

As any internal security team should, Bugcrowd used their internal Incident Response Plans (IRPs) to triage and appropriately validate whether any areas were affected by this supply chain issue, and whether there was any impact. Bugcrowd’s internal security team maintains a fully dynamic Software Bill of Materials (SBOM), including all transitive dependencies, for all code repositories and endpoints, allowing for prompt triage of all assets (instead of attempting to figure things out in retrospect). Based on this information, Bugcrowd was able to confirm that we were unaffected. We believe this type of preparedness will be beneficial for other teams too, so Bugcrowd intends to publish some of our SBOM code over time, in an open-source capacity (implying the code will be publicly available).

Bugcrowd Triage

As situations develop that look like they could eventuate in large scales of critical findings, we monitor closely, and ensure we’re developing collateral in real time. This served us well with Log4J, and this event had the potential for being of a similar scale.

Upon being aware of CVE-2024-3094, we expanded our monitoring efforts beyond our primary channels to include social media, forums, slack channels, and press coverage. This strategy was set to ensure timely and accurate identification and triage of any developments related to this vulnerability. In the event of relevant discoveries, our plan was to assign a specialized internal team with dedicated staff across multiple regions, aimed at providing swift and precise evaluation of any findings linked to this issue.

Bugcrowd Customer Service

As we coordinated our internal response to CVE-2024-3094, our technical customer service teams, including TCSM and Support, were also briefed and prepared with statements for customers inquiring about the issue. These communications were designed to inform customers about the steps they needed to take to check their systems, as well as to provide transparency about Bugcrowd’s own security measures in response to the vulnerability.

What happens next?

One of the consequences of this finding is that Jia Tan contributed not only to this library, but also others, leading to a larger review effort being needed to see if other components may also be in a currently vulnerable state. Beyond that, the question becomes how to avoid these issues in the supply chain – or whether they can be avoided at all. Open Source software is the backbone of many services, and companies that we rely upon – and not everything can be closely scrutinized to the level of detail needed to avoid these attacks.

We’ve outlined our sources below, where updates can be found (the most notable for this event currently being @AndresFreundTec and boehs.org). Alternatively, if you’re concerned about this vulnerability and would like to have the most up to date information that we have available, please don’t hesitate to reach out to your technical customer success manager, and they can provide you with a more recent update.

Sources

Follow Michael on Twitter @codingo_ to keep up with his latest content! Want to discuss this with the Crowd? Join our Discord and sign up for a Researcher Account to get involved!

Tags: