“My job here at Atlassian is to commit crimes and then write very, very detailed confession letters – metaphorically speaking.”
Meet Alex: an engineer on our security intelligence team with a wry wit and a penchant for pop-color hair. Less metaphorically speaking, the team’s job (our red team, in particular) is to hack Atlassian’s systems exactly as real attackers would. Except, instead of selling any data they manage to “steal” on the dark web, they show our security and product teams how they did it, thus driving security improvements.
With more data and applications moving into the cloud every day, security is a prime concern for consumers and companies alike. That doesn’t mean we’ve all become experts overnight, though. There are things we know that we know, and other things that we know we don’t know – those are relatively easy to navigate. It’s the unknown unknowns that tend to “byte” the hardest.
So I asked Alex and the red team about some of the most common openings attackers use to exploit SaaS products, and how to guard against them. Here’s what they said.
Undocumented APIs
“One of my go-to hacking tools is Inspect Element,” Alex tells me. While there’s definitely a whiff of sarcasm coming through, it’s clear he’s only half-joking about this Inspect Element thing. Yep, literally the menu option you see when you right-click on a web page. It lets you access the developer tools that expose the requests the page was making, which means an attacker can look for calls to something other than the site’s own public API – either a private API or a 3rd-party API.
Sometimes responses from that other API will include stuff it really shouldn’t, like a customer’s information, or even the email addresses of all the site’s users. As if that’s not scary enough, access control errors like this can snowball into full-blown PR disasters if the names of an API’s endpoints are easy to guess. All it takes is for an attacker to try something simple like adding “/users/1” to the end of a URI. If that call returns a user’s profile, it’s only a matter of writing a simple script that calls for users 2, 3, 4, and so on … and boom: they’ve got everyone’s files.
These APIs may not be documented, but that doesn’t offer as much protection as you might think. “Private APIs are more likely to have mistakes in them,” Alex cautions. “They change all the time, so there are more chances for security problems to sneak in. Public APIs, by contrast, are less vulnerable because they tend to come under more scrutiny.”
For example, looking at API requests once lead Alex to exploit a consumer app that lets you find out which of your Facebook friends use that same app. Maybe they’d be OK with you knowing that. But maybe not. “There was no security bypassed or any hacking, really. It was just there.”
Exposing sensitive information in your code
If Alex or anyone on the security intelligence team wanted to hack someone in a malicious way – as opposed to the benevolent hacking that comprises their day jobs – they’d find a Git repo and start sifting through it, looking for passwords. “This is a common method of attack because there’s such a high return on investment,” they say.
SaaS applications are almost never entirely self-contained. They need to be able to connect with other applications, or other parts of your own network, from time to time. In other words, app developers write code that needs to be able to log into stuff like your AWS account or code that needs an API token to be able to access an external API. Which is all fine.
But where do you store the API token so the code can get at it? The simplest answer, of course, is to put it in your code. Or, if several parts of code need to access it, you might store it in a config file which you’d then add to your repository.
“The thing is,” Alex says, “Git is built for collaboration. Information stored in a Git repo inevitably ends up on the public internet.” Of course, most professional development teams set their repos to be private. But Git maintains a history of all changes throughout the repo’s life. So if it is accidentally (or intentionally) made public at some point, that history also becomes publicly visible. Even if you’ve deleted a file ages ago, it’s still Git – the repo history stays around forever.
“People absolutely dig back into a repo’s history and look for things that are supposed to be secret,” they tell me. “And there are programs that will automate it for you. Like, there’s a simple, publicly available script that you just point at someone’s GitHub account. It downloads all their repos and looks for all the secret stuff.”
So what’s a well-meaning, safety-conscious developer to do? At the risk of oversimplifying, don’t put secrets in your code. That said, don’t overcomplicate it, either. “To get real with you, security comes from stuff being simple. Vulnerabilities come from stuff being complicated.”
In some cases, an application-side password vault is useful. For the most part, however, this attack vector can be closed off by supplying API keys and other authentication info in environment variables and having the code access them through those.
Server-side request forgery using EC2
Known by its acronym, SSRF is a vulnerability that allows an attacker to abuse server-side applications by tricking them into making calls to systems the attacker wouldn’t otherwise have access to. By sending requests through a trusted server, attackers can access or manipulate sensitive data like users’ account information or purchase histories.
SSRF vulnerabilities are most commonly spotted by bad actors when the application uses request parameters containing full URLs, which they can then change to point to their intended target inside your system. If that server happens to be an AWS EC2 instance, things can get even worse.
“There’s sort of a magic address particular to EC2,” Alex explains. “And if you go to that magic IP address, it gives you metadata about the instance you’re on.” This then lets an attacker request even more privileged metadata, including security credentials. “Once they’re in, they can basically take control of the whole EC2 instance.”
Knowing when you’ve been hit with an SSRF attack is really tough unless you’ve got the right logging in place. AWS GuardDuty offers some built-in alerts, but the key is to have this and other alerts in place before an attack happens. Determining whether an SSRF attack has been performed after the fact requires some seriously fancy forensics (if it can be determined at all).
Although there’s no single iron-clad solution to SSRF, Alex recommends following the principle of least privilege as a starting point. In other words, by giving each layer of your application access to only the data and resources it really needs, you can limit the damage an attacker can do if they succeed in breaching your system.
Other recommendations from our security engineers include:
- When connecting to user-provided URIs, configure and use a proxy server that doesn’t have access to your internal network.
- If possible, set up firewall rules that prevent your public-facing application from connecting to internal locations.
- Consider configuring your AWS account to require IMDSv2, which is harder to access using SSRF than IMDSv1.
- Use a detection tool that looks for service-specific and injection-based vulnerabilities in your application stack.
“Nobody expects the Spanish inquisition!”
Before talking to Alex, I had no idea how unsophisticated – yet highly effective – many of the most popular attack techniques are. Some are so simple it seems like a caveman (or a writer) could pull them off. Which, when you think about it, seems like it could be exactly the reason they work so well.
Humans are prone to cognitive biases, logical fallacies, and other mental shortcuts. When it comes to protecting ourselves, we tend to focus on the more elaborate scenarios, while often forgetting the obvious ones. Every year there are stories of people who are burgled, despite the alarm systems on their homes, because they left the porch door wide open. It’s a good reminder that no attack vector is ever “too obvious.”
Stay vigilant out there, friends.