Video Discussion: Unraveling the Mysteries of Cloud Pentesting

Matt Fiedler:

The title and topic today is "Unraveling the Mysteries of Cloud Pen Testing." We're going to talk about this from the perspective of both a pen tester trying to scope and deliver a good cloud pen test, and someone on the buying side thinking about engaging with cloud pen testing in any capacity. I'll be referencing AWS and Azure throughout, but the principles apply to all environments and cloud providers, so keep that in mind.

A little background on me: as Jack mentioned, I'm a solutions architect here at Evolve Security. My previous roles were in security engineering, where I focused on application, API, and cloud pen testing. I studied computer science in grad school at Johns Hopkins, focused on cybersecurity and quantum computing. And a fun fact: I was a professional baseball player in the St. Louis Cardinals organization — I spent three years in their minor league system and have a lot of fun stories from that.

For today's agenda, we're going to set the stage with the history of how cloud pen testing got to where it is today, talk about the general landscape of what we need to consider in the cloud, discuss how to define your pen test, cover a couple of technical details, and then talk about how to maximize the outcome of the engagement.

To set the historical perspective: cloud pen testing is a relatively new endeavor, especially compared to traditional network and application pen testing. Part of that is the wider adoption of the cloud, which is almost ubiquitous now, and part of it is how the cloud service providers — AWS, Azure, and GCP — approached it. Historically, up until about five years ago depending on the vendor, the service providers required that you, as a customer, submit written approval to engage in pen testing. That was a pretty big hurdle. On the pen tester side, they'd flag pen testing activities, block your IPs, or send warning emails. So it was really a pain to conduct pen testing in the cloud, and that's only changed within the last five years — more specifically the last two or three — as cloud providers opened it up to pentesters and allowed their customers to engage in true pen tests.

There was an interesting moment in 2018 — not that it was the first time, but it was an inflection point — where Rhino Security Labs published a meaningful blog called "AWS Privilege Escalation Vulnerabilities," and also published their full pen testing tool called Pacu, designed specifically for AWS pen testing. This might not have been the first, but it was one of the more notable points where there was demonstrated research that took advantage of not CVEs — not zero-day exploits or vulnerabilities specific to AWS infrastructure — but identifiable misconfigurations built into the intentional functionality of AWS. Then, maybe coincidentally, maybe not — it's never been stated, and I don't think it's necessarily an A-to-B thing — within the next six months, AWS changed their pen testing policy so you no longer had to submit a written request; they permitted any of their customers to engage in it. Whether or not it had to do with Rhino Security Labs, I'm not sure, but it marks a general shift in industry sentiment.

Over the past couple of years, I've seen the shared security model shifting. We're probably all familiar with it: the idea is that your cloud provider is responsible for security of the cloud — the infrastructure, hardware, and availability are handled and abstracted away from your responsibilities — while all the security in the cloud (the resources you deploy, your permissioning, the applications you build) remains the customer's responsibility. What I mean by "shifting" is that it's become more clearly defined that the cloud service providers do not want to be responsible for their customers' security. That's not a bad thing, and it's not to say they don't invest heavily in securing their own infrastructure and building security tools you can use. But we've seen more and more that they're eager to move away from the liability of security breaches in the cloud. There have been a bunch of notable ones, and almost without fail, the CSPs go on record saying, "This is not a failure of AWS, Azure, or GCP — this is a failure of the client to deploy it securely."

As that's been delineated, we've seen some positive developments. There's been a ton of cloud pen testing innovation over the past couple of years — if you search YouTube, all the meaningful cloud pen testing talks have probably taken place in the last two or three years. The knowledge sharing and tool development are happening at an increased pace, which is great. There's also a lot more awareness; anecdotally, organizations seem more and more aware of their responsibility to secure their cloud environment, which has changed even from five years ago.

As this has unfolded, it's also brought a lot of challenges. Cloud security complexity is really high — there's a ton of gray area. In a traditional architecture, you might have had an internal network with a DMZ clearly segmented from the external network, with well-defined components. The cloud blurs those lines: there are applications, resources, a dynamic set of external assets you might be deploying, internal assets changing all the time, Lambda functions. It's really changed the game in understanding where your security risk changes, and it's increased the attack surface.

On this screen, I have an example of an insecure AWS policy that would allow a privilege escalation to the administrator level. This demonstrates how it's not so much zero-days and CVEs that are exploitable in the cloud, but the intentional configuration within how you design and deploy your IAM and your resources. In this example, say you have a user in your solutions engineering group who doesn't have administrative permissions to anything, but has a policy applied that allows them to attach a group policy to their group. All they do is attach the administrator group policy to the group they're in — and now they're administrators. It's oversimplified, but this has nothing to do with how AWS designed their security, and everything to do with how the client configured security in their environment.

The final thing to consider in our lay of the land is how all paths seem to lead to the cloud today for pretty much every organization, and each of these pillars has different risks. Generally: we have our users, who are always going to be the weakest link in the security chain — so how do we design permissions securely, knowing this is always the easiest foothold? We have the resources we deploy — over 200 services is the current estimate of how many the major cloud providers offer, so you can build your architecture with these Legos, but that comes with a ton of complexity, because you might use 50 different services, each with intricate configurations, and you have to consider how they relate to each other. There's the idea of "data at the edges" — in a distributed work environment, we use distributed systems to collaborate, so a lot of sensitive data lives in things like SharePoint and Google Drive, and the surface for where we keep data has expanded. There's the hybrid environment, which is largely out of scope today, but it's well documented that APT groups use the cloud to establish persistence after compromising an on-prem environment, which has a lot to do with log ambiguity and the difficulty of correlating cloud logs with on-prem logs. And of course third parties — architectures are far more integrated with different services now, and how you store the API keys, credentials, and environment variables that go with those integrations really matters and is typically not designed for.

So that's the background of how we got here. The first and most important question to ask — whether you're the pen tester delivering this or engaging a pen test vendor to test your cloud environment — is: what are we actually testing? Architectures used to be more well-defined, where you could point and say "this is our external network, this is our internal network, this is our web app." Those have congealed into one thing. If you just say "I want to test my cloud environment," that's great, but it's sort of a meaningless statement — do you want to test your applications? Are you concerned with the users? With how you've designed the internal network? With what you're publishing to the external web? All of that?

The two things I drill into are the perspective we're testing from and the objective we're trying to achieve — and these go hand in hand. On perspective, there are two, maybe three, to consider. First, your public resources — the external network in your cloud environment. A lot of times a client says "we want to test our cloud," and when you drill in, they mean the websites they've deployed in AWS or Azure, an app they've hosted there, or their external network at large. That's a different set of testing than testing the internal network — where you're hosting VMs, EC2s, how you've configured your cloud environment, and the policies applied to your resources. Those are two entirely different perspectives.

I've put an asterisk on a third one, because I don't think it's a perspective specifically, but consider what layer of the stack you're testing — a SaaS layer or the infrastructure layer. The clearest example: do you want to test Microsoft 365 (your productivity suite — SharePoint, Outlook), or Azure Resource Manager (where you manage your VMs, compute, and configured resources)? It can get blurry, because Azure AD can be used for both, so sometimes you want to do both. Platform as a service is an even more complex thought exercise, because a PaaS tool is something like Salesforce, and you can't pen test Salesforce in the traditional sense of running the same set of exploits — though they do have a policy where you can submit a written request. So understanding whether it's SaaS, PaaS, or IaaS you're going after is useful.

As you consider the perspective, you also want to consider the objective, which is even more broadly applicable to pen testing in general. As pen testing has become more popular, people engage without ever asking themselves: what's our objective? Are we trying to meet compliance? Do I have a specific worry? Being able to clearly answer "why am I engaging with a pen test?" is super meaningful. As pen testers, we need to ask this of our clients before we ever start, because there's no one way to conduct a pen test — there are different elements you can focus on. Having a clearly defined objective before you start is super important, and even more so in cloud environments.

So, on perspective and objective: let's say we want to validate our external controls, identify gaps, and know what resources we're publishing to the web — or maybe we've written an application on the external network with all these users, and we want to test that. That leads you to a more traditional external network pen test or a web application pen test, which is honestly an easier conversation than a full cloud pen test. Especially if you're new to pen testing — meaning your organization is doing it for the first time — we almost always suggest starting here, because it's the easiest way to get your initiatives going, and these are valuable findings, since it's your highest-probability attack vector.

As we move up the security maturity roadmap, your next objective might be to assess the cloud environment itself: How are you deploying resources? What policies have you applied to your users? Are you doing simple things like enforcing MFA? Are permissions strictly scoped so not just anybody can publish a resource or spin up an EC2 instance? These can be found through a cloud configuration review — a cloud security assessment where you go through a set of predefined standards, mapped to the CIS benchmark. These are predefined misconfigurations we're all aware of that are common ways organizations make themselves vulnerable in the cloud. It's a great option because it tends to be a shorter, more budget-friendly engagement that gives you a good lay of the land, letting you harden those issues before moving on to more objective-based, assumed-breach-style pen testing — which is what we call our cloud pen test, and what our contemporaries in the industry call a true cloud pen test.

So after you've done external network pen testing on your cloud environment, done a cloud configuration review, and hardened it to the best of your abilities, you're ready for the next step: cloud pen testing, which is more of an assumed-breach scenario. You start from the perspective that something's been compromised — a user, an application — and you simulate true threat activities inside your cloud network, often tied to a specific objective.

Let me talk about what you might find from each style of test. On the external cloud pen test, it should feel very similar to an external network pen test — though the tools would be more specific to AWS or Azure. We're assessing your external footprint: enumerating all your external resources, asset and port discovery, traditional network scanning, and looking for common misconfigurations. What we tend to find at this layer: publicly exposed storage — it always amazes me, but some company has exposed a public S3 bucket or a blob with sensitive data; that happens all the time. Another — harkening back to "data at the edges" — is a ton of keys and tokens stored in GitHub, accidentally pushed to a public repo. Also keys, tokens, and APIs hard-coded into web application endpoints. Misconfigured cloud services — one we've seen a few times is AWS Cognito, a user management system, which had signup enabled by default (at least recently); if you can discover the requisite knowledge — basically a user pool ID and a user group ID — and the setting is enabled, you can register yourself for that application whether it's intentionally allowing registration or not. And weak password policies unbelievably still exist — weak passwords are probably still the number one way into environments. And unpatched software vulnerable to a certain CVE.

This is a great place to start, but when considering your long-term pen testing roadmap, this only scratches the surface of your cloud environment, because it's pretty unlikely that within a cloud pen test you'll actually gain a foothold into the environment. You might, but it typically excludes things like social engineering, and it's always time-boxed. So the likelihood of getting into the internal network — and then having enough time to assess that next layer of defense in depth — is unlikely. It's a great exercise to begin, but it's not looking at the cloud as a whole.

When we move to the internal side, this is where we as pen testers are given some level of access to the environment. If you're just starting, a great thing to do is a cloud configuration review. Your pen tester only needs a read-access account, so you're not giving them any major permissions. This works because the cloud is all API-based — so you can run automated scripts and tooling to query the API and get all the information about the cloud environment. It's time-effective and looks at the broad configuration of how you've deployed your cloud: Do you have MFA enabled? Logging enabled? Are you monitoring your systems? Is encryption enabled? Is your storage — buckets or blobs — publicly or privately exposed? It's a predefined set of misconfigurations we know about, and it's the first pass at hardening the environment. If you made an analogy to traditional pen testing, this is the vulnerability scanning phase compared to a true pen test — if you've never done a pen test, it makes sense to start here, because it lets you harden the low-hanging fruit before diving into the more simulated stress testing.

The cloud pen test then begins with that same configuration review — the first thing you do is enumerate the environment through those API calls, so you know the general configuration. But you're now also assessing the configuration of the resources themselves, contextualizing those findings and digging deeper into what the configurations between resources mean together. This can't be predefined — it's the reason there's no good automated solution for a cloud pen test, or really any pen test: you need the context of your environment and your risk profile for a pen test to be meaningful. Things we find here: a ton of permissive trust relationships you might never be aware of, cleartext environment variables everywhere, insecure assume-role policies and permissions that allow privilege escalation, and even things like permissions to deploy resources — you might not realize that somebody on the finance team has permissions to deploy an EC2 instance or a Lambda function. That's probably not intentionally designed, but easily happens as your enterprise environment grows. So the critical difference: the cloud configuration review looks at how you've misconfigured the cloud, and the cloud pen test looks at how you've misconfigured your resources within the cloud and how that all plays together.

When doing a cloud pen test, it's really important — as a client buying it, and as the pen tester — to be clearly set on your objective, because you won't be able to evaluate every single resource or explore every single attack path. You need an objective like: "We want a simulation where we have a compromised developer account, or a web application on the external network with IAM permissions tagged to it. We want to know what happens if that resource or user gets compromised, because we think our developers only have access to the dev environment and don't push code to production — so we want to make sure nobody in that dev environment can get to production," or "we want to make sure that if our web app is compromised, there's no way to get to our cardholder data environment." Because of the complexity, you can't test everything, and you need that north star.

On the workflow: you want to start from an initial compromise but have an objective. In traditional internal network pen testing, when you do an assumed breach, everyone knows you're going after DA — domain admin. In the cloud you can similarly aim to get to global admin, to control the whole environment — and that can be an objective. But I find that tends to be a proxy for "we have this set of data or systems that's business-critical and would be a disaster if accessed." That's when we start threat modeling: before the engagement begins, we want to know where the concerns are and what data is most sensitive, and then we do everything we can to get to that. That's the stress testing of the cloud pen test, because we're contextualizing the test to your environment, compared to the config review's predefined misconfigurations.

Here's a general mapping. Once we're inside — using the developer example — we start querying the API. The great thing is we don't need you to give us your /24 and start scanning; we query the API and it tells us where all the endpoints are, the IPs of the EC2 instances, what's internal, what's external. So when we move into the more traditional network scanning phase, we're not just blasting scanners everywhere — we know exactly where to point them. As we scan, we're also enumerating the IAM permissions, because IAM — your identity and access management, permissioning, and policies — is the central nervous system of your cloud environment across all providers. That's where everything goes, or where everything goes wrong. Those permissions apply to users, groups, and resources — your servers, applications, and Lambda functions can all have IAM permissions. So we enumerate and look for weird sets of permissions and relationships, because that's where you use "live off the land" techniques — taking advantage of the functionality AWS, Azure, or GCP provides and misusing it. That leads to lateral movement, assuming roles, or escalating privileges based on policies, all shooting for that ultimate resource compromise — data exfiltration, global admin, pushing code to production, whatever your golden ticket is. And this process repeats as you identify new opportunities.

Some things you might find in a cloud pen test — this isn't an overly technical talk, so I won't go too deep — but you're answering the question of "what if," the same as an assumed breach on the internal network. Let's assume one of your users gets phished, one of your applications gets compromised, or in the worst case a malicious insider starts going after the environment. Let's test what could happen.

One thing you might find is multi-step attack chains you'd never think of. A basic one: say your developer has no IAM permissions — you never want them to become an admin. But that developer can SSH into an EC2 instance because they need to for their work, and that EC2 instance has a set of IAM permissions; the application they're working on can assume the role of that EC2 instance, and that instance has admin permissions. Now your developer has admin permissions without a vulnerability anywhere in that chain. Another common one: say your application has a traditional vulnerability, a server-side request forgery. In the cloud this is particularly malicious, because almost all cloud resources — unless configured correctly — have access to a metadata server, which stores various IAM credentials. I've personally experienced compromising an application and then, because I could reach the metadata server, gaining access to a meaningful portion of the whole cloud environment. These are specific examples that can take any form, but you're trying to demonstrate impact against a defined objective — and that objective should be arrived at by the client, or by the pen tester and buyer working together to say "here's our threat model, let's go after it."

On maximizing outcomes: as I've engaged with more prospective buyers, it's a confusing topic. A few basic scoping things to consider — and you can do this as a buyer too. First, you really should have a clearly defined objective, which goes back to the perspective-and-objective model: what do we want tested, from what perspective, and why? If you can clearly answer those, it steers you definitively in the right direction.

Second, cloud pen testing really requires some level of complexity in the environment and some level of security maturity, because it's a more advanced set of testing. If you have a 15-person company where basically everybody has the same permissions and you're not doing complex things in the cloud, a cloud configuration review makes way more sense — because even if you find something in a cloud pen test, you'll spend a lot of money to maybe already know that you don't have a lot of permissioning going on. Cloud pen tests tend to make more sense for larger enterprises with a lot of different roles, teams working on different things, and a more complex threat model.

On scoping numbers — and this applies to all cloud providers — you want to know how many accounts there are. Taking AWS as an example, in the past everybody shoved everything into one AWS account; that's no longer considered a good security model, even with VPCs. Now we typically see accounts dedicated to different aspects of the business — maybe a finance account. And when I say account, I don't mean a user account, I mean the AWS account itself, sitting right beneath the OU (organizational unit). Organizations can now have 5, 10, 15, 20, or a hundred accounts, and when it scales, you need more strategic approaches — testing a hundred AWS accounts is good for you if you have the budget, but most likely nobody does, so you need to be strategic. It's also useful to know roughly how many services and resources are in those accounts — 50 resources or 5,000 resources leads to a very different amount of effort.

A general rule of thumb: at Evolve — and I think most of our contemporaries who are good pen testing organizations — we're not going to deliver a cloud pen test in under a week, and realistically it's more than that, maybe much more. It's a more expensive type of test, because you're asking somebody to drop into an environment they're entirely unfamiliar with, that's extremely complex, and then build out multi-step attack chains — that's not possible in a day or two. If your budget dictates a few days of testing, the cloud configuration review is a great option. It's important to be honest about the costs, because you'll find vendors willing to commoditize pen testing to a level where they offer very cheap services that aren't really a pen test. Being honest about what you're selling and buying leads to better outcomes, because expectations are aligned.

To summarize: the goals of a cloud pen test are to validate the preventative controls, design, and architecture you have in place. I wouldn't think of this as red teaming, where you're testing your detection controls — red teaming is a useful but notoriously expensive engagement, only right for certain organizations, and it's not the goal of a cloud pen test. You want to check that the way you've designed your systems, security architecture, and controls is actually preventing malicious behavior — not "is this showing up in logs?" There are a lot of common ways companies misconfigure the cloud environment, and that's where the cloud configuration review comes in. And there are more complex ways organizations misconfigure their resources within the cloud, and that's where the cloud pen test comes in. Then you get to stress-test realistic attack paths to answer that question of "what if we are breached — what could happen?" From both the pen tester and buyer side, the things to consider are the perspective and the objective. If you can't clearly answer those, I don't think you can deliver or engage in a good pen test — and a sub-consideration is where you're at on the security maturity roadmap.