Video Talk: Bad Penetration Testing

Jack Ekolof:

We'll get things officially started. I'm Jack Ekolof, and I'm MCing this webinar today in coordination with Colleen and Paul Petefish. Paul is Founder and CEO at Evolve Security, and he's going to be talking today about a fun topic: bad penetration testing. We're grateful for everyone's attendance, and look forward to kicking things off.

Paul Petefish:

Thanks, Jack. Greetings, everybody, and thanks for joining us this afternoon. I'm coming to you live from our Chicago headquarters and the home of our offensive security operations center at 123 North Wacker.

The title of the talk is "Bad Penetration Testing." What drove the want to have a talk like this is that I, our Vice President of Offensive Cyber, and Jack all speak with a lot of customers and prospects who've had really bad experiences with penetration testing — their results, the methodology, a number of things. So we're going to talk about that, share our experiences, and cover what we believe you should be looking for when choosing an offensive security or pen test vendor. We're also going to talk about the CVE problem, and how pen testing is evolving into less of a point-in-time and more of a continuous model. I want to very much encourage engagement and participation from the audience.

One reason I believe we see so much bad penetration testing — bad methodologies, bad reporting, a number of issues — is that the business has a low barrier to entry. Anybody with a laptop and some understanding of pen testing can hang their shingle and start a pen test business. So you see a lot of that, along with other issues we'll get into.

A little about myself: I'm Paul Petefish, CEO and co-founder of Evolve Security. I've been in cybersecurity for about 20 years. I'm a technical CEO — I started off in a defensive SOC many years ago for a company called Solutionary, a first-generation MSSP. After about six months, I was recruited into Solutionary's attack and pen team — I think I was the second or third person to join it. I stayed with Solutionary until the end of 2015, after we were acquired by NTT (Nippon Telegraph and Telephone), made sure that was a smooth transition, and then started Evolve Security. It's been a great ride, and I learned a lot at Solutionary — it informed my thesis on the future of offensive security, and I'll come back to that.

So, we have a CVE problem. CVEs — Common Vulnerabilities and Exposures — are a nonprofit, government-run database that tracks known technical IT vulnerabilities in software, infrastructure, applications, and operating systems, or at least commonly used applications. A couple of things: first, you'll notice that starting in 2016, the count of CVEs grows exponentially year over year — at the end of last year we closed at about 30,000 new CVEs, ranging from low risk to critical. Second, these are known vulnerabilities reported to the CVE body. Rob Kraus and I have co-authored a couple of CVEs, so we know the process — but there are a lot of vulnerabilities that aren't reported, which would be considered more like zero-day attacks. Keep this in the back of your mind, because it's a striking representation of the acceleration and velocity at which new vulnerabilities are being discovered every year, especially since 2016.

Traditional pen testing has always been a great place to start. We do a lot of it here at Evolve Security. This isn't a discussion about how pen testing itself is bad — it's not. It's a great place to start, and a very pragmatic way to verify the security controls of your tech stack. The caveat is that it's point-in-time. If I ever took over a tech stack as an IT manager, director, or CISO, I would most certainly begin with a pen test to get the lay of the land and understand the security posture — it's a quick, pragmatic way to do that. Likewise, if I were acquiring a technology or merging with a company — which we support a lot with pre-acquisition validation — a pen test helps the acquiring company understand what they're getting into, baseline it, and inform future security investment and remediation.

Now, a couple of problems with traditional pen testing. One-time-a-year pen testing likely gives an inflated sense of security. That's my biggest concern for leadership — it's sometimes looked at as too much of a "check the box," which gives a false sense of security. You're really only as good as your last pen test. If you're going 340 days between cycles — since it takes a couple of weeks to do the field work — that's a long time, especially in a dynamic, changing environment like the cloud and modern app-dev cycles. Second is variability of delivery and resources. We hear this a lot: "I didn't get such a good resource the last couple of times." Pen testing teams have specialists, and based on availability, you'll get a different perspective — which is good sometimes, but also introduces variability in the delivery approach. And as I mentioned, cloud infrastructure and applications are constantly changing, so validating their cyber sturdiness once a year doesn't make a lot of sense. It's better than nothing, and I'd always encourage at least traditional annual pen testing, but there are other options.

Let me cover the types of pen testing that are common right now. First is the audit or consulting style — that one-time "state of the nation" test that's really more of an audit. I don't mean a compliance audit; I mean you're going through a methodology and somewhat of a checklist to validate that certain controls are in place, and if something looks interesting, you go down that rabbit hole to chain vulnerabilities and misconfigurations together. Second is pen testing as a service, a newer, retainer-based model that works well for flexibility. We've seen it work well with large organizations that don't have a well-defined scope but need to get started — they can pull from the retainer, with the flexibility of quick starts. It also works well for applications where you need small buckets of testing for new features, without a full pen test cycle against the entire app every time.

Third is bug bounty, which I define as focused, hardcore vulnerability hunting and exploitation — an outsourced group of highly skilled people (probably the 80/20 rule: 20% are highly skilled, 80% mid-level) who are paid for what they find and go find really interesting, hard-to-find things. Bug bounty is good for sophisticated organizations with large cyber budgets or a very high risk profile. Fourth is red teaming, which gets confused with pen testing a lot — a red team is usually two or more individuals spending months going wide across the entire scope of the organization, very slowly, trying not to get caught while gaining an initial foothold, moving into exploitation, maintaining the foothold, and then writing a thorough narrative of how it unfolded. It's very expensive and takes a long time, but has its place with mature organizations that have high risk profiles. And last is the continuous model — an always-on recon and discovery, probing IT infrastructure and assets for new vulnerabilities in near real time, validating them to ensure they're real issues, and then ranking the actual risk appropriately.

Now let's talk about choosing a penetration test company. Here's what I'd do personally. First: is offensive security — specifically penetration testing — their focus and the majority of what they do? Or are they a larger managed security services provider with a group of pen testers, but whose main business is operating a SIEM or log monitoring? That's not a bad thing — some MSPs have good pen test teams — but I always want to start with the expert in whatever I'm acquiring. Second: methodology. Do they have a strong, defensible approach? There are very detailed open-source methodologies available, but every pen test company should have their own version. They're all about the same, give or take, for traditional pen testing. If anyone wants to see what a good, well-thought-out one looks like, reach out to Jack Ekolof and he can share Evolve's as an example of a good baseline.

Third: the testers and engineers. I'm a firm believer that pen testing is still a very human-run process, and there has to be a human element — I'll talk more about that later. Where are the testers based? Are they W2 employees or 1099? Are they trusted, vetted 1099? Are they onshore or offshore? You get what you pay for. Not saying there aren't great offshore providers, but more often than not, the cheaper options give you less testing and a less-good outcome.

Jack Ekolof:

I'm going to pause you here — we've got a question from Justin.

Audience question:

On the previous slide, you were talking about the different types of pen testing. I was curious: do you see them as exclusive, where you just choose one, or would you get a mix of them?

Paul Petefish:

Good question — I think you need a mix. For example, you might choose pen testing as a service in a continuous model, or go with a continuous approach using attack surface management, and then add a red team or bug bounty on top once or twice a year for that deep dive. So it's not exclusive. I think you need a continuous, always-on approach — and there are two ways to get that right now — plus a deep dive, and I think red team and bug bounty will serve a purpose in the future.

Fourth: reporting. Is the org offering a dynamic portal with vulnerability lifecycle management, tracking over time, and remediation tracking? Or just static PDFs? Rest in peace to PDF reporting on pen tests — that's one of the things that drove us to create a dynamic web portal, because it allows for so much more flexibility, communication, and tracking. You could argue that a one-time hardcore red team is more likely to have a static narrative report, and that's fine, because it's more of an audit style. I'm talking about getting away from audit style as your only way of testing, toward a dynamic, continuous approach — and ideally a blend.

Fifth: communication. Is it traditional — over email, maybe a kickoff call, and then you don't hear from them until a report is available, and then you're off on your own? Or are there real-time, consistent updates through a portal or chat ops — a private Slack or Teams channel? The more sophisticated and larger organizations we work with like that continuous, open communication channel. It's also a stepping stone to more purple teaming — working with their blue team and testing their ability to detect and respond.

Jack Ekolof:

I'll jump in with a question from the chat. This person didn't want to speak out loud, but they said they haven't had a penetration test yet, and their peers say they should ask for a red team. Should they start with a red team?

Paul Petefish:

The answer's no — unless you have a lot of money you're trying to get rid of. If you haven't had a pen test, you need to walk before you run — or, a better analogy, you need to train before you jump into a marathon. A red team will quickly infiltrate and exploit, but you're not going to get the value out of it. You'll get a lot more value starting with either point-in-time testing or a continuous approach. Great question.

Sixth: technology and automation. Do they have tech that strengthens their approach and focuses their manual effort? Is it an individual with a laptop, a known scanner, and a couple of tools on their Linux hacking distro — or is there real automation and automated validation that helps the testers spend less time on the monotonous stuff a machine can do, and more time on the manual stuff a machine can't? Seventh: what are their key performance indicators, and how do they communicate them? Is it strictly focused on vulnerability counts — how many we find — or on time to find and time to fix? We focus a lot on mean time to find and mean time to fix: how long did it take to find the vulnerability, and, just as important, how long to fix it? And that leads into the last one, remediation testing. If the company isn't discussing or offering remediation testing — whether it's an add-on or rolled into the package — that's a problem. It's not just about finding the issues; you have to go fix them. You need a partner focused on leaving you in a better security posture, not just the fun of finding things and getting domain admin.

Jack Ekolof:

One question, Paul — you talked about mean time to find and mean time to fix. How can someone communicate that to the rest of the business, so non-IT folks understand the value of reducing it? What does it mean to the business?

Paul Petefish:

It comes down to risk to the business. I always use the physical analogy in cyber: if I know the front door of my business is unlocked, or the lock is broken, I have higher risk if that lock has been broken for 300 days and no one found it, versus finding it within a week and fixing it within a week. If it took 300 days to find and another 100 days to fix, that's far more risk — potentially very impactful, depending on what happens because of that broken lock — versus tightening and lowering our risk profile by finding that broken lock within a week and fixing it within a week.

Let's talk a little about cloud and application testing. First, it's only getting cloudier — we rarely talk to an organization without some cloud footprint. You may think you don't have any instances in Azure or AWS, but I guarantee you have M365, O365, or G Suite handling your email — that's cloud infrastructure, more SaaS-based than infrastructure-based, but extremely important to the business. We always encourage customers to look at their core cloud infrastructure and providers — email, file share — which almost always comes back to Microsoft and M365. Cloud infrastructure is one of the most dynamic things in the IT environment; there are a lot of settings that change often. You may be adding or removing users daily or weekly, and if we miss multifactor on one of those users — especially an admin — that's a risky proposition.

On cloud testing and scope, we look at it in two parts. First, the admin console for your cloud provider, which has the basic settings and security configurations — when I log into my AWS admin console, that's what we're testing, looking for best-practice configurations: Is multifactor on? Are anonymous shares or anonymous S3 buckets turned off? Second, network instances and microservices, which goes back to traditional pen testing. If I have a server farm running Linux in an AWS environment with a bunch of servers, cloud pen testing would include going into that environment and testing those instances, just like if they were in a data center. Microservices are similar, but instead of a full server with all ports and services, we know we're dealing with just a database or file-hosting service — so it's a more specific attack methodology, but the same mindset.

Jack Ekolof:

We have a question on cadence: "My budget doesn't allow for multiple pen tests. I perform an external pen test annually and I have internal controls. Do I really need to test my internal network?"

Paul Petefish:

Yes — because it's a trust-but-verify exercise, and you don't know what you don't know. Nobody purposely misconfigures a firewall, or purposely doesn't enable a security feature, or purposely misconfigures endpoint monitoring. These are typically oversights or a lack of knowledge about where the most impactful place to put security dollars is. So whether it's internal, external, cloud, or application, it's all part of the IT scope and should be tested on a continuous, consistent basis. From a budgetary perspective — five years ago, a continuous model really didn't exist well, because vendors were selling a set of cycles: buy 4, 6, or 12 pen tests. Now we're getting into true continuous pen testing, but it's cost-prohibitive — and that's where attack surface management comes in. It provides an ideal blend of consistent monitoring with a machine, which keeps costs low, and then once something suspicious is identified, the human picks it up, validates it, verifies it, and exploits it if possible.

Jack Ekolof:

We've got another question.

Audience question:

I was curious how often you'd suggest performing a pen test in a zero trust environment. Is it the same as a traditional environment?

Paul Petefish:

I think it depends. In that case, you're really looking more at the implementation and validation of that zero trust and those controls — the architecture, how it was implemented, and whether it's been verified to truly function the way you expect. It's a different approach and methodology. Great question.

Jack Ekolof:

We've got a question around cloud infrastructure.

Audience question:

If you have a managed service provider managing your infrastructure, does your system require agents to run on those servers? We've run into problems where our MSP is a shared tenant with their other customers, so they've been reluctant to put agents on the infrastructure. Do you guys run into that?

Paul Petefish:

Evolve Security does not use an Evolve-specific agent to provide its pen testing or continuous attack surface management offering. In certain types of pen testing methodology, like assumed breach, we do use C2 (command-and-control) agents. It's a good question — if you want to learn more about how we specifically do our tech and methodology, I'd love to follow up with you one-on-one, or with Jack and our services VP.

One point on application and API testing: attack and pen testing against applications — web, thick client, mobile clients (Android, iOS), and the APIs they talk to — is considered dynamic code analysis, versus static code analysis. Static code analysis is where you take the code, run it through a code review tool, and it looks for common things like unfiltered input or missing authentication on a method. The biggest thing we run into — and this is especially important with SaaS companies — is understanding and working within their SDLC and release cycles. A typical sprint for most SaaS companies might be every two weeks, so every two weeks new code goes into production. That code could be a change in the authentication method, or as benign as new graphics. But if there are that many changes throughout the year, testing that app or API only annually really doesn't provide the security standards it may require. So working within their SDLC (or SSDLC) and understanding the release cycle is important. We've had situations where the pen-test-as-a-service retainer model works well here — if it's a simple, low-risk feature, a tester might spend one to three days looking at that change, find the issue, help the dev team fix it, and then it gets pushed to production. There's typically no need to do a full retest of the application every two weeks — it's cost-prohibitive, takes time, and doesn't make sense.

Let me talk a little about offensive security, which is somewhat of a new term. Traditionally it was associated with NSA-style tactics or truly hacking back — if you're being hacked by a nation-state or a financially motivated actor, offensive security might have meant hacking back to stop them or take down their servers. That definition has changed. Now it means more proactive cybersecurity — vulnerability management, scanning, pen testing, and attack surface management all fall into the offensive security bucket. Also, the good folks at Offensive Security — a great training provider, and one of our training partners for OSCP, the gold standard for pen test certifications — have eased off guarding and trademarking the "offensive security" label, which has opened it up to the broader market.

Let me explain what I mean by offensive security following the defensive security model. When I started in cybersecurity, I started in a defensive SOC doing log monitoring. We had a homegrown SIEM called ActiveGuard — one of the first of its kind — that collected firewall and server logs and tried to correlate whether something bad was happening, like someone brute-forcing an FTP server. Before that, log checks were done by IT administrators when something went wrong, or the really diligent ones would manually go through logs or build custom scripts and grep through logs for anomalies — and that was considered cutting-edge at the time. That was the start of defensive security. Now defensive security has matured into real-time automated log analysis, and real-time endpoint protection that not only alerts but takes action. We've gone from manual, to intrusion detection, to intrusion prevention. That's the evolution of defensive security, and I believe offensive security is following suit.

That's where the once-a-year (or even twice-a-year) pen test has to evolve into something more continuous — and this comes back to the CVE problem. You need continuous monitoring and validation of security vulnerabilities, and you have to continuously measure and improve the effectiveness of your cyber stack and controls. In my opinion, that's the only way to keep up with the CVE problem. Remembering that chart going up and to the right after 2016 — if 30,000 known vulnerabilities are released between your tests, that's a problem, and it's not reasonable to think an annual test will keep up.

Two options to solve the problem. One is in-house vulnerability management and an in-house pen test team, with sophisticated enterprise tooling looking at your network, applications, and cloud stack for known vulnerabilities on a consistent, automated basis, fed into a platform where humans validate and prioritize the issues — plus a pen test team constantly doing manual validation and testing. Some organizations have this — the big banks, government, high-risk organizations with massive budgets. But for the majority of organizations, something like an attack surface management service that marries the tech with the human element is a more reasonable, approachable, and affordable option.

Jack Ekolof:

I've got a question for you, Paul: how does this impact insurance premiums — one pen test versus multiple?

Paul Petefish:

Great question. Traditionally, cyber insurance questionnaires used to be one page: "Do you have a firewall? Check — we'll give you cyber insurance." The insurance providers and underwriters got burned by that over the past couple of years with all the ransomware attacks. Now you have multi-page questionnaires: Do you have multifactor? Endpoint protection? And we're starting to see, "Do you perform pen testing? How often?" So it's that third-party validation that helps underwriters lower your risk profile to give you cheaper insurance — or allow you to have it at all. We've seen companies that, based on past issues, have gone through dozens of cyber insurance providers and still struggle to get coverage, which is a shame. Good question.

A couple of final thoughts. Proactive, consistent testing and offensive security are increasingly important, mostly because of the CVE problem. Red teaming will always have a place — whether red teaming or bug bounty programs — and may replace what we'd consider traditional pen testing. But continuous testing like attack surface management will replace traditional pen testing, if that makes sense. Purple teaming is increasingly important — more customers want it included. That's where the offensive team and pen testers have a chain of communication with the blue team (whether internal or an outsourced MSP/MSSP), supplying real-time information about what they're doing, how, and when. For example, "We're getting ready to attack a web login with SQL injection," and the expectation is the defensive team comes back and says, "We see it — are you coming from this IP? Yes. Great." They don't necessarily stop the attack, because that would hinder our ability to test — sometimes they do, sometimes they don't — but that collaboration and communication is increasingly important.

There are a lot of cheap options in pen testing. Anybody can start a pen test company with a laptop — that doesn't mean they're bad; they might be one of the greatest pen testers on the planet — but you certainly get what you pay for, and anything cheap is likely scanner-based. I'm not talking next-gen AI-driven pen testing; I'm talking basic vuln scanner results. If you're just looking to check the box and you're not truly worried about your security posture, that's an option — but buyer beware.

And last, a couple of things on the human element — people versus the machine. I'm a firm believer that you still need people in the chain, and people are a valuable part of offensive security and next-gen services like attack surface management. People bring a cleverness and creativity that machines can't yet. I know there are a lot of AI-based and automated pen testing companies out there — they've been around for a decade — and someday we'll get to a place with a pretty decent automated pen test, but I think we're far from that. You still need people in the process — I don't think, I know you do — and true automated pen testing just doesn't exist yet. There's also safety and availability: the human element makes sure an exploit doesn't run that isn't thoughtfully planned and appropriate for the target. Is this a production system? Their production customer portal? Should I really run this exploit in the middle of the day, or at night, or should it just be documented? That's where the human element comes in, from a safety and availability perspective. And we all know automated systems — SIEM, endpoint protection, name your automated security tech — are notorious for creating a lot of noise and false positives unless they're extremely well-tuned and maintained, which again comes back to a person being involved in that chain.

So that's it. I really appreciate everybody's time and attention.