Webinar: Continuous Pentesting in 2024

Jack Ekolof:

Welcome to the Evolve Security May webinar, "Continuous Pen Testing in 2024." Appreciate everyone's attendance — we'll kick things off with introductions. I'm Jack Ekolof, the MC for today's event. We've got Rob Kraus, Vice President of Security Services here at Evolve, as well as Paul Petefish, Founder and CEO, who's going to kick us off on this highly relevant topic.

Paul Petefish:

Thank you very much, Jack, and welcome, everybody — I appreciate you spending some of your afternoon with us. As Jack mentioned, my name is Paul Petefish, co-founder and CEO of Evolve Security. I've been in the cyber space for about 20 years, the majority on the offensive side. I spent most of my career with a first-generation MSSP called Solutionary, which was acquired in 2013 by NTT (Nippon Telegraph and Telephone), one of the largest holding companies in the world.

Why is that relevant? I'm a technical CEO. I started off in the Security Operations Center, watching the evolution of defensive security unfold — from log monitoring and custom SIEM tools, to how you blend the right mix of man and machine, plus the processes and procedures that create impact in a cybersecurity program. My thesis — what we've built the organization and our next-gen offensive practice on — is that offensive security is going to follow the maturing and adoption path that defensive security already went through. For example, believe it or not, when I was in the operations center in the mid-2000s at Solutionary, we had customers who weren't checking their logs, or weren't paying to have their logs monitored in real time — it might be weekly or daily monitoring. Now, that would be unheard of; it has to be real time, because attacks are quick and attackers are becoming far more sophisticated and impactful, especially with the spread of ransomware. So that sets the stage for how we think about next-generation offensive cyber and how it works into continuous pen testing. Rob.

Rob Kraus:

Thank you, Paul. My name's Rob Kraus, Vice President of Security Services here at Evolve Security. I've been in this space a long time too. I started off as a penetration tester, moved into vulnerability research and reverse engineering, and have over 25 CVEs to my name for vulnerability research I've conducted personally. Offensive-based penetration testing has been my bread and butter for a very long time. We still conduct some traditional penetration testing, but we've seen a large migration of our customer base toward continuous penetration testing, and we feel that's the next evolution. We're on the cutting edge of affordable services to make sure we're constantly looking out for our customers' networks and environments. Thank you, Paul.

Paul Petefish:

Thanks, Rob. I want to encourage audience engagement — Jack's fielding the chat, so fire off questions along the way; it's nice to have a dialogue rather than a lecture. Our focus is next-gen offensive cybersecurity — more real-time, continuous testing. The basis for everything is continuously measuring and improving the effectiveness of the cyber stack and tech stack, whether that's a cloud, application, or network layer. We do that primarily through pen testing as a service and technical service management — technology-led, with the human element delivering that last mile in the service chain.

So let's talk about why you'd begin pen testing continuously. Adversaries have been conducting consistent, constant recon and discovery against target assets for a long time — that shouldn't be news to anybody. The industry is realizing it's time to flip the script, get away from the point-in-time assessments most organizations are used to, and move toward a continuous testing model, so you have a better idea of what your vulnerable footprint looks like and how to reduce those exposures.

Rob Kraus:

Today, organizations are hyper-focused on digital transformation, relying heavily on technology and cloud platforms, and the threat landscape is getting larger because of that. It's long been said that the cloud is just storing your data on somebody else's computer — the risk is still there. Moving to cloud doesn't necessarily make you more secure; there are payoffs, but there are also trade-offs. So you have to keep tabs on the attack surface — not just from the cloud perspective, but your organic IP ranges, addresses, and FQDNs — addressing challenges like shadow IT, and knowing your known and unknown perimeter assets.

If we look at the count of CVEs — Common Vulnerabilities and Exposures, tracked by the National Vulnerability Database and organizations like CVE Details — the count over time has not decreased; it's constantly increased. In 2017, the ability to report vulnerabilities became much easier, which is why you see a significant spike then, and constant growth since. Year to date — and it's the end of May — we're already at 17,165 newly reported vulnerabilities, which is more than half of what we saw in all of last year. Compared to the same point in 2023, that's a 44% increase. I pulled this data and updated the slide last night, so it's very accurate.

Paul Petefish:

Just a quick comment to be clear: CVE — Common Vulnerabilities and Exposures — is the government-backed nonprofit that tracks known reported vulnerabilities. Obviously there are lots of unknown vulnerabilities that don't get reported, or get reported later. The synopsis is that CVEs are growing close to exponentially over the past three or four years. And the drop you see for 2024 on the chart isn't CVEs going down — that's just year to date, and we're already up 44%, so we expect 2024 to finish well above 2023.

Rob Kraus:

Let's look at the vulnerability history. Beyond the constant growth of total vulnerabilities, something important is the number of critical vulnerabilities, historically about 9.8% of the total. In 2022, there were 2,472 critical vulnerabilities reported; in 2023, that number more than doubled. Broken down weekly, that's approximately 559 new vulnerabilities reported every week, with 99 of those critical. That doesn't mean 99 critical vulnerabilities will impact your infrastructure, but it's significant when you think about recent ones like the Palo Alto remote code execution vulnerabilities — it can be a challenge to keep up.

One of the metrics we follow is how long it takes to identify a vulnerability — we call that the mean time to identify (MTTI) — and then the mean time to remediate (MTTR). The quicker you can identify, the quicker you get to what we call the "left of the attack." If you think about the Lockheed Martin Cyber Kill Chain, the premise is that the sooner you detect vulnerabilities or malicious activity, the less potential exposure you have for loss. So with continuous scanning, the quicker you identify and remediate vulnerabilities, the more meaningfully you reduce the potential impact.

Paul Petefish:

This is really what Rob is talking through with mean time to find and mean time to fix — that's how Evolve Security looks at the vulnerability lifecycle. The actual count of vulnerabilities to an organization is the traditional way of looking at it, and it's important — zero-days like the Palo Alto one, or Shellshock, or a new OpenSSL vulnerability can really throw those numbers off, so it's always going to be a little wavy. But when you think about the number of vulnerabilities, it's not if it's wavy — it's how long it took you to find it. Did it take six months? Eleven months, because you're only doing an annual pen test? Or did you find it within 20 days and fix it within 20 days? That's really where the impact and the de-risking happen.

Rob Kraus:

While preparing for this presentation, we did some data mining across our clients who have continuous testing in scope, and I want to highlight a few interesting findings. We're seeing that approximately 1% of all vulnerabilities identified by the Darwin attack surface management platform are critical — less than we expected, which indicates organizations are maturing their ability to address critical vulnerabilities rapidly. We find significantly fewer critical vulnerabilities with attack surface management than with customers doing a single point-in-time assessment.

We also found that 92% of all identified vulnerabilities are categorized as medium severity — pretty common across all industries and verticals. Of those medium-severity vulnerabilities, approximately 74% are related to SSL/TLS cipher strength, certificates, and related implementation issues — easy enough to fix in most cases, but a lot of organizations deprioritize them. We found that 61% of the high-severity vulnerabilities relate to outdated Apache and Apache Tomcat implementations, which doesn't surprise us given Apache's web-server market share. And — I'd think I'd be surprised after pen testing as long as I have, but — we're still seeing about 2% of continuous testing clients with open Microsoft Windows NetBIOS ports exposed externally, which is just poor security hygiene. Two weeks ago, it was escalated to me that we had someone still running a Windows 2000 server with an NT 4 service pack installed, vulnerable to some pretty nasty stuff. The companies we found those vulnerabilities on didn't even realize they had them exposed to the internet, so thankfully they remediate immediately, and we immediately verify the fix.

Jack Ekolof:

We've got a question: there's been news recently about the National Vulnerability Database being behind on categorizing vulnerabilities. How does that impact what you're doing, Rob?

Rob Kraus:

A lot of organizations rely on the NVD and other sources. The NVD publishes the MITRE database, and CISA has the Known Exploited Vulnerabilities feed. It doesn't surprise me — think about how many people are submitting vulnerabilities and how many researchers are out there. My own experience submitting vulnerabilities over the years is that it sometimes takes weeks to get an answer, then you finally get a CVE number, and then you go through the responsible disclosure process with the affected company. So even when a vulnerability is identified and reported, there's a delay.

And it doesn't surprise me that with AI, there's going to be more of a lag — because AI is a force multiplier not just for legitimate businesses building products, but for attackers. Years ago we talked about "script kiddies," who used other people's scripts, running them blindly against targets hoping to compromise something with little knowledge. Today you've got "AI kiddies" — "I need an exploit to do this," and ChatGPT will create it in minutes; you run a couple of tests, and next thing you know, you have a working exploit. That's very real. So legitimate researchers are constantly submitting new vulnerabilities. With the 29,065 vulnerabilities identified last year and the 17,000-plus this year, it wouldn't surprise me if there's double that in backlog — it's just going to take time to publish. So look at CVEs and MITRE as "here's what we know about," but there's a whole iceberg of unreported vulnerabilities, and that's where manual testing adds value.

Paul Petefish:

Thanks, Rob. So, the "why" here is pretty clear: the bad guys aren't letting up — it's more aggressive. There's more cybersecurity research, and more financial incentive for the bad guys, with ransomware being a big component. The impact to businesses is real. Before, it might be fines or reputation damage; now, if you can shut down a business, it's a totally different landscape — and a different landscape requires different ways of doing things. That's the evolution of pen testing.

If we look at this diagram, each large box represents a year — 365 boxes for 365 days. For traditional pen testing, let's assume an external pen test, fewer than 50 host assets, starting January 1. Traditional pen testing takes, say, 14 days of field testing with an engineer or two — reconnaissance, vulnerability scanning, then exploitation and vulnerability chaining. But you can quickly see you're leaving a large gap in coverage the rest of the year. What's needed is a more continuous model. There's been a half-step between traditional and continuous — organizations doing it quarterly, which is better; you get a bit more coverage and your mean time to find goes down. But you're still left pretty exposed.

Some organizations talk about "continuous pen testing," but it's really that quarterly or monthly model. The only way to get true continuous pen testing — before what we designed with Darwin ASM — was to have an assigned internal red team or full-time external pen testers whose only job, all day every day, is to test the environment, network, cloud, or apps. Larger, more sophisticated organizations with bigger budgets have that, because they're larger targets with the risk profile to accommodate it. Most organizations don't. So you have to combine technology with the right processes and procedures to blend man and machine into a true continuous pen testing model.

On these charts, the light-blue dots with the border are manual testing and exploitation — a human being behind the console doing the testing, using more manual tools. The wider circles in the center are automated recon and discovery. We have a recon-and-discovery engine called Sparrow that constantly crawls through TCP ports and UDP services, and hooks into Azure and AWS, looking for new externally exposed assets coming online — immediately port-scanning and service-checking them to build an asset inventory of what's exposed to the internet. Once a new port or service is identified — and this happens continuously, at minimum daily, potentially every 10 minutes depending on scope — it's handed off to a pen tester for manual interrogation, vulnerability hunting, and potential exploitation in real time, as those assets come live. Throughout the year, people's external perimeter changes — they accidentally open a firewall admin console, or expose NetBIOS ports — and that's identified in near real time and handed to a person for manual testing. We believe this is the only way to get a true continuous pen testing model, with the right blend of man and machine.

Lastly, you'll notice the same amount of manual testing at the beginning of our test — we include what would be a traditional point-in-time pen test cycle within our Darwin ASM offering, to meet compliance standards. I'd argue a continuous testing model goes well beyond the rigor and intent of most pen test compliance requirements even without it, but to give customers and their auditors comfort, we include that point-in-time assessment and the standard report artifact. Let me pause for questions.

Jack Ekolof:

We've got a question: are you performing continuous pen testing primarily from an external standpoint, or is it a mix of internal and external for your clients?

Paul Petefish:

Great question. Right now this is EASM — external attack surface management. Our internal capabilities are in development. I can't go into much detail on exactly how we'll do that, because there are trade secrets involved, but that's next. Thanks for that.

All right, Rob, do you want to talk about some of the risk-management value props?

Rob Kraus:

Sure, thanks, Paul. When you look at the value proposition for risk management of your external attack surface: first, early detection. We talked about getting left of the attack in the kill chain — continuous pen testing helps organizations identify vulnerabilities, hopefully before they can be exploited, and reduces that mean-time-to-identify and mean-time-to-remediate window. Second, minimizing downtime — depending on the vulnerabilities targeted, an attack could lead to denial of service or unavailability, so this matters, especially if you have SLAs requiring 99% uptime. Third, avoidance of regulatory fines — when we designed Darwin ASM, we were conscious about meeting the rigors of GRC (governance, risk, and compliance) initiatives. And fourth, protecting brand reputation — everyone's seen news stories about major retailers and hospitals being attacked, usually followed by a negative impact on stock, shareholders, and brand confidence. Pen testing in general is good cybersecurity hygiene, but continuous pen testing takes that to a different level.

I'll leave you with three key takeaways. One: a lot of our customers are surprised that the barrier to entry for continuous testing services is much smaller than in previous years.

Paul Petefish:

Let me hop in there quick, Rob — sorry to cut you off. This goes back to my point: to have continuous testing previously, you had to have full-time testers on staff, because the technology and operations hadn't been invented yet. That's extremely expensive and a high barrier to entry. You're not going to see most SMBs — or even less sophisticated enterprises — have their own red team or pen testers on staff.

Rob Kraus:

Excellent point, Paul. When Jack and I work with clients on architecting solutions and present pricing on an external pen test versus continuous penetration testing, they're often surprised the delta in price really isn't that large. There is a difference, because of the constant testing cycles, but it's not out of reach for most organizations, which is why we see a lot of clients converting to continuous testing.

Second — point-in-time assessments provide little value. They're good for a snapshot, but if you're really strengthening your security posture with the number of vulnerabilities coming out daily, you need constant testing. The analogy I use: you pay a monthly bill for a home alarm system, but how many friends and family do you know who only turn it on once a year when they go on vacation? That's not being as proactive as you could be, given you already have the technology and are already paying for it.

I predict that in the future, as this gets adopted more, governance, risk, and compliance bodies — there's usually some lag — will start pushing for it. It wouldn't surprise me if future iterations of PCI (we're on 4.0 now), HIPAA, or others start pushing for more proactive penetration testing instead of once-a-year or quarterly assessments. And, as mentioned, continuous testing supports and exceeds many GRC requirements. That's the last slide — Paul, any closing notes?

Paul Petefish:

Just a quick comment: point-in-time assessments do provide some value — it's better than nothing, and we're not discouraging people from doing them. What we're saying is there are options to get a lot more value for a little more investment. And I believe traditional pen testing will continue to be valuable, but more in terms of very specific, technology-focused testing — like testing the technical implementation of smart contracts, or being an expert in pen testing autonomous vehicles, or large language models. I think you'll see a shift toward hyper-focused, specific testing on one hand, and broad continuous coverage on the other.

Jack Ekolof:

We've got a question: is continuous penetration testing just scanning all the time?

Paul Petefish:

Good question. We don't like to use the word "scanning," because we don't want this associated with just a vuln scan or vuln scanner — it's a lot more than that. The Sparrow engine uses many different tools, tactics, and techniques — some enterprise-grade, some custom-built, some open-source frameworks — plus connections into cloud APIs, to get a proper understanding of the footprint and available attack surface.

Rob Kraus:

To add to that — with our open-source intelligence capabilities, we pull in whatever data we can, like enumeration of subdomains and assets you may not even realize you have. And it's a constant refining of our signal-to-noise ratio to make sure we're sending high-quality notifications to clients. It's not a "scan and dump," which is what you get from a lot of shops — they scan you, provide a report, and call it a pen test. Paul and I grew up in pen testing; we know the difference, and the value we provide is proving impact, as opposed to just saying, "You have a vulnerability to patch."

Jack Ekolof:

Another question: do you incorporate phishing exercises with your continuous testing?

Paul Petefish:

Great question. At the moment, we don't officially incorporate phishing into the ASM delivery — that's typically an additional module. Not saying it won't be in the future, but we don't include it today.

Jack Ekolof:

Next question: how does this benefit me if I have a fixed scope and my environment isn't changing often?

Paul Petefish:

It's always interesting to hear that. Of course, no one purposefully opens up their firewall interface to the internet, exposes NetBIOS, or fails to patch Apache because they don't care. About 99% of the mistakes and holes you see in IT are honest mistakes — ignorance, not giving something enough attention, or lack of experience. That's the stuff you have to worry about and keep an eye on. The goal is for us to verify that your environment isn't changing, or that you don't inadvertently expose something — a host, subdomain, or service you didn't realize you had. A common example is the cloud: developers inadvertently spinning up microservices or hosts and exposing them to the internet by accident during dev work. Go ahead, Rob.

Rob Kraus:

A lot of it is ports, protocols, infrastructure, and administration — but it's also what we don't know today that we can be educated on tomorrow. There may not be a gaping hole in a Palo Alto firewall today, but even if your infrastructure hasn't changed, a vulnerability gets disclosed. Recently there was CVE-2024-3400, a Palo Alto remote code execution vulnerability. You may not have changed your environment between yesterday and today, but now you're vulnerable. So you also get the benefit of updated detection capabilities — keeping track of new CVEs, plugins, and detection capabilities. Even if nothing changes in your environment, a new vulnerability could be disclosed tomorrow. Think about Heartbleed and Shellshock — some of the older ones that made their way around. Even organizations doing a great job managing their infrastructure were caught with a gaping hole and had to immediately detect and address those vulnerable instances.

Paul Petefish:

Thanks, Rob — that's a huge point I'd missed. A good part of the time, when something serious happens, it's outside the engineers' or admins' control, because it's third-party software that's become outdated, or the vulnerability isn't part of their dev cycle.

Jack Ekolof:

We've got a question: what is driving the increase in vulnerability reports and CVEs?

Rob Kraus:

I can address that quickly. In 2017, MITRE did a really good job of reworking a lot of their processes, staffing, and their ability to intake vulnerability reports created by researchers. That by itself led to what I'd call a "false increase" — I think vulnerabilities existed for years prior, but we saw a significant spike in 2017. This was back in the days of "no more free bugs" and bug bounty programs coming online and hitting their maturity strides, so reporting just got better.

Over the last couple of years, AI is a large contributor — good for people using it for good, but also for the bad guys prototyping exploits and scripting malicious code. We're also at a point where tools we've used for years are much more mature. Even at Evolve Security, we have the pleasure of increased development capabilities because we can write code more efficiently — and some of those same efficiencies benefit malicious actors. There's also been an increase in CNAs — CVE Numbering Authorities — that researchers can submit to. Years ago it was just a handful; now there are more, which is likely a contributing factor.

Paul Petefish:

A couple more things: when you look at the exponential growth of technology, we're creating more and more code and software every day, so you'll naturally have an association between more software and more vulnerabilities. And there are a lot more practitioners now — both black hat and white hat — who contribute to CVEs.

Jack Ekolof:

One last question: when did you first notice the shift to the continuous testing model?

Paul Petefish:

Not to toot our own horn, but we've been discussing and developing a continuous model since 2017, and it's taken us a while to get here. It's part of adoption, market and customer education, and the development of the technology, processes, and procedures. So from our perspective it's 2017 — but I still think we're at the very early edge of this, at the very first wave if you look at the Gartner hype cycle.

Jack Ekolof:

We're all super appreciative of everyone's attendance today. That's the end of our presentation. Thank you, Paul and Rob, thanks everybody for joining, and a special thanks to everyone who engaged with questions. We'll see you next time.