Webinar: From Findings to Fortification Post Penetration Test Risk Management

Jack Ekolof:

Good afternoon. I'm Jack Ekolof. Welcome to the Evolve Security October webinar. Today we're discussing "From Findings to Fortification: Post-Penetration Test Risk Management." We've got Rob Kraus, Vice President of Offensive Cyber here at Evolve Security, leading the discussion today, as well as a member of his team, Jason Frank, Offensive Security Engineer. They're going to talk about their experiences delivering pen tests, and how to fortify and make real use of those results in action. With that, we'll kick things off to Rob and Jason.

Rob Kraus:

Thank you, Jack, for the introduction. My name's Rob Kraus, and I'm the Vice President of Offensive Cyber. I've been in the security industry for quite some time. I actually started my entrance into the cyber realm years ago at AT&T, as a security architect on the U-verse rollout. From there I moved into penetration testing, and then further into reverse engineering, vulnerability research, exploit development, and so on. So I have the pleasure today of working at Evolve Security as the Vice President of Offensive Cyber, working with a team of very skilled and talented security engineers. That being said — Jason.

Jason Frank:

Thank you, Rob. Hello, everybody. My name's Jason Frank. I've been working at Evolve for a little over two years now. This is my first job in the space. I was lucky enough to find the Evolve bootcamp and went through that, got a very good foundation, and started pretty much right out of the bootcamp as an analyst. I moved over to pen testing about nine months ago, and I have a lot of direct interaction with clients in their remediation efforts.

Rob Kraus:

Thank you, Jason. Today's presentation isn't a sales presentation, but we do have some attendees who aren't existing customers, so, Jack, do you want to give the one obligatory overview slide so people know who we are, and then we'll jump straight into the presentation?

Jack Ekolof:

Sure. Evolve Security is an offensive firm, based out of Chicago. We focus on a couple of areas: continuous pen testing via attack surface management, and pen testing as a service. We've also got a large suite of traditional pen tests, from application to cloud to wireless testing. And one of the crown jewels of the organization is what Jason talked about — our cybersecurity academy. It's a great source of talent for us, and Jason is a testament to that. We train toward OSCP and provide a cybersecurity bootcamp.

Rob Kraus:

Thank you, Jack. I'm going to start off by giving a little context and background on the inspiration for today's presentation, share some of my strategic vision and some of the things I've observed over the years, and then Jason is going to speak directly to what he observes on the front lines. As a member of our offensive security operations center, and as a security engineer conducting penetration tests against our clients and providing those readouts, Jason will share where he sees organizations struggling to take the results they receive and make effective use of them to increase their cyber resilience.

First, let's talk about how we got here. The data I'm presenting today is pulled from a couple of different sources — notably CVE Details, which tracks the increasingly large number of vulnerabilities reported through MITRE's CVE program. CVE stands for Common Vulnerabilities and Exposures. Years ago, MITRE started collecting information so that when researchers or security practitioners identify new vulnerabilities in software, applications, or hardware, there's a way to report and track them — so that people implementing those products in their environments understand the risks involved.

The first thing I want to talk about is the number of CVEs per year. We see a significant increase year over year — the trajectory is moving upward and not slowing down. Last year there were just over 29,000 new CVEs reported via MITRE, an increase of about 15.2% between 2022 and 2023. This year, as of a couple of days ago, we're already at 31,705 new vulnerabilities reported — an increase of 9.1% over last year's total. We're coming up on the end of October, and we've already surpassed last year's total by about 9.1%.

There are a couple of contributing factors. The tools available today are arguably a lot better for researchers — both white-hat and black-hat, good guys and bad guys. New technologies are being implemented where we don't necessarily know what new types and classes of vulnerabilities may surface. And the adoption of artificial intelligence is making writing code, identifying exploits, and doing vulnerability research a lot easier than ever before. So you can imagine the perfect storm here, and why it's so important to be able to say: what are my risks from a vulnerability perspective? How do I become more efficient and effective at managing these vulnerabilities? Because the vulnerabilities clearly aren't going away.

Compounding this is the continued exposure and increasing size of the attack surface for a lot of organizations. We're far from the legacy world where an organization had a specific data center, kept all their eggs in that basket, and wrapped security controls around it. With the adoption of cloud computing — platform as a service, infrastructure as a service, software as a service — we're outsourcing a lot of things we used to be responsible for internally to third parties, which expands the attack surface even more. The mantra for a lot of organizations today is "why build it from scratch when someone else has already built it?" — but with that comes risk. So we're going to talk about how to manage those risks and take proactive efforts to bolster our cyber resilience.

Just as clarification: cyber resilience is a confusing term for people who aren't familiar with it. Think of cyber resilience as an organization's ability to continue delivering goods and services to its customers despite a major cybersecurity incident. What are the tools, controls, policies, and procedures we can implement to thwart attackers and build resilience into our operations, so we can keep delivering services even in the event of a security incident?

To that end, we recommend a risk-based approach to prioritizing efforts. From an operational perspective, a vulnerability is a risk to the organization. If a malicious actor identifies and exploits a vulnerability, what's the potential business impact — small, or potentially huge? What is our tolerance for risk at an organizational level for the different services we provide? Not all assets are built the same, have the same value, or would have the same impact if attacked.

Think about it this way. Years ago, it was common for hardware manufacturers to post user manuals, administration manuals, and assembly manuals on FTP servers for people to download. Set up properly, that's a service with very little risk — it's open information, not secret or proprietary. But today, consider a patient portal storing sensitive data — patient information, billing, protected health information (PHI). That has a very different business impact if a malicious actor accesses it. So one key thing in a risk-based approach is building out that risk register: identifying asset values and potential business impact based on different risks. Do I have patient data? Financial data? Compliance requirements that insist I have certain controls in place?

We also think about cost-benefit analysis. Risk is really about impact and the likelihood of that impact. There are a few acronyms worth walking through. First, single loss expectancy (SLE): how much money could I lose if a malicious attacker performed, say, a distributed denial-of-service attack against my e-commerce site, or a SQL injection attack exposing all my client data? You multiply the SLE by the annual rate of occurrence (ARO) — how many times a year we predict it may happen — and that paints the picture of your annual loss expectancy (ALE). Quick math: if a single loss expectancy for a DDoS attack is $50,000, and it happens three times a year, my annual loss expectancy is $150,000.

From there, ask: am I willing to risk losing $150,000 a year? For most organizations, I'd hope the answer is no — but what's reasonable to protect that data? We don't want to spend more on protections than the potential annual loss expectancy. It makes no sense to spend $500,000 on a solution that only protects you from $150,000 of loss. Conversely, if a solution that prevents those attacks costs $25,000, that makes all sorts of sense. Also look at operational impact: can I deliver services to my teams and customers? Can my internal employees do their jobs day to day? Take all of that into consideration.

Beyond a risk register, we can leverage cyber threat intelligence (CTI) to quantify the types of attacks we may see. If I'm in a financial institution and I know there's a Chinese or Russian advanced persistent threat (APT) group specifically targeting financial institutions, I can use a resource like the MITRE ATT&CK framework to look at that APT's tactics, techniques, and procedures (TTPs), and identify controls to thwart those attacks before the attacker even reaches my front doorstep.

We also use common vulnerability scoring systems to rank vulnerabilities — critical, serious, high, medium, low. And contextualization and enrichment are really helpful. I have a screenshot here — all in a test environment, not client IP addresses — showing a set of serious vulnerabilities. The "known exploited" feature in our platform lets clients know that, out of a hundred serious vulnerabilities, maybe only about six look like they've been weaponized and are known to be exploited in the wild.

Why is that useful? Smaller cybersecurity shops get assessments done, a lot of vulnerabilities are identified, and it can be very confusing how to prioritize what to fix first. Indicators include risk, impact, and likelihood against your organization, and operational impact — but also whether a vulnerability is weaponized. Are the bad guys actually exploiting these? Are there exploits residing in rootkits that make it easy for attackers? Maybe I should prioritize those first. This matters because, at the end of the day, we do not have an endless bench of security professionals sitting around. For most organizations, security is a subset of IT, or its own pillar, but those benches usually aren't full of unlimited resources. We have to prioritize.

Another thing I strongly recommend is industry-specific intelligence. Beyond adversary TTPs, there's great information in the Information Sharing and Analysis Centers — ISACs. There's FS-ISAC, Auto-ISAC, energy ISACs, and many more, dealing specifically with collaboration and information exchange. In the financial sector, you may be collaborating with your competitors for a common cause: what threats against my institution should I be aware of, and what are we actively seeing malicious actors target?

These are starting recommendations — a way to begin thinking about managing risk. This isn't a risk framework in itself; there are many risk-management frameworks out there. NIST — the National Institute of Standards and Technology — has a Risk Management Framework (RMF) and a Cybersecurity Framework (CSF). The RMF can serve as the basis for how you address and assign value to different types of risk, including risks outside of cyber — what if a delivery driver slips off the brake and drives through my front door? The CSF focuses more specifically on cybersecurity risks. There's also COBIT, and another very popular one, CIS — the Center for Internet Security — Top 18. These are all great frameworks. Nobody has to start from scratch; these resources are freely available in most cases, and they're a good place to start if you're building out a risk-management program.

With that, I'm going to take a break and have Jason take over to talk about what he's seeing from the front line — where organizations are falling short day to day. That's what brought us to put this presentation together. Jason, why don't we jump into the biggest things you're seeing?

Jason Frank:

As I mentioned, I have a lot of experience directly interacting with clients and their remediation efforts. I'll go over a few of the things I've seen clients struggle with the most, post-pen-test.

The first is failure to prioritize risk-management activities. That's a big one — clients will treat all vulnerabilities the same. Tying back to what Rob mentioned about known exploited vulnerabilities versus those that aren't: if a client doesn't consider business impact or whether something is being actively exploited, and instead focuses remediation on lower-severity vulnerabilities that aren't weaponized and don't have proof-of-concept code out there, that's not a good use of time.

The second is incomplete or delayed remediation. A lot of times clients wait too long to start — they'll begin six months down the road after a test is completed, or they'll only fix certain things and leave more critical vulnerabilities untouched. That leaves systems vulnerable longer than they need to be. If a client doesn't start remediating until months after the test, they may only fix a few issues and could get compromised — or there are vulnerabilities that could be chained together from the ones that weren't fixed, plus newly discovered ones.

Third is overlooking root causes. Clients will fix the symptom and not the cause — they won't dig deeper to figure out why an issue is happening. If you take that approach, you'll likely see the same types of vulnerabilities keep showing up. But if you dig in and figure out why they're happening, you can prevent them in the future. For example, if a client doesn't have a proper patch-management strategy, they keep fixing the same vulnerabilities that pop up — when instead they could put a proper patch-management strategy in place and proactively stop them.

Fourth is a lack of communication. A lot of times remediation falls solely on the shoulders of the IT people, and they take the project and run with it without getting all stakeholders involved. There are other departments that should be involved. If you fix something and aren't aware you broke part of a website you're not even involved with, that can be prevented by having all the stakeholders there from the start and putting a proper plan in place before fixing things.

And finally, testing frequency. Right now we have some pretty outdated standards on how often tests should be completed — a lot of companies test once a year, or maybe quarterly, and in this day and age that just doesn't seem to be enough. You saw earlier how Rob showed the year-over-year growth of vulnerabilities and how frequently new ones are discovered. More testing, more frequently, is definitely the way to go.

Rob Kraus:

Thank you, Jason. We're going to look at some of the most common vulnerabilities we identify and some of the root causes, drawing on our visibility from the thousands of penetration tests, vulnerability scans, and assessments we perform a year.

One of the biggest trends we're seeing today is weak API security implementation. Developers have gotten much better over the years with front-end development security — preventing things like cross-site scripting and SQL injection. But there's a much larger reliance on APIs today than ever before, and unfortunately we're starting to see the same mistakes being made with APIs that we used to see with traditional web applications. For several years now, OWASP — the Open Web Application Security Project — has published a Top 10 for API security, which is an invaluable resource. We're seeing vulnerabilities that developers had addressed from the GUI perspective resurface — things like insecure direct object reference, and not properly constraining data requests to authorized users. Because our clients are becoming more aware of this, they're leveraging our API pen testing services specifically to beat up those APIs. Many clients have multiple web applications served by a single API, so we're actually seeing a trend where we test APIs more directly first, and then go after the web applications.

That relates to implementing and verifying secure coding practices, and to a failure to incorporate system hardening. We see organizations deploying but not hardening systems prior to deployment. Best case: configure the system as it's supposed to be configured, assess it to make sure it's properly hardened, and then deploy — as opposed to getting it up and running, deploying, and then discovering you've got database ports exposed to the internet. I checked earlier today, and about 9% of our ASM (attack surface management) clients have NetBIOS and SMB ports facing the internet. These are avoidable; we just need to verify that things are as secure as they can be.

Then there's ineffective or missing cloud security controls and reporting. Cloud is a great place, and AWS, Azure, and Google have all developed and published a lot of great security tools and capabilities. But in our cloud security assessments, we're finding that end users and administrators aren't leveraging those controls to better protect their environments.

Weak or compromised user credentials is another. We're seeing much better implementation of multi-factor authentication, but also a lot of legacy systems with published user credentials that have come out of data dumps. There are brokers and clearinghouses where you can search against your organization and see the credentials that have been dumped — some of them pretty fresh. Our penetration testing teams go out and look for them ourselves at the beginning of a pen test, emulating what attackers would do.

Poor network, application, and data segmentation is another — we still see a lot of flat networks, with no enclaves carving out areas that should be more highly secured. PCI does a pretty good job of defining how assets that store, transmit, and process credit card data should be protected and segmented from traditional user-land — but outside of those requirements, a lot of organizations still have flat networks and don't do a great job of segmentation.

There's also a lack of visibility into internal and external network activity. People say, "Oh yeah, we haven't seen any attacks against our network." But when you ask what they're doing to monitor, they don't have any MSSP or SIEM in place to collect the information that would tell them whether they've been targeted. There were reports a couple of years ago citing that the average dwell time — malicious actors residing within networks they've compromised — was north of 200 days before they were even discovered. So that visibility is important.

Lack of security awareness training, testing, and feedback is another, though thankfully we see it getting better. People are the weakest link in most cases. Phishing attacks are still a problem today because they work. Security awareness training, testing, and feedback to employees is super important. It'll always be a problem — it's never going to be fixed — but we can minimize the risk through training.

Another good framework to consider: years ago, Lockheed Martin published the Cyber Kill Chain. It runs from step one to step seven and looks at how an attacker unfolds their attack — reconnaissance, weaponization, delivery of payloads, exploitation, installation, command and control for persistence, and finally actions on objectives. Our job as security practitioners is to identify and stop the attacker as early in the kill chain as possible. The closer to steps one, two, and three you are, the better. So what security controls can you put in place to prevent attackers from getting in easily?

On proactive cyber-resilience considerations — some defensive things we see customers starting to implement more today. Zero trust architecture gets away from the old notion of "it's not that I don't trust you — I don't trust anybody." It doesn't have to be applied across your entire organization; you can wrap zero trust around your crown jewels and put more scrutiny into those protections. Microsegmentation of applications and networks is another. Privileged access management (PAM) — making sure privileged accounts can only do what they're supposed to, limited to the people, processes, or services allowed. MFA, thankfully, is being more widely adopted. Automated threat intelligence and response — if we see an attack happening, what automated steps can the defensive controls take to stop it before the attacker gets too far? Employee training and security culture initiatives, as mentioned.

Something we don't see organizations doing a lot of, but that we think has a lot of value, is tabletop exercises for incident response. Nobody wants an attack, but the likelihood of an impactful incident is high regardless of your industry — it's going to happen sooner or later. Tabletop exercises are that staged walkthrough: if an incident happened, what would I do, and how can I control and minimize the impact and losses? They're a great way to prove out your processes, procedures, and capabilities, identify gaps, and make sure everybody knows their role during an incident. Business continuity and disaster recovery planning is part of that, and data encryption and backup recovery is super important, especially with malware-related attacks that hijack systems.

Build out a vulnerability and risk-management strategy as a sub-component of your overall security program. And make sure your development processes are secure with validation in place. Too many times I see this: if you're producing public-facing code, have a process to perform regression testing — make sure it functions as it's supposed to and doesn't diminish value for your consumers — but also make sure you're actually looking for vulnerabilities before pushing to production. A lot of smaller organizations struggle with setting up separate environments for development, test, QA, user acceptance testing, and production. It sounds like a lot, and it is, but it's all designed to help you deliver high-quality, secure products without the fear of "we missed something, we forgot to secure something." Make sure the UAT acceptance criteria are met for functionality — but also that we're doing security.

To summarize, here are some core best practices. Invest in an effective vulnerability management lifecycle. At Evolve Security, when we provide information to clients, we have workflows that let them go in and say, "I just received a bunch of vulnerabilities that Jason validated for me — I need to address those." Within our management lifecycle, clients can say, "That's a risk; we get it — we're going to accept it," or "we're going to remediate it." Then you close the loop by having your provider come back and test after you've fixed the issues — it's super important to verify that the fixes actually took place. It sounds complex, but done right, you should be able to streamline that program. Anything too daunting, complicated, or hard to get through usually fails and gives more opportunity for failure. So your vulnerability management program is important, but streamlining it so it's not prohibitively complex is just as important.

The last two points tie together: measure your risk-management program's progress, and build a feedback loop. Jason's earlier example — organizations not remediating quickly enough — leaves you exposed longer, giving attackers more time to find vulnerabilities and craft exploits. One set of metrics we live by at Evolve Security is mean time to identify and mean time to remediate: how long does it take you to find the vulnerability, and what's your company's SLA for remediation? What's the delta between when I identified it and how quickly I need to manage it? Those are KPIs to think about. And with the feedback loop, periodically step back and ask: what's working great, and what do I really need to focus on and fix?

That's the last slide. Do we have any questions?

We've got one: how do you prioritize internal versus external remediation efforts? This has a lot to do with risk and context. If we're looking at an external footprint, I have some clients that literally have two IP addresses — one for a non-interactive brochure website, and another for their telecommuting workers to connect into the internal network. They may not have a lot of their crown jewels facing externally; they may be housing most of their important data internally and want to focus more on that. I'm not saying ignore the external, but the external effort might be significantly less than the internal effort. It's a bit of a challenging question, because the best way to evaluate it is to ask: what are the vulnerabilities and risks exposed externally versus internally, where are they in relation to the crown jewels, and how much of my time should I spend on each? Every company is different — even a hundred clients in the financial sector will have different risks and different appetites for how much risk they're willing to take.

We've got a second question on KEV: how can they enrich data and prioritize their remediation efforts effectively? I touched on that slightly, but — Jason, do you want to take this one, since you're on the front line every day looking at dashboards and prioritization for clients?

Jason Frank:

Of course, thanks, Rob. Just to reiterate, KEV stands for Known Exploited Vulnerabilities. A popular source is the website cisa.gov. If a vulnerability is listed on the KEV catalog, it's actively being exploited in the wild — meaning there's already weaponized code developed for it. That code may or may not already be released publicly or for sale to other threat actors. So it makes sense for security teams to know whether the vulnerabilities present in their environment are being actively exploited, because those should be prioritized first, for obvious reasons. For more information, go to cisa.gov and look for the KEV catalog — you can see all of them there. There are other sources as well; that's just a common one.

Rob Kraus:

Jason, I've got a question — not from the audience: are there any vulnerabilities you come across more than others that you think are impacting customers?

Jason Frank:

Obviously anything on the OWASP Top 10 — that list is there for a reason; they show up time and time again. In web applications, I'd say SQL injection and cross-site request forgery, some of the big ones on that list for sure.

Rob Kraus:

Great. We've got another question: "I have a hard time prioritizing vulnerability and risk management with my C-suite executives. How can I have a more meaningful conversation? They don't seem to get it."

That's my favorite question of the day, because it's a big challenge. As I explained, years ago I was a pen tester, vulnerability researcher, and reverse engineer — my perspective then is very different from where I am today as a vice president with a fiduciary responsibility to the company. One thing I've learned is that you have to talk the same language.

Here's an example. If I'm responsible for security and I go to my CFO and say, "We've got some SQL injection vulnerabilities, and we want to put a solution in front of our web application to help prevent some of the risk and potential data loss — it's going to cost $25,000," all the CFO hears is: "My security guy is asking me for more money to implement a tool." That's probably not the best way to have the conversation.

Instead, think about what's meaningful to the CFO. If I go to my CFO and say, "We just got our report back. We have a couple of SQL injection vulnerabilities the development team needs to fix, and we have risks related to injection attacks and potential loss of data. The solution we've identified costs around $25,000, but based on the annual loss expectancy I've calculated, it's potentially going to save us $250,000 in losses — whether from downtime, recovery from exfiltrated data, or loss of customers due to different types of attacks" — that's going to resonate more with the CFO.

So think about your audience. If you're trying to get support from executive leadership, they're looking at this in terms of impact to the organization, potential losses to revenue, and potential losses to customers. Think about it from that more abstract level. We could sit here all day and pontificate over different examples, but that's the one I'd use: talk the language of the C-suite. They'll understand it, they'll respect it more, and they'll be able to make a more informed decision with the numbers — as opposed to just hearing, "I've got a problem to solve, I need $25,000." Great question.