When Firewalls Fail Gracefully

The latest NSS Labs Enterprise Firewall Comparative Report was published this month and, as usual, provided a deep insight into the state of the enterprise firewall market.

Seven of the most widely deployed products were tested using real-world attack scenarios, enterprise-grade workloads, and adversarial evasion techniques to measure their resilience, reliability, and performance.

The results reveal a security landscape that remains uneven: most products blocked the majority of exploits and malware, but a few stumbled when exposed to modern, and not so modern, evasion techniques.

However, the story doesn’t end with the Comparative Security Map – it is also a case study in vendor accountability. How vendors respond when weaknesses are exposed in independent tests such as this tells us a lot about how they are likely to support their enterprise customers in a pinch. It also tells us how seriously they take engineering challenges that could result in serious failures, or even breaches, when installed in live environments.

Palo Alto Networks and Fortinet, though not the highest-scoring participants, stand out precisely because they treated the findings as an opportunity to rectify shortcomings in their products that could have a serious impact on their customers. Within days of publication, both vendors confirmed patches for the issues identified and scheduled retests for the affected products. That kind of responsiveness deserves as much attention as raw test scores.

The Test That Matters

NSS Labs enterprise-firewall evaluations are the most comprehensive in the industry. The 2025 round measured not only exploit and malware detection, but also resilience against 53 evasion categories, false-positive accuracy, TLS/SSL handling, and sustained throughput under realistic enterprise workloads.

In other words, this isn’t a marketing test with cherry-picked “perfect” network traffic and well-known basic exploits and malware. Each firewall was deployed in-line between trusted and untrusted networks, then stress-tested with:

  • A broad range of “real world” network traffic designed to emulate typical enterprise traffic, both encrypted and plain text.
  • 3,326 exploit samples from vulnerabilities found in the wild in enterprise environments.
  • 11,311 malware samples drawn from active campaigns.
  • 5,752 evasion variations spanning 53 evasion categories, crafted to bypass defenses.
  • 55 performance stress tests spanning HTTP, HTTPS, and UDP traffic, created to measure throughput, stability, and reliability under stress.

This combination produces an in-depth view of security efficacy, together with an evaluation of performance using mixtures of real-world traffic. In today’s enterprise networks, where more than 95 percent of web sessions are HTTPS, it is important for firewalls to be able to handle encrypted traffic.

How the Vendors Fared

Three of the seven firewalls achieved Recommended ratings: Check Point, Juniper Networks, and Versa Networks. All delivered security effectiveness above 99 percent with false-positive accuracy in the high 90s.

Three vendors received Caution ratings: Cisco, Fortinet, and Palo Alto Networks. Their placement wasn’t due to catastrophic malware or exploit detection failures, since each still handled most malicious payloads effectively, but because of critical failures in their ability to resist low-level evasion techniques.

This continues to be an issue today, just as it was at the inception of NSS Labs 1.0 in 2007. You might think that we should be seeing 100% resistance by now, but instead coverage appears to be cyclical. It seems that vendors will work hard to build robust code that handles evasions well, but later engineering teams deprioritize that area of development, or complex new features simply break it.

Two key points are evident:

  1. Evasion handling is a powerful differentiator today, just as it has always been.
  2. Throughput disparities can be significant, especially when encrypted traffic is thrown into the mix.

What Went Wrong—and Right

While malware and exploit detection rates across the board were excellent (most above 99 percent), the evasion results exposed real-world risk. A single missed evasion can allow bad actors to reuse entire classes of exploits allowing malicious traffic to go undetected.

Cisco failed one critical TCP-segmentation evasion, reducing its exploit-evasion resistance to 40 percent; Fortinet missed one transport-layer variant, scoring 60 percent; and Palo Alto Networks failed both network and transport-layer categories, resulting in 0 percent exploit-evasion resistance.

Why Responsiveness Matters

However, it is not all about pure test results, but rather how a vendor responds to those results that really matters. That defines the kind of relationship they are likely to have with their customers, and how seriously they take their engineering mission. In cybersecurity, perfection is fleeting. Every product eventually encounters a configuration bug or parser flaw. What separates mature vendors from pretenders is how quickly and transparently they respond.

Palo Alto Networks and Fortinet publicly acknowledged the test outcomes, issued software updates within a couple of weeks, and scheduled retesting. That is what enterprise customers should be looking for from their security partners: transparency and the willingness to participate in independent tests in the first place, followed by the desire to act on the results of those tests to improve their product expediently where necessary.

NSS Labs urges enterprises to hold vendors accountable and demand transparency. Vendors who view testing as collaboration rather than confrontation, will build lasting trust as well as solid products.

Performance Under Pressure

Security effectiveness means little if performance tanks under real workloads. NSS Labs Rated Throughput metric weights encrypted traffic at 95 percent, mirroring modern conditions. Versa achieved the highest sustained throughput (7.6 Gbps) with strong security; Juniper balanced speed and protection; Fortinet offered excellent value; Palo Alto trailed but excelled in accuracy.

False Positives: The Hidden Cost

NSS Labs replaced its previous price-per-protected-megabit metric with false-positive accuracy as a more meaningful measure of operational overhead. Cisco’s 80 percent accuracy implies legitimate traffic was incorrectly blocked one-fifth of the time, which may cause issues in live deployments. Conversely, Palo Alto, Versa, and Fortinet all exceeded 99 percent in terms of resistance to false positive scenarios.

The New Baseline: Encryption Everywhere

With more than 95 percent of global web traffic encrypted, enterprise firewalls need to be able to handle it without suffering significant performance degradation. All firewalls handled decryption properly, but some paid steep penalties in terms of performance. Versa and Juniper maintained 80–90 percent efficiency, while Palo Alto and Cisco lagged near 70 percent.

Beyond the Scoreboard

At first glance, a Caution rating in the CSM might appear damning, but within weeks those numbers will likely change as fixes are validated and re-tested. Resilience isn’t static; what defines market leadership is the ability to recover quickly, transparently, and collaboratively.

Independent testing remains the crucible through which trust is forged. The vendors who embrace scrutiny, fix what’s broken, and invite another round of validation are the ones enterprises should bet their networks on.

Because in the end, cybersecurity isn’t about being flawless. It’s about being fast, honest, and relentless in pursuit of better protection.

Beyond Assumptions: Why Validation is the Next Frontier in Cybersecurity Defense

Last week, CISA published an incident response report detailing how a federal civilian executive branch (FCEB) agency was breached through exploitation of a known and documented vulnerability in GeoServer (https://www.cisa.gov/news-events/alerts/2025/09/23/cisa-releases-advisory-lessons-learned-incident-response-engagement). This was not a “sophisticated zero-day,” but a widely reported weakness defenders have been aware of for some time (https://nvd.nist.gov/vuln/detail/cve-2024-36401).

This breach underscores a sobering reality: attackers don’t need innovation when defenders rely on assumptions.

Known Exploits, Unknown Effectiveness

Each time an advisory like this is released, many CISOs and CTOs are left asking the same question: “Would this have worked against us?”

The uncomfortable truth is that in many environments, the answer is uncertain. Security leaders often deploy products with the expectation of protection, but without direct validation, those expectations may not hold under real-world attack conditions.

This is why we at NSS Labs regularly evaluate security products against actual exploit samples going as far back as 10 years—including the very vulnerability used in this breach. That type of validation gives defenders evidence, not just hope, that their technologies will withstand known threats.

The Flaw in “Defense-in-Depth by Assumption”

Defense-in-depth is a well-established strategy. Multiple layers of technology—firewalls, intrusion prevention, endpoint agents, and monitoring—create redundancy and resilience. But the mere presence of these controls is not enough.

  • Deployment ≠ Effectiveness. A product installed in the stack doesn’t guarantee it will perform as intended.
  • Context Matters. Effectiveness depends on how controls are configured, tuned, and integrated into the environment.
  • Silent Gaps Exist. Without validation, security teams may not realize that certain attack vectors bypass defenses entirely.

The CISA advisory makes clear: organizations cannot rely on “best practice” architectures alone. They must prove their defenses actually work.

Validation as the Next Frontier

Cybersecurity has long emphasized the importance of prevention and detection. The next frontier is validation: treating effectiveness as a measurable, verifiable outcome.

Validation is not theoretical—it’s practical. The critical difference between assumption and assurance is data. Testing security products against real exploits, simulating adversarial behavior, and quantifiably measuring whether defenses hold provides assurance that investment translates into protection. Independent testing bodies such as NSS Labs help provide this evidence, bridging the gap between vendor claims and operational reality.

A Practical Checklist for CISOs

Security leaders looking to strengthen their posture against known vulnerabilities can use the following framework:

  1. Inventory What Matters
    1. Catalog critical applications, platforms, and workloads.
    2. Prioritize those most tied to mission outcomes.
  2. Map Defenses to Assets
    1. Identify which controls protect which workloads.
    2. Look for overlaps, blind spots, and single points of failure.
  3. Validate Against Exploit Samples
    1. Test defenses against real-world exploits and malware, not just lab simulations.
    2. Leverage independent testing where available.
  4. Simulate Adversarial Behavior
    1. Ensure that each component in the defensive chain is independently tested and validated.
    2. Focus on tactics, techniques, and procedures (TTPs) that correspond to those utilized by threat actors in the current threat landscape.
  5. Make Validation Continuous
    1. Move from one-time testing to an ongoing validation cycle.
    2. Adjust configurations, patching, and investments based on results.

Final Thoughts

The lesson from this breach is clear: known vulnerabilities remain one of the most significant risks to enterprise security. Defense-in-depth alone is insufficient if it is built on untested assumptions.

The industry must embrace validation as a core pillar of cybersecurity strategy. By demanding measurable proof of effectiveness—through independent testing, adversarial approaches, and continuous validation—CISOs and CTOs can move from assumption to assurance.

The goal is simple yet profound: to ensure that when the next advisory is issued, leaders can answer with confidence:

“Yes—our defenses will hold.”

Reintroducing NSS Labs: A New Chapter in Cybersecurity Assurance

AUSTIN, TX – July 9, 2025 — NSS Labs, long known as the gold standard for independent security product testing, is back—with a new ownership structure, revitalized leadership, and a mission laser-focused on the evolving demands of today’s cybersecurity and advanced technology ecosystem. Led by Founder and CEO Vikram Phatak, NSS Labs 2.0 is now wholly owned and operated by its senior partners and executive team.

The reimagined NSS Labs will deliver confidential, data-driven testing services tailored to three core audiences:

  • Enterprises will benefit from objective testing and continuous validation of security technologies—on-premises, in the cloud, or delivered via third-party services. These assessments support strategic initiatives including risk governance, supply chain validation, vendor accountability, and regulatory compliance.
  • Security Vendors will gain rigorous third-party validation of product efficacy using real-world attack scenarios. Testing data will help refine product strategy, accelerate go-to-market timelines, and support credible claims in a competitive market.
  • Service Providers will receive evaluations built for multi-vendor environments, helping them benchmark offerings, support procurement decisions, and communicate service value with independently validated data. Testing also supports operational readiness and future roadmap development.

In addition to its commercial services, NSS Labs is also the Official Testing Partner of CyberRatings.org, the non-profit that publishes public test results and research on cybersecurity technologies. NSS Labs contributes by developing test methodologies, authoring both individual and comparative reports, and producing educational and thought leadership content for the broader cybersecurity community.

Originally founded in 2007, NSS Labs quickly became a globally trusted authority, shaping product development, procurement, and strategy decisions across the cybersecurity ecosystem. After private equity ceased operations in 2020, Phatak acquired select assets and intellectual property from the custodian of NSS Labs 1.0 and recognizing renewed demand from stakeholders, reassembled veteran talent and new industry leaders to launch NSS Labs 2.0.

“We’re not just relaunching NSS Labs—we’re rebuilding it for the future,” said Phatak. “We’ve preserved the integrity and rigor that put NSS Labs on the map, and supercharged it with interactive tools, modern methodologies, and a team with decades of hands-on experience evaluating cybersecurity products across a wide range of categories—including endpoint, cloud, AI, and post-quantum cryptography.”

What’s New at NSS Labs 2.0

  • Interactive, Data-Driven Tools: Stakeholders can engage with test data through intuitive interfaces, enabling real-time comparison and deeper insights into product performance.
  • Expanded Testing Portfolio: Beyond traditional technologies like enterprise firewalls and SD-WAN, NSS Labs now evaluates advanced solutions such as AI/ML-powered tools, ransomware defenses, and post-quantum cryptographic systems.
  • Tailored Services by Audience: Purpose-built programs for enterprises, vendors, and service providers combine transparency, speed, and rigorous technical validation.

A Legacy That Inspires Confidence

NSS Labs 2.0 draws on decades of hands-on experience testing hundreds of cybersecurity products across diverse categories. Each evaluation begins with the development of robust, detailed methodologies, followed by rigorous real-world testing to ensure products meet the demands of today’s fast-evolving threat landscape.

The lab’s live, real-time infrastructure tests against a broad array of exploits, evasions, malware, and malicious URLs—mirroring the complexity of real-world attacks. Unique to NSS Labs are thousands of hand-crafted evasions that mimic sophisticated threat actor techniques, designed specifically to bypass detection systems. Scalable cloud infrastructure simulates large enterprise environments, enabling high-fidelity assessments under realistic conditions. 

Led by Visionaries, Powered by Experts

The new NSS Labs is guided by a leadership team with deep expertise in cybersecurity testing, strategy, and innovation:

  • Vikram Phatak, Chief Executive Officer & Founder
  • Ian Foo, Chief Technology Officer & EVP of Product
  • Carma Austin, Chief Strategy Officer
  • Cathy Main, Chief Marketing & Communications Officer
  • Tim Otto, Vice President of Test Operations
  • Thomas Skybakmoen, Vice President of Research

To learn more about NSS Labs and its services, visit www.nsslabs.com.

About NSS Labs

NSS Labs delivers research-backed insights through its advanced testing platforms, empowering enterprises, security vendors, and service providers to make informed, evidence-based cybersecurity decisions. By handling the heavy lifting of testing for effectiveness, performance, and suitability, NSS Labs helps clients move beyond assumptions to gain actionable clarity. Its auditing and governance services offer continuous assurance that deployed security technologies are performing as expected—protecting investments and supporting accountability.