NSS Labs’ test methodologies are constantly evolving to reflect the needs of enterprise consumers. Last year, NSS introduced a “resilience” category to our security testing for inline devices (e.g., next generation firewalls [NGFWs], next generation intrusion prevention systems [NGIPS]). NSS defines resilience as a product’s capability to continue providing protection for a vulnerability against a known exploit after various modifications have been made to the original exploit.

Resilience is an important capability since exploit modifications often require little technical skill to implement. For example, a script-based drive-by exploit delivered in HTML may include a great deal of content that can be easily modified, such as the names of variables and functions; the order in which they are declared; adding and removing whitespace; swapping payload; changing comments; and/or a combination of the techniques just described.

Enterprises rely on network security products to provide protection for unpatched and/or vulnerable applications in their environments, picking off exploitation attempts “on-the-wire.” The goal of NSS’ resilience testing is to evaluate the quality of a product’s vulnerability protection signatures in order to determine whether they can adequately detect or match the essential exploitation elements of a vulnerability (in other words, the “trigger”), or can be easily bypassed by a relatively novice adversary making simple changes to a readily available POC exploit.

Inline security products are largely dependent on pattern-matching signatures (e.g., “match if this string of bytes is seen within x bytes of this other string of bytes”) to identify malicious content after normalization of network streams. These products are unlikely to further process the content (e.g., decode, deobfuscate, or render the HTML and associated script) [1] prior to inspection due to the impact it would have on throughput.

I wanted to share some resiliency results from our most recent NGFW v8.0 and NGIPS v4.0 public tests. In these tests, we used a well-known public drive-by exploit for CVE-2014-6332 (a.k.a. “Windows OLE Automation Array Remote Code Execution Vulnerability”). We chose this exploit because of the variety of content available for modification and for its reliability in delivering payloads to our intended victim (Microsoft Windows 7 SP1 with Internet Explorer 11).

Our set of test cases started with basic content transformation cases where a single technique was applied. Techniques were then stacked together, increasing the likelihood that we would modify or break up signature strings used by the device under test. Techniques utilized for this particular exploit included:

• • Adding whitespace (spaces and linefeeds)
  • Renaming of VBScript procedures and variables
  • Varying chr()/chrw()/chrb() VBScript function usage for character conversion
  • VBScript obfuscation with online tool (character/command conversion utilizing Execute()/chr()/CLng() functions)
  • Payload swapping/obfuscation
  • Combining/reordering of VBScript functions
  • String splitting/combining
  • Numeric value/equation modifications

In order to confirm that our traffic was being properly classified and normalized prior to inspection, we also delivered each of our test cases in chunked and/or compressed streams, and/or over a different TCP port.

As a result, during resilience testing 10 out of 10 NGFW products (10 of 10 participating vendors) were bypassed, and six out of seven NGIPS products (five of six participating vendors) were bypassed. Representative subsets of the resilience cases tested, along with total product bypass counts for each test case delivered over TCP port 80, follow (‘ P ’ = pass/block; ‘ F ’ = fail/miss):

Figure 1 – NSS Labs’ NGFW 8.0 Resilience Test Results

Figure 2 – NSS Labs’ NGIPS v4.0 Resilience Test Results [2]

As you can see, individual techniques generally had little impact. But as techniques were combined, the contents of the original exploit were obfuscated enough that pattern-matching signatures began to fail, as illustrated by the increase in bypasses. The results also indicate that some vendors’ signatures were more resilient than others.

While possible, further processing beyond normalization and categorization of streams prior to inspection is historically a relatively expensive proposition for these inline security products. An NGFW or NGIPS could go so far as to send files (including HTML) to the cloud or another on-premises sandbox for execution/deobfuscation and/or more extensive analysis before rendering a verdict to block or allow, but the delay would likely be considered unacceptable to the average user, particularly for something like web browsing. More extensive analysis to detect and block more complex attacks is generally reserved for other layers of a “defense-in-depth” security posture, such as endpoint, proxy, and/or sandbox security technologies, where the content can be more thoroughly normalized and/or detonated prior to inspection.

For NGFW and NGIPS devices, short of additional processing, pattern-matching signatures that identify heavily or suspiciously obfuscated content—not the underlying malicious content itself, can provide some additional protection but are also prone to false positives. Signatures that match fingerprinted delivery techniques and/or payloads by various exploit kits and threat groups can also be utilized. But the most resilient vulnerability protection signatures require extensive research and understanding of a CVE in order to write intelligent signatures targeting the essence/trigger of the vulnerability (relatively expensive but resilient), as opposed to simply adding strings seen in public exploits to a list to pattern-match against (relatively cheap but not resilient).

So, what’s the takeaway? While our resilience testing reveals some limitations, NGFW and NGIPS technologies remain an important part of a “defense-in-depth” security strategy. Our testing indicates which vendors are making the requisite investment in time and expertise to write signatures that offer the most resistance to exploit modifications. Combined with performance and other evasions testing with the same device configurations, we can determine which products offer not only the most effective security, but also which are most efficient at doing so. For additional details, including which vendors missed which resilience cases, individual test reports are available in the Research Library.

[1] Additional NGFW JavaScript obfuscation testing data is available here: Wait, I thought my NGFW Detected That

[2] While the NGIPS device from Vendor 5 blocked all test cases delivered over TCP port 80, nearly every test case delivered over a specific non-standard TCP port bypassed the device.