We recently published a methodology specific to evasions testing at NSS Labs. Evasions testing is a category of the Security Effectiveness testing that is included in most of our test methodologies. This new evasions test methodology provides a more detailed description of how NSS performs evasions testing and highlights key aspects of our private and public (or “group”) tests.
Private testing is an opportunity for vendors to have their products tested by NSS in order to identify and remediate any defects in GA or non-GA code builds, and/or new or existing hardware, with the knowledge that these results will not be made public. Private tests also provide vendors and enterprises with visibility into the robustness of products and policies prior to deployment. These engagements are paid for and protected under NDA. Private tests competently exercise all aspects of a product against NSS’ current test methodology and reveal areas that require improvement. Public (group) tests are not paid for, are executed against current methodologies—and fully exercise all aspects of a product under that methodology; are only performed with GA code on commercially-available hardware, and the results are made public. To ensure that any issues identified in previous testing have been adequately addressed, and to confirm that products are indeed providing the level of protection claimed (and not just becoming more proficient at taking the same test), we modify our testing between private and public test cycles for any given test methodology.
During the NGFW v7.0 private testing engagements, we used a publicly available tool called “HTTP Evader” for HTTP evasions testing. HTTP Evader delivers an EICAR test file utilizing a variety of HTTP evasion techniques, providing a total of over 700 test cases. For NSS’ public testing, we used a tool developed in-house dubbed “Evader++,” which delivers a live exploit targeting CVE-2014-6332, and for which all vendors being tested had base coverage (i.e., the exploit was blocked with no evasion techniques applied). Evader++ consists of a subset of the 700 HTTP Evader test cases, implementing 26 test cases with live exploits, and utilizes the same HTTP evasion techniques included with HTTP Evader—sometimes using different combinations of those techniques. So, the key differences between NGFW v7.0 HTTP evasions in NSS’ private and public testing are: different tools, different payloads, different combinations of evasion techniques.
The NGFW v7.0 public test HTTP evasions results were somewhat surprising: six out of the 10 participating vendors missed at least one of the 26 HTTP evasion cases, with one vendor missing 11 cases, and another missing 10. Given the trivial nature of reproducing these evasion techniques, we decided to give vendors 90 days from notification of the test results to remediate the misses before making details of the test cases known. Only generic “Test Case 1” through “Test Case 26” identification numbers were provided in the individual test reports. To facilitate timely remediation, each vendor was provided with packet captures for each of the evasion test cases their product missed, and when necessary, follow-up web conferences were held with individual vendors to walk through reproduction of the misses in our lab.
Now that the vendors have had 90 days to fix any issues and provide updates to their customers, we are providing descriptions of the 26 test cases used by Evader++ during NSS’ NGFW v7.0 group test:
For details on which vendors missed which evasions, along with requirements for remediation, individual test reports are available in our library.
And one final note: We will be expanding the number of Evader++ HTTP evasions test cases we use in the near future, and we will be evolving the Evader++ framework to include testing of other evasion methods, such as HTML and network evasions. Stay tuned.