For the past several years, Breach Detection Systems have been one of the most rapidly adopted security technologies. And now NSS Labs has just published the much anticipated first Breach Prevention System Group Test.
As you know, NSS Labs is committed to providing fact-based data and objective group test results that enable organizations to make better informed cybersecurity purchasing decisions. As with all NSS Labs group tests, there is no fee for participation, and the test methodology is available in the public domain to provide transparency and to help enterprises understand the factors behind test results. To learn more about our group test policies, click here.
On occasion, there are results that we feel would benefit from additional context. While we provide detailed prevention and detection rates in the Individual Test Report we realize that many will only look at the Security Value Map (SVM) and not read those detailed results. Given that, and after discussions with Juniper Networks, one of the test participants, we believe there is more worth sharing. We believe in this case there is more to be considered than the SVM depicts and have decided to make the Juniper BPS Test Report available to the public so that they have the best possible information available with which to make an informed decision. More below:
According to Juniper, “With pretty much all solutions, if a day zero malware has to go through the sandbox for detection, the sandbox detection times could go into minutes. While some vendors solutions are designed to hold the file longer for the verdict and then make a decision to block or not, Juniper’s solution is designed to provide optimal user experience and the file is sent through if there is no cache hit.”
Security and usability are often a tradeoff and it is understandable why a vendor might choose this approach in an attempt to achieve that balance. It certainly isn’t the first time in our industry this design choice has been made to improve user experience and this statement by Juniper is consistent with what we observed during the BPS Group Test.
Juniper then goes on to state, “…and then a remediation action is initiated through multiple ways. This remediation action is by communicating with an end-point partner and/or through applying a policy on the network element closest to the end-point(switch) through SDSN Policy Enforcer(PE).”
NSS Labs believes that a remediation action is a useful and valuable tool in the defensive arsenal though it is no replacement for preventing the compromise of a system. The BPS test accounts for post-compromise detection through the adjustment of the TCO as a recognition of the value of operational efficiency. This is limited to TCO because a detection or remediation through infrastructure or a third-party endpoint is post-compromise and suffers from lost visibility. It is a mitigation action driven by detection, not a prevention action.
Juniper further states, “ In the NSS tests, we neither tested an end-point solution nor did we test with PE.”
This statement by Juniper is consistent with what we tested and what we observed during the BPS Group Test. While it is possible a third-party endpoint solution might have improved the protection offered by the solution on its own merits, no remediation action would have changed the results we observed. While the Juniper solution does offer the benefit of incremental prevention of known threats, it is not resistant to common evasion techniques.
When asked why they chose to not submit a complete solution, Juniper’s response was that they believed the test was a mixture of detection and prevention vs. being a prevention test. While it is true that this Breach Prevention System test accounts for detection within 15 minutes of compromise, that detection credit manifests in the operational cost of the product and does not apply to the security effectiveness score. This same approach also extends to measuring and scoring of evasions. Juniper detected a number of evasions post-compromise and was given credit in the TCO for these detections however that credit also does not extend to the Security Effectiveness metric.
It is also important to note that Juniper additionally detected several evasion cases after the 15-minute window of opportunity used in this test and in accordance with our criteria, they did not receive credit for those. This is a result of the high standard of the test and our commitment to equal assessment and comparison, not the failing of the product to ultimately detect these techniques. In a test with a less stringent window of opportunity Juniper may have detected more evasion test cases than are reported here.
So why not pull Juniper out of the Group Test? First, NSS Labs’ longstanding policy is that once a vendor has seen the results of a public test, we will not pull that product from the test. And second, Juniper is marketing their ATP solution as a prevention product that also protects against evasive malware. It is our belief that given this marketing, enterprises are purchasing the product with the expectation that it will prevent (block) known and unknown threats. While it is unfortunate that the BPS test does not reflect the full capabilities of the Juniper product, we believe the results are accurate for the assessment as a Breach Prevention System. We look forward to the opportunity to assess the full Juniper solution in our next Breach Prevention Group test and our Breach Detection group test.
As always, we encourage enterprises to read the individual test reports for all submitted products to determine which are best suited to meet the requirements of their business use cases.