We don’t follow up every NSS Labs test with a blog response to a vendor, but after the fun and games following our recent BDS test, we find ourselves in a similar position. This time it is Palo Alto Networks blogging about our NGFW group test, the results of which were published last week and can be found here.
While Lee Klarich’s blog was very carefully worded, he never actually addressed the main issue at hand: Palo Alto Networks NGFW misses several critical evasions that leave its customers at risk. The blog did, however, contain some serious inaccuracies that I would like to address:
| PALO ALTO NETWORKS’ CLAIM | NSS’ response |
| “Palo Alto Networks intentionally did not participate in the 2014 NSS Next-Generation Firewall Comparative Analysis report that was recently published. This means that unlike all of the other vendors in the report who configured and tuned their products specifically for this test, there was no input from us on the configuration of our device.” | PARTICIPATION IN AN NSS GROUP TEST IS NOT OPTIONAL – if you sell into a particular market, or if our enterprise clients want to see your products tested, then we will test them. It is always worrying when a vendor is so resistant to having its product tested – usually an indication that it’s engineers know there is something wrong.
Palo Alto Networks was treated exactly the same as every other vendor in this test. NSS tests all NGFW products with the predefined vendor recommended settings. |
| “The reason we did not participate in this test is that over time we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a “pay to play” approach and produces questionable objectivity and accuracy in results.” | “Pay to play” is a very strong accusation and is dead wrong, and Palo Alto Networks knows it (click here to see the public apology Palo Alto Networks was forced to make the last time it slandered NSS Labs over this issue).
The fact is that NSS does not charge any vendor for participation in any of our public group tests. THE ENTIRE TEST IS DONE ON OUR DIME, and the only input we ask from vendors is support in terms of supplying the most appropriate device, along with engineering support before and during the test, should we need it. As far as tuning goes, WE DO NOT ALLOW VENDORS TO TUNE THEIR NGFW DEVICE. All NGFW products are tested using the predefined vendor recommended settings – the exact same settings that are used in deployments by enterprises around the world. This applies to all NGFW products. When it comes to NGFW, NSS research shows that most customers deploy these devices with the default/recommended configuration out of the box. This, therefore, is how we deploy NGFW’s in our test harness. To reiterate NO TUNING IS PERMITTED. |
| “One year ago, we did participate and scored 96.4%.” | True. In fact, when Palo Alto Networks does well, it likes to point out how good our tests are. |
| “We take the efficacy of our Next-Generation Firewall very seriously. We are trying to understand why they could have come to such a drastically different result compared to the same tests run against the same technology in 2013.” | It was not the same technology. If you study the SVM graphic and the test results, you will see that the actual block rate was not significantly lower in 2014 (92.5%) than it was in 2013 (96.4%). The slightly lower score was due to a reduced ability to block recent exploits (particularly 2013-2014 exploits). However, where version 4.1.9 of PAN-OS handled evasions properly version 6.0.3 of PAN-OS tested in 2014 proved to be susceptible to multiple evasion techniques, which resulted in an overall reduction in security effectiveness (60.1%). |
| “The issues they’ve raised have never been observed in other tests conducted internally or with our install base of over 19,000 global enterprises.”; | This is why independent testing is so important. If you don’t know how to test for evasions, you will never see what NSS engineers saw in this test. And if you are a customer with this technology deployed in your network, you will never see an indication that anything is amiss until you get a call from a law enforcement agency letting you know you’ve been breached. By definition, evasion techniques allow the attacker to evade detection by the NGFW device, meaning that nothing will ever show in the logs. |
| “It is also interesting to note that they say that we updated our OS in that time and broke the technology. There is no basis for that claim…” | The 2013 test was based on PAN-OS v4.1.9. The 2014 test was based on PAN-OS v6.0.3. That is two major revisions of the OS. It is reasonable to conclude that bugs were introduced that led to the failure in our evasion tests. Either that or Palo Alto Networks intentionally disabled certain evasion protections in order to improve performance of the device. |
As you would expect in a test of this type, Palo Alto Networks was not the only vendor whose product finished below the average security line on our SVM. The difference is that the other vendor didn’t bother attacking NSS in public but instead focused on fixing the issues found in the test, and will shortly be resubmitting that product for public testing. Cyberoam is to be commended for this approach.
Since Palo Alto Networks has decided to launch a PR offensive in lieu of fixing the deficiencies in its product, we are providing the information to the public for free. This will allow the company’s customers to at least attempt to improve the protection offered by PAN-OS devices to the best of their ability until such time as Palo Alto Networks decides to take this issue seriously. Here are the key points:
- All PAN-OS devices require a configuration change to detect even the most basic TCP stream segmentation evasions. The “Mismatched overlapping TCP segment” protection in the Zone Protection profile is not enabled by default, which allows attackers to bypass the device completely using TCP stream segmentation with overlapping data evasion techniques. NSS strongly recommends that this protection is always enabled – any PAN customer that has not checked this box is at EXTREME RISK.
- All PAN-OS devices will fail to detect attacks utilizing TCP Split Handshake Spoofing unless SYN flood protection is enabled, SYN Cookies are selected as the protection action, and unless the SYN flood packet rate threshold is triggered. This may not be a workable solution in many real-world environments, leaving the device open to another possible evasion method.
It should be noted that according to the test methodology for NGFW, the device should have been deployed with vendor-preconfigured settings only (i.e., default “out of the box”). However, we elected to enable these settings to minimize the negative impact on the Palo Alto Networks device in the test in an effort to be as fair and transparent as possible, since the company had refused to participate in the test.
NOTE: SINCE THESE EVASIONS CAN BE MITIGATED VIA A CONFIGURATION CHANGE, THEY WERE NOT COUNTED AGAINST PALO ALTO NETWORKS IN THIS TEST AND WERE NOT THE CAUSE OF ITS CAUTION RATING.
A CAUTION RATING WAS ISSUED BECAUSE PRODUCTS RUNNING PAN-OS V6.0.3 ARE SUSCEPTIBLE TO ADDITIONAL SEVERE EVASION FAILURES, WHICH CANNOT BE PUBLICLY DISCLOSED WITHOUT PUTTING PALO ALTO NETWORKS CUSTOMERS AT RISK, SINCE THERE ARE CURRENTLY NO KNOWN WORKAROUNDS. THIS MAY ALSO AFFECT OTHER VERSIONS RELEASED AFTER THE LAST KNOWN GOOD VERSION TESTED BY NSS, V4.1.9.
All NSS clients have full access to details of the evasions missed and are welcome to discuss these personally with our analysts and test engineers as part of their subscription.
As a public service to Palo Alto Networks customers, who are likely at severe risk, we are making this same information available FREE OF CHARGE. If you are a Palo Alto Networks customer, please click here and we will provide you with the full Palo Alto Networks PA-3020 v6.0.3 Product Analysis Report.*
In the interim, I sincerely hope Palo Alto Network executives will begin to take this issue seriously and move quickly to protect their customers. As ever, NSS is more than willing to work with any vendor to fix this kind of problem and help make our networks safer. Further, if/when Palo Alto Networks fixes its product, we will test it to confirm, and will publish the results immediately. Until that happens, we are also happy to work with Palo Alto Network customers to help minimize their risk as far as possible given the current limitations of the device.
*UPDATE: Palo Alto came in with a new version of the product (v6.0.5-h3) in October 2014 which fixed the evasions discussed in this blog.