NEXT GENERATION FIREWALL COMPARATIVE REPORTSecurityJULY 17, 2018Author – Thomas SkybakmoenTested ProductsBarracuda Networks CloudGen Firewall F800.CCE v7.2.0Check Point 15600 Next Generation Threat Prevention (NGTP) Appliance vR80.20Cisco Firepower 4120 Security Appliance v6.2.2Forcepoint NGFW 2105 Appliance v6.3.3 build 19153 (Update Package: 1056)Fortinet FortiGate 500E V5.6.3GA build 7858Palo Alto Networks PA-5220 PAN-OS 8.1.1SonicWall NSa 2650 SonicOS Enhanced XG Firewall 750 SFOS v17 MR7Versa Networks FlexVNF 16.1R1-S6WatchGuard M670 v12.0.1.B562953EnvironmentNSS Labs Next Generation Firewall Test Methodology v8.0NSS Labs SSL/TLS Performance Test Methodology v1.3NSS Labs Evasions Test Methodology v1.1This report is Confidential and is expressly limited to NSS Labs’ licensed users.

NSS LabsNext Generation Firewall Comparative Report – Security 071718OverviewImplementation of next generation firewall (NGFW) devices can be a complex process with multiple factorsaffecting overall security effectiveness. The following factors should be considered over the course of the usefullife of the device: Deployment use cases:o Will the NGFW be deployed to protect servers or desktop clients, or both?o How old are the operating systems and applications?Defensive capabilities in the deployment use cases (exploit block rate)Anti-evasion capabilities (resistance to common evasion techniques)Device stability and reliabilityIn order to determine the relative security effectiveness of devices on the market and to facilitate accurate productcomparisons, NSS Labs has developed a unique metric:Security Effectiveness Exploit Block Rate1 * Evasions* Stability and ReliabilityFigure 1 – Security Effectiveness FormulaBy focusing on security effectiveness as a whole instead of on exploit block rate alone, NSS is able to factor in theease with which defenses can be bypassed, as well as the reliability of the device.Figure 2 presents the overall results of the tests.Block RateEvasionsStability andReliabilitySecurity EffectivenessBarracuda Networks95.4%100%100%95.4%Check o Alto .8%Sophos93.5%25%100%25.0%Versa .1%VendorFigure 2 – Security Effectiveness1Exploit block rate isdefined as the total number of samples (live exploits and exploits from NSS Exploit Library) that are blocked under test.This report is Confidential and is expressly limited to NSS Labs’ licensed users.2

NSS LabsNext Generation Firewall Comparative Report – Security 071718NSS research indicates that NGFWs are typically deployed to protect users rather than data center assets and thatthe majority of enterprises will not separately tune intrusion prevention system (IPS) modules within their NGFWs.Therefore, during NSS testing, NGFW products are configured with the vendor’s pre-defined or recommended (i.e.,“out-of-the-box”) settings in order to provide readers with relevant security effectiveness and performancedimensions based on their expected usage.The comprehensive NSS Exploit Library covers a diverse set of exploits focused on several hundred applications andoperating systems. Protection from web-based exploits (live attacks) that are currently targeting client applicationscan be effectively measured using the NSS Labs cloud platform for continuous security validation. Figure 3 depictshow each vendor scored against live exploits and the NSS Exploit Library. For details on block rate, please see theLive Exploits and NSS Exploit Library chapters.100%95%90%Live Block 85%90%95%100%NSS Exploit Library Block RatesFigure 3 – Protection Against Live Exploits and Exploits from the NSS Exploit LibraryThis report is Confidential and is expressly limited to NSS Labs’ licensed users.3

NSS LabsNext Generation Firewall Comparative Report – Security 071718Table of ContentsTested Products . 1Environment . 1Overview. 2Analysis . 5Live Exploits. 5NSS Exploit Library. 6Exploit Block Rate by Year . 6Coverage by Attack Vector . 7Coverage by Impact Type . 9Evasions . 9Stability and Reliability . 12Security Effectiveness . 13Test Methodology . 14Contact Information . 14Table of FiguresFigure 1 – Security Effectiveness Formula . 2Figure 2 – Security Effectiveness . 2Figure 3 – Protection Against Live Exploits and Exploits from the NSS Exploit Library . 3Figure 4 –Live Exploits . 6Figure 5 – Exploit Block Rate by Year – Recommended Policies . 7Figure 6 – Attacker-Initiated Exploit Block Rate (Server Side) . 8Figure 7 – Target-Initiated Exploit Block Rate (Client Side) . 8Figure 8 – Overall Exploit Block Rate . 8Figure 9 – Attacker-Initiated Exploits and Evasions (Server Side) . 10Figure 10 – Target-Initiated Exploits and Evasions (Client Side) . 10Figure 11 – Exploits and Evasions (Combined) . 10Figure 12 – Evasion Resistance . 11Figure 13 – Stability and Reliability. 12Figure 14 – Security Effectiveness . 13This report is Confidential and is expressly limited to NSS Labs’ licensed users.4

NSS LabsNext Generation Firewall Comparative Report – Security 071718AnalysisThe firewall market is one of the largest and most mature security markets. Firewalls have undergone severalstages of development, from early packet filtering and circuit relay firewalls to application-layer (proxy-based) anddynamic packet filtering firewalls. Throughout their history, however, the goal has been to enforce an accesscontrol policy between two networks, and they should therefore be viewed as an implementation of policy.A firewall is a mechanism used to protect a trusted network from an untrusted network, while allowing authorizedcommunications to pass from one side to the other, thus facilitating secure business use of the Internet. With theemergence of HTML 5, web browsers and security threats, however, firewalls are evolving further. NGFWstraditionally have been deployed to defend the network on the edge, but some enterprises have expanded theirdeployment to include internal segmentation.As Web 3.0 trends push critical business applications through firewall ports that previously were reserved for asingle function, such as HTTP, legacy firewall technology is effectively blinded. It is unable to differentiate betweenactual HTTP traffic and non-HTTP services tunneling over port 80, such as VoIP or instant messaging. Today,application-level monitoring must be performed in addition to analysis of port and destination. Firewalls areevolving to address this increased complexity.It is no longer possible to rely on port and protocol combinations alone to define network applications. The NGFWmust be capable of determining which applications are running regardless of which ports they are using and thussecure them effectively. This section verifies that the device is capable of enforcing the security policy effectively.Live ExploitsThis test uses NSS’ continuous live testing capabilities to determine how effective products are at blocking exploitsthat are being used, or that have been used in active attack campaigns.2Protection from web-based exploits targeting client applications, also known as “drive-by” downloads, can beeffectively measured in NSS’ unique live test harness through a series of procedures that measure the stages ofprotection.Unlike traditional malware that is downloaded and installed, “drive-by” attacks first exploit a vulnerableapplication then silently download and install malware.2See the NSS Continuous Security Validation Platform for more details.This report is Confidential and is expressly limited to NSS Labs’ licensed users.5

NSS LabsNext Generation Firewall Comparative Report – Security 0717180%10%20%30%40%50%60%70%Barracuda Networks80%90%100%92.3%Check Palo Alto Networks99.4%SonicWall97.6%Sophos89.3%Versa Networks58.7%WatchGuard94.0%Figure 4 –Live ExploitsNSS Exploit LibraryNSS’ security effectiveness testing leverages the deep expertise of our engineers who utilize multiple commercial,open-source, and proprietary tools as appropriate. With more than 1,900 exploits, this is the industry’s mostcomprehensive test to date.Exploit Block Rate by YearContrary to popular belief, the biggest risks are not always driven by the latest “Patch Tuesday” disclosures. NSS’threat research reveals that many older attacks are still in circulation and therefore remain relevant.Different vendors take different approaches to adding coverage once a vulnerability is disclosed. Attempts toprovide rapid coverage for vulnerabilities that are not fully understood can result in multiple exploit-specificsignatures that may be inaccurate, ineffective, or prone to false positives. Vendors that have the resources to fullyresearch a vulnerability should be able to produce vulnerability-oriented signatures that provide coverage for allexploits written to take advantage of that flaw. This approach provides more effective coverage with fewer falsepositives.This report is Confidential and is expressly limited to NSS Labs’ licensed users.6

NSS LabsNext Generation Firewall Comparative Report – Security 071718Versaar arVendorNetworksksVendors may retire older signatures in an attempt to alleviate a product’s performance limitations; however, thismay result in inconsistent coverage for older vulnerabilities and varying levels of protection across products. Figure5 classifies coverage by disclosure date, as tracked by CVE numbers. The heat map displays vendor coverage byyear (dark green high coverage; dark red low coverage).Figure 5 – Exploit Block Rate by Year – Recommended PoliciesCoverage by Attack VectorExploits can be initiated either locally by the target (desktop client) or remotely by the attacker against a server.Since 2007, NSS researchers have noticed a dramatic rise in the number of client-side exploits, as these can beeasily launched by unsuspecting users who visit infected websites. At first, IPS products did not focus on thesetypes of attacks as they were considered the responsibility of antivirus products.This approach is no longer viewed as acceptable and, despite the difficulty of providing extensive coverage forclient-side attacks, the IPS (and NGFW) industry has attempted to provide more complete coverage of theseattacks. This is particulary important for NGFW devices, which are typically used to protect client desktops ratherthan data centers and servers; the latter comprise deployment scenarios where separate, dedicated firewall andIPS devices are more common.Attacks can be categorized as either attacker initiated or target initiated. Attacker-initiated attacks are executed remotely by the attacker against a vulnerable application and/oroperating system. These attacks traditionally target servers (which is why they are often referred to as serverside attacks).Target-initiated attacks are initiated by the vulnerable target (which is why they are often referred to as clientside attacks). The attacker has little or no control over when the target user or application will execute thethreat. Target examples include Internet Explorer, Adobe Reader, Firefox, QuickTime, and Microsoft Officeapplications.This report is Confidential and is expressly limited to NSS Labs’ licensed users.7

NSS LabsNext Generation Firewall Comparative Report – Security 0717180%10%20%30%40%50%60%70%80%90%100%Barracuda Networks97.6%Check Palo Alto Networks100.0%SonicWall98.7%Sophos97.6%Versa Networks99.9%WatchGuard99.3%Figure 6 – Attacker-Initiated Exploit Block Rate (Server Side)0%10%20%30%40%50%60%70%80%90%Barracuda Networks100%97.1%Check %Palo Alto Networks100.0%SonicWall99.9%Sophos94.3%Versa Networks99.6%WatchGuard99.4%Figure 7 – Target-Initiated Exploit Block Rate (Client Side)0%10%20%30%40%50%60%70%80%Barracuda Networks90%100%96.2%Check lo Alto Networks98.6%SonicWall99.1%Sophos94.6%Versa Networks98.5%WatchGuard98.0%Figure 8 – Overall Exploit Block RateNSS research indicates that most enterprises are forced to support a heterogeneous mix of desktop clientapplications. Further, enterprise IT departments are often unable to positively identify which client applicationsare running on their employees’ desktops, and which are not.This research provides new clarity regarding tuning best practices and indicates that it is still necessary to tune anNGFW that is protecting servers in a DMZ or data center. Research also indicates that with regard to protectingThis report is Confidential and is expressly limited to NSS Labs’ licensed users.8

NSS LabsNext Generation Firewall Comparative Report – Security 071718desktop client applications with an NGFW, it is often best to enable a (nearly) full complement of signatures, sinceit is not feasible to tune an NGFW based on specific desktop client applications.Given the rapid evolution of criminal activity targeting desktop client applications, enterprises will need todedicate more resources to client-side protection in 2018.Coverage by Impact TypeThe most serious exploits are those that result in a remote system compromise, providing the attacker with theability to execute arbitrary system-level commands. Most exploits in this class are “weaponized” and offer theattacker a fully interactive remote shell on the target client or server.Slightly less serious are attacks that result in an individual service compromise, but not arbitrary system-levelcommand execution. Finally, there are attacks that result in a system- or service-level fault that crashes thetargeted service or application and requires administrative action to restart the service or reboot the system.Clients can contact NSS for more information about these tests.EvasionsEvasion techniques are a means of disguising and modifying attacks at the point of delivery to avoid detection andblocking by security products. Failure of a security device to correctly identify a specific type of evasion potentiallyallows an attacker to use an entire class of exploits for which the device is assumed to have protection. This oftenrenders the device virtually useless. Many of the techniques used in this test have been widely known for yearsand should be considered minimum requirements for the NGFW product category.Providing exploit protection results without fully factoring in evasions can be misleading. The more classes ofevasion that are missed (such as HTTP evasions, IP packet fragmentation, TCP stream segmentation, RPCfragmentation, URL obfuscation, HTML obfuscation, resiliency, and FTP evasion), the less effective the device. Forexample, it is better to miss all techniques in one evasion category, such as FTP evasion, than one technique ineach category, which would result in a broader attack surface.Furthermore, evasions operating at the lower layers of the network stack (IP packet fragmentation or streamsegmentation) have a greater impact on security effectiveness than those operating at the upper layers (HTTP orFTP obfuscation.) Lower-level evasions will potentially impact a wider number of exploits; missing TCPsegmentation, for example, is a much more serious issue than missing FTP obfuscation.TCP Split Handshake attacks can deceive the IPS engine into believing that the traffic flow is reversed and the IPSdoes not need to scan the content, which exposes the NGFW to previously known attacks.The resiliency of a system can be defined as its ability to absorb an attack and reorganize around a threat. When anattacker is presented with a vulnerability, the attacker can select one or more paths to trigger the vulnerability.NSS will introduce various, previously unseen variations of exploits to exploit the vulnerability and measure thedevice’s effectiveness against them. A resilient device will be able to detect and prevent against