Cut the confusion: How to choose the scanner, Pen Test, bug bounty or platform-based security test that’s right for you.
Breaches are all too common today, as determined cybercriminals have become better organised and more targeted in their attacks. In many cases, a C-level executive loses their job as a result. That doesn’t have to be you or your organisation.
The right testing solution is key to keeping you safe. While searching for the one that’s the best fit for your organisation, be sure to prioritise your goals. Are you seeking holistic security to mitigate the chance of a breach? Are you focused solely on compliance? Is there a customer or partner insisting that you get a checkup? Are you looking for a point-in-time test or continuous security as your network and applications evolve?
Remembering those objectives as you navigate this guide will help maximise the following insights. However, before we delve into the detailed breakdown of alternatives and testing components, let’s begin with some context.
Penetration Testing has been around since the early 1970s. It has become more common as IT systems and services have evolved to become a crucial part of business operations. Organisations bring in specialists who use the same tactics, techniques and procedures TTPSS) that an attacker would deploy. This third-party test provides an accurate and unbiased assessment of network and system security.
However, as digital environments have become increasingly pervasive, so have their attack surfaces. While humans are creative, we’re finite, too. Scanners emerged in the late 1990s to provide additional scale, rather than depth, to security testing operations. Eventually, the need for additional talent and rigour in proactively finding and fixing vulnerabilities gave rise to crowdsourced security testing in the early 2000s.
28% of vulns uncovered are high severity.1 This means that without testing and remediation, the risk of breach is significant. This is something executives care about.
To be useful, each vulnerability found should be validated with explicit steps to reproduce, allowing clients to perform quick remediation.
Organisations become more secure by identifying and mitigating vulnerabilities, thereby reducing the likelihood of being breached.
Industry best practices are brought to bear on the task of securing your organisation by employing regulations and compliance criteria in the completion of a security test.
Effective security means both protecting high-value assets and strengthening the baseline level of security across the entire organisation. In 2019, over 17,000 reported vulnerabilities were identified in the U.S., with the total number of discovered vulnerabilities likely being significantly higher. To be secure, organisations need to identify and patch every critical vulnerability in every key system, as an attacker only requires one to be successful.
Security and Penetration Testing have evolved again to keep pace with continuous software development cycles and a continuous need for high-quality security insights.
Criminals sometimes focus on a particular asset and employ multiple attacks, each with several steps, to gain entry. Testing at a deep level can mitigate these kinds of attacks.
Attackers often use automated “bots” to look for easy ways into a network or asset. Broadly, yet shallowly, testing using scanners can help mitigate these kinds of vulnerabilities.
Security tests come in four basic categories:
Scanners are used for broad attack surface coverage against assets that are relatively low risk. While scanners won’t provide the depth of security testing necessary for holistic security (scanners cannot perform multi-step attacks or offer the creativity that researchers can), they will give a “wide-but-shallow” measure of resistance against known vulnerabilities. Examples of players in this category include Tenable, Rapid7, WhiteHat, and Qualys.3
While scanners are ubiquitous and inexpensive, they have some fundamental limitations when employed as standalone solutions. For example, higher-value assets will almost always require some level of human interaction. Scanners also cannot perform complex, multi-step exploits or zero-day attacks like humans can. For these reasons, although scanners are considered an essential element of a security test, they are not regarded as sufficient in themselves to obtain a realistic assessment of security risk.
What used to be a “Penetration Test” has undergone significant changes over the years. The traditional Penetration Test was designed to provide a best-effort, point-in-time, creative, and primarily manual assessment. More recently, however, the term Pen Test (especially in the private sector) has evolved into a more limited version of itself, often involving tests that are performed solely against a checklist. Throughout the remainder of this document, we will refer to the “traditional Pen Test” as the more current “downscoped” version. The majority of Pen Testing teams consist of one or two people.
The Big Four consulting firms—Deloitte, E&Y, Pwc, and KPMG—are good examples of this category. More specialised players include NCC Group, Bishop Fox, and Cypherr. And finally, there are a host of smaller, independent regional Penetration Testing firms (also known as boutique consulting firms) that utilise this process.
The efficacy of this method depends on the depth of the assessment an organisation requires and the quality of the testers available to the provider. The advantages include simplicity and finite scope. Disadvantages include the absence of competition among testers, a lack of incentive for creativity, a limited skill set applied to each vulnerability, a lack of real-time insights into findings, and delayed remediation.
Bug Bounty security testing harnesses a diverse set of testing skills, using bounties to incentivise ethical hackers to emulate the behaviour of the adversary. This allows them to evaluate the target’s overall security rather than test predefined security controls. In the process, it also allows them to fill some of the gaps where traditional Penetration Testing falls short. There are several subcategories involved in this grouping (see next page for details). Some players in this space include Cobalt, Bugcrowd and HackerOne. Many of the companies above are oriented more toward performing checklists for their broad customer base and reserving the true crowdsourcing methodology for their large enterprise customer, but they are categorised here for simplicity.
The advantage of bug bounty security testing is that it creates attractive incentives for ethical hackers to discover more vulnerabilities than a traditional Penetration Test would. A wider range of researchers and skills (often involving 50 or more researchers applying to a given test) and competition bring out overall better performance and increased depth of assessment. This category is more complex and offers varying levels of control. A good buying decision requires discernment on the part of the buyer. (See the next page for more details on the pros and cons.)
The most robust testing solution—the crowdsourced security testing platform—combines the creativity and ingenuity of crowdsourced vulnerability discovery with the methodology-driven approach of Penetration Testing, and the scalability and coverage of a high-end scanner. This enables organisations to conduct targeted Penetration Testing, find unknown vulnerabilities, and gather new intelligence in a scalable way. This intelligence then feeds into the machine-led, human-augmented scanning system, teaching it what suspected vulnerabilities look like. The platform then conducts scalable, broad attack-surface coverage of the remaining assets and identifies sources of risk for the research team to investigate.
The crowdsourced security testing platform transforms all these components into a continuous, always-on Penetration Testing process with well-orchestrated coordination between researchers, scanners, and compliance activities. It brings together a crowd of top security researchers with a high-end, AI/ML-enabled scanner and orchestrated workflows to engage them in testing. Another way to describe it is that all three components are integrated and managed by a smart platform to optimise the benefits of each modality. To this day, Synack remains the sole representative of this category, although many bug bounty players claim to be in this category.
Together, researchers and smart technology work in concert through an integrated platform, which coordinates their interactions, allowing them to augment each other to provide both high-quality insights and continuous coverage. Because of the precision that comes from the app’s smart orchestration, instead of a cap being placed on the bounty, the provider assumes responsibility for the full cost of testing, and all important vulnerabilities are brought to your attention.