Home / Resources / Penetration Testing / The Buyers Guide to Penetration Testing

Cut the confusion: how to choose the scanner, pentest, bug bounty, or platform-based security test that is right for you.

Breaches are all too common today as determined cyber criminals have become better organised and more targeted in their attacks. In many cases, a C-level executive loses their job as a result. That doesn’t have to be you—or your organisation.

The right testing solution is key to keeping you safe. While searching for the one that’s the best fit for your organisation, be sure to prioritise your goals. Are you seeking holistic security to mitigate the chance of a breach? Are you focused solely on compliance? Is there a customer or partner insisting that you get a checkup? Are you looking for a point-in-time test or for continuous security as your network and applications evolve?

Remembering those objectives as you navigate this guide will help maximise the following insights. But before we go into the detailed breakdown of alternatives and testing components, let’s start with some context.

Genesis of security testing

Penetration testing has been around since the early 1970s. It’s become more common as IT systems and services have evolved to be a crucial part of business operations. Organisations bring in specialists who use the same tactics, techniques, and procedures (TTPs) that an attacker would deploy. This third-party test provides an accurate, unbiased assessment of network and systems security.

However, as digital environments became increasingly pervasive, so did their attack surfaces. While humans are creative, we’re finite, too. So, scanners emerged in the late 1990s to provide additional scale (if not depth) to security testing operations. Eventually, the need for additional talent and rigor in proactively finding and fixing vulnerabilities gave rise to crowdsourced security testing in the early 2000s.

Benefits of pentesting

Executive buy-in

28% of vulns uncovered are high severity.1 This means that without testing and remediation, the risk of breach is significant. This is something executives care about.

Actionable data

To be useful, each vulnerability found should be validated with explicit steps to reproduce, giving clients the ability to do quick remediation.

Vulnerability risk reduction

Organisations become more secure by finding and reducing vulns, thereby mitigating the ways in which they might be breached.

Unbiased assessment

Industry best practices are brought to bear on the task of securing your organisation by employing regulations and compliance criteria in the completion of a security test.

Effective security strategy

Effective security means both protecting high-value assets and shoring up the base level of security across the entire organisation. In 2019, there were over 17,000 reported vulnerabilities in the U.S. (and the total number of discovered vulnerabilities is likely much higher).2 To be secure, organisations need to find and patch every critical vulnerability in every important system since an attacker only needs one to be successful.

What do we mean by test depth and breadth?

Security and penetration testing has evolved again to keep pace with continuous software development cycles and a continuous need for high quality security insights.


Criminals sometimes focus on a particular asset and perform many attacks with multiple steps to try to get in. Testing at a deep level can mitigate these kinds of attacks.


Attackers often use automated “bots” to look for easy ways into a network or asset. Broad (but shallow) testing using scanners can shore up these kinds of vulnerabilities.

Types of security tests

Security tests come in four basic categories:

  • Scanning using software to search for vulnerable or unauthorised systems and services [machine-led]
  • Traditional Penetration which involves evaluating systems for common vulnerabilities, leveraging the Open Web Application Security Project (OWASP) or other standards body [consultant-led]
  • Bug Bounty Testing in which researchers are allowed to attack the asset in their own creative ways, incentivised by bounties [crowd-led]
  • Crowdsourced Security Testing Platform which combines the best elements of the above three categories—this is the next generation of pentesting [platform-led, human-enabled]


Scanners are used for broad attack surface coverage against assets that are relatively low risk. While scanners won’t provide the depth of security testing necessary for holistic security (scanners can’t perform multi-step attacks or offer the creativity that researchers can), they will give a “wide-but-shallow” measure of resistance against known vulnerabilities. Examples of players in this category include Tenable, Rapid7, WhiteHat, and Qualys.3

While scanners are ubiquitous and inexpensive, they have some fundamental limitations when employed as stand-alone solutions. For example, higher-value assets will almost always require some level of human interaction. Scanners also aren’t able to perform complex, multi-step exploits or zero days like humans can. For these reasons, although scanners are considered an essential element of a security test, they are not considered sufficient in themselves to get a realistic assessment of security risk.

Traditional pentesting (checklist-based)

What used to be a “pentest” has changed significantly over the years. The traditional penetration test was designed to provide a best effort, point-in-time, creative, and primarily manual test. More recently, however, the term pentest (especially in the private sector) has devolved into a lesser version of itself, which often entails performing tests solely against a checklist. Through most of the rest of this document, we will refer to the “traditional pentest” to mean the more current “downscoped” version. The majority of pentesting teams consist of one or two people.

The Big Four consulting firms (Deloitte, E&Y, PwC, KPMG) are good examples of this category. More specialised players include NCC Group, Bishop Fox, and Cipher. And finally, there are a host of smaller independent regional pentesting firms (also known as boutique consulting firms) that use this process.

The efficacy of this method depends on the depth of the assessment an organisation requires and the quality of the testers available to the provider. The advantages include simplicity and finite scope. Disadvantages include: no competition among testers, no incentive for creativity, a very limited skill set brought to bear on each vulnerability, no real-time insights into findings, and delayed remediation.

Bug bounty testing

Bug Bounty security testing harnesses a diverse set of testing skills, using bounties to incentivise ethical hackers to emulate the behaviour of the adversary. This allows them to evaluate the target’s overall security rather than simply test predefined security controls. In the process, it also allows them to fill some of the gaps where traditional pentesting falls short. There are several subcategories involved in this grouping (see next page for details). Some players in this space include Cobalt, Bugcrowd, and HackerOne. Many of the afore-mentioned companies are oriented more toward performing checklists for their broad customer base, and reserving the true crowdsourcing methodology for their large enterprise customers; but they are categorised here for simplicity.

The advantage of bug bounty security testing is that it creates attractive incentives for ethical hackers to find more vulns than the traditional pentest would. A wider range of researchers and skills (often 50+ researchers applied to a given test), and competition brings out overall better performance and increased depth of assessment. This category is more complex and offers varying levels of control. A good buying decision requires discernment from the buyer. (See the next page for more detail on the pros and cons.)

Crowdsourced security testing platform approach

Combining the essential elements of a security test

The most robust testing solution—the crowdsourced security testing platform—combines the creativity and ingenuity of crowdsourced vulnerability discovery, the methodology-driven approach of penetration testing, and the scalability and coverage of a high-end scanner. This enables organisations to conduct targeted penetration testing, find unknown vulnerabilities, and gather new intelligence in a scalable way. This intelligence then feeds into the machineled, human-augmented scanning system, teaching it what suspected vulnerabilities look like. The platform then conducts scalable, broad attack-surface coverage of the remaining assets and identifies sources of risk for the research team to investigate.

The crowdsourced security testing platform transforms all of these components into a continuous, always-on penetration testing process with well-orchestrated coordination between researcher, scanner, and compliance activities. It brings together a crowd of the top security researchers with a high-end, Artificial Intelligence / Machine Learning-enabled (AI/ML) scanner, and orchestrated workflows to engage the crowd for testing. Another way to say it is that all three above components are incorporated together and managed by a smart platform to get the best of each modality. To this day, Synack is the sole representative of this category, though many bug bounty players are claiming to be in this category.

Together, researchers and smart technology work in concert through an integrated platform, which coordinates their interactions; so they augment each other to provide both quality insights and continuous coverage. Because of the precision that comes from the app’s smart orchestration, instead of a cap being placed on the bounty, the provider assumes responsibility for the full cost of testing, and all important vulnerabilities are brought to your attention.