Last Friday we took part in an SC Magazine webinar that examines the differences between penetration testing and bug bounties. Jason Haddix, former HP Fortify Pen Test Lead and now Head of Trust and Security at Bugcrowd, spoke with Wade Billings, VP of Technology Services at Instructure, the company behind learning management system Canvas.
During the webcast, you’ll hear about the key differences between the two security assessment models, and learn more about why Canvas has swapped their last three penetration tests for bug bounties and the results they’ve seen. This post will highlight the main topics discussed during the webinar.
First things first, definitions…
The definition of “penetration testing” encompasses a wide range of activities. For the purpose of this discussion, we’re discussing vulnerability assessment activities in which external consultants are hired for time-boxed and fixed-cost engagements to find vulnerabilities. We’re not talking about red team engagements or other goal-based assessments to exploit potential vulnerabilities and achieve a goal (i.e. taking over the network).
Bug bounties are similar to penetration tests in that they are used by organizations to engage with external testers to find vulnerabilities, but have three key advantages: testers, coverage and results model.
#1 – Testers: Many vs. Few
A typical penetration test entails one or two testers. Bug bounties increase that number, bringing in massive diversity in perspective and expertise.
The Bugcrowd community is made up of tens of thousands of researchers, and for organizations looking to focus testing or restrict testing access, Bugcrowd offers private programs that enable crowd vetting based on skills, activity or even geography.
The diversity and breadth of the crowd contribute to the uncovering of higher quality or more obscure vulnerabilities (discussed further below). In fact, many of these researchers are full-time security professionals, participating in bug bounty programs on the side.
To learn more about common questions such as “how do you trust these researchers” and “how do you get the necessary coverage,” watch the webinar itself.
#2 – Coverage: Ongoing vs. Point-in-Time
Penetration testing engagements offer a point-in-time look at your applications and can never provide continuous coverage. Thus, the value and credibility of penetration tests go down steadily over time as code–and security vulnerabilities–gets released. By allowing for constant testing, bug bounty programs provide continuous coverage.
More than that, bug bounties come in a variety of flavors; public and private, continuous or on-demand.
Jason and Wade discuss the different types of programs, the model Canvas used, and how you can ensure continuous coverage in the webinar.
#3 – Model & Results:
Perhaps the most unique and compelling aspect of the bounty model is the incentivization Instead of paying a flat fee for time, organizations running bug bounty programs pay per bug. Only unique, valid and in scope vulnerabilities receive rewards, and the higher the severity, the higher the payout. This increases competition among researchers and encourages researchers to find and submit high-quality bugs.
Canvas Case Study:
How do bug bounties actually pan out?
Throughout this talk, Wade walks us through the process of scoping, implementing, learning from and iterating upon their bounty program. In 2014 they launched their first On-Demand bounty program which they swapped in for their annual security audit. Since then, Canvas has engaged with the crowd on a time-boxed and continuous basis.
The results speak for themselves.
Want to learn more about these differences? Watch the webinar and download our newest guide “Penetration Testing vs. Bug Bounties.”
This is an ongoing discussion, and we welcome your input at hello@bugcrowd.com. Stay tuned over the following weeks as we answer many questions you might have about the differences, and chat with industry players on the subject. Subscribe to our blog at right to receive blog content notifications.
