Our Approach to External Security Testing
Atlassian is regularly asked for penetration test reports by customers seeking assurance of the processes we have in place to identify (and fix) security vulnerabilities in Atlassian Products and Cloud. Our external security testing approach is built around the concept of 'continuous assurance' – rather than a point-in-time penetration test, we have an always-on, always-testing model using a crowd-sourced bug bounty.
Our philosophy and approach
Atlassian is well known for our values, and those values genuinely influence everything we do – including our approach to security testing. In practice, our values have led us to the following philosophies and approaches:
- Bugs are an unavoidable part of the development process - the question is not whether we have bugs, the question is how effectively and quickly we find them and address them. This doesn’t mean we like bugs or aren’t constantly innovating ways to reduce their frequency and severity, but when it comes to software bugs, denial is not an effective approach.
- We aim to increase the cost of finding and exploiting vulnerabilities in our products and services - through rapidly identifying and resolving issues we aim to increase the economic cost of finding security bugs. By increasing the cost of exploiting a vulnerability – by making it take longer, require more knowledge and more resources from the bad guys – we reduce their return on investment. If we can reduce it far enough, it becomes prohibitive or not attractive to them.
- We support and use industry standards - standardisation in our terminology and approach helps us ensure we’re covering everything, and helps customers understand how to understand what we do. For example, common ratings of vulnerabilities using the Common Vulnerability Scoring System (CVSS) ensures clarity for severity of a particular vulnerability between us and our customers. We also follow the vulnerability management processes outlined in ISO 27001 and the Cloud Security Alliance (CSA).
- External security researchers are a valuable extension of our team - if an Atlassian product has a vulnerability in it, it is in everyone’s interest – both ours, and our customers’ – to find it and fix it as quickly as possible. External security researchers who help us do this work are a valuable extension of our security team and should be rewarded appropriately. With the help of external security researchers, we are able to scale our team far beyond traditional approaches!
- We are open and transparent about our security testing program - our bug bounty program provides statistics about bugs found, we are open about the speed at which we try to fix security bugs, and we make available summary test reports, when possible.
Ongoing security assurance
We do use specialist security consulting firms to complete penetration tests on high risk products and infrastructure. This may be a new infrastructure set up for us (e.g. our Cloud environment), a new product (e.g. Trello) or a fundamental re-architecture (e.g. the extensive use of micro-services).
Our approach to penetration testing in these cases is highly targeted and focused. Such tests will generally be:
- White box - The testers will be provided with design documentation and briefings from our product engineers to support their testing
- Code assisted - The testers will have full access to the relevant code base to help diagnose any unexpected system behaviour during testing and to identify potential targets
- Threat based - Testing will focus on a particular threat scenario, such as assuming a compromised instance exists, and testing lateral movement from that starting point
We do not make these reports or extracts available for external consumption due to the extensive information made available to the testers in conducting these assessments.
The majority of these systems and products will subsequently be included in our public bug bounty program, providing the external assurance that our customers seek.
Our bug bounty program is hosted by Bugcrowd. The goal of this program is to ensure that our products are being constantly tested for security vulnerabilities. This is the centerpiece of our external security testing program and is the result of significant research, analysis and comparison between testing models.
We believe the crowd of independent security researchers participating in our bug bounty provides the most effective external security testing process because:
- A bug bounty is always on. Penetration tests are generally time-boxed to a few weeks. In a truly agile development environment with frequent releases, continuous testing is a necessity.
- A bug bounty has 60,000+ potential testers. Penetration tests generally have 1 or 2 individuals. Regardless of how good those 1 or 2 individuals are, they will never surpass the aggregate capability of the team of bug bounty participants.
- Bug bounty researchers develop specialised tooling and process vertically (specific bug types) and horizontally (specific bounties). This specialization provides the greatest chance of identifying obscure - but significant - vulnerabilities.
We continue to use penetration testing and specialist security consultants for internal support, but for our broad-coverage external program, a bug bounty is a better fit. We believe the combination of approaches maximises our chances of finding vulnerabilities.
More than 25 of our products or environments – ranging across our server products, mobile apps and Cloud products – are in-scope for our bug bounty program. Details of the number of vulnerabilities reported, our average response time, and average payout, are all included on the Bugcrowd site, with more than 800 testers having registered specifically for our program.
Vulnerabilities we seek to be identified through our bug bounty program include common types captured in the Open Web Application Security Project (OWASP) and Web Application Security Consortium (WASC) threat lists. They include:
- Cross Instance Data Leakage/Access
- Remote Code Execution (RCE)
- Server-Side Request Forgery (SSRF)
- Cross-site Scripting (XSS)
- Cross-site Request Forgery (CSRF)
- SQL Injection (SQLi)
- XML External Entity Attacks (XXE)
- Access Control Vulnerabilities (Insecure Direct Object Reference issues, etc)
- Path/Directory Traversal Issues
As part of our strive to be open and transparent, we invite anyone to visit our bug bounty program page, sign up for the program, and test us.
When a vulnerability is identified by one of our users during the course of their standard use of the product (as opposed to being identified through a specific attempt to test the system, which we require to occur via our bug bounty program), we welcome notification via our Atlassian Support Team. We respond promptly to any vulnerabilities submitted and will keep inquiring parties up to date as we investigate and respond to the issue.
Before disclosing an issue publicly we ask that researchers first request permission from us. Atlassian will process requests for public disclosure on a per report basis. Public disclosure requests will only be considered once the reported vulnerability is fixed.
Exclusions from our security testing program
Just as we are open and clear about the testing we do, we are also open and clear about the testing we don’t do ourselves, or don’t currently support. The following are excluded from our security testing program:
Marketplace Apps - Marketplace Apps are strictly excluded from our bug bounty program and we do not complete code review or security testing on these Apps. We will pass on any App vulnerabilities reported to us, however they will not be eligible for bounty.
Customer-initiated testing - In line with our Customer Agreement, we currently do not allow customer-initiated testing of our production environments. We are committed to being open and publish statistics from our bug bounty program to give customers comfort that we have an active security testing program. Our customers and their security advisers are of course welcome to sign up to our bug bounty program and complete testing through that process.
Certain low-risk vulnerability types - Our products are developed to unleash the potential in every team, and this requires collaboration. Vulnerabilities related to enumeration and information gathering are generally not considered significant risks.
Measuring the right things
We have a bug-fix policy which functions as an internal Service Level Agreement (SLA) between our product teams and our security team. When categorizing bugs, we use the Common Vulnerability Scoring System (CVSS version 3), which helps communicate the severity of vulnerabilities to our customers. We attempt to meet the following timeframes for fixing security issues:
- Critical severity bugs (CVSS v2 score >= 8, CVSS v3 score >= 9) should be fixed in product within 4 weeks of being reported.
- High severity bugs (CVSS v2 score >= 6, CVSS v3 score >= 7) should be fixed in product within 6 weeks of being reported.
- Medium severity bugs (CVSS v2 score >= 3, CVSS v3 score >= 4) should be fixed in product within 8 weeks of being reported.
These timeframes are re-evaluated on an annual basis and adjusted as needed based on the changing threat environment.
Further, given our goal is to increase the cost of finding and exploiting vulnerabilities in our products, having a way of quantifying that cost is needed. We use the 'bounty' offered security researchers for finding a vulnerability as a proxy. Simply put, over time, the number of bugs identified through our bug bounty program should decrease, and as that happens we will need to increase the amount we are willing to pay for such bugs if we want reports to continue coming in. If the amount of effort required to find a vulnerability with a $3000 reward eventually makes it not worthwhile (since it takes more than $3000 worth of effort) we will have increased the cost of finding that vulnerability.
In other words, you should expect to see our bounty payments increase over time.
Atlassian has a mature, open and transparent external security testing program built around our bug bounty.
Want to dig deeper?
We’ve referred to quite a few other documents and resources in this brief paper, and we encourage you to dig into them to understand more about our approach to security testing.
- Atlassian Trust Center
- Atlassian Security Practices
- Atlassian’s CSA STAR entry
- Atlassian’s Bug-fix Policy
- Atlassian’s Vulnerability Reporting Page
- Atlassian Bug Bounty Home Page
- Atlassian Security Incident Responsibilities