Code Reviewer already knew the rules. We gave it history.
Rovo Dev Code Reviewer catches PR issues related to code bug, code design, code readability, maintenance, testing, etc. It works well. But we saw an opportunity to take it further: to give it something closer to the institutional memory that experienced reviewers build over time, an awareness of past review discussions, team conventions, and the patterns that have shaped a codebase. This is the story of how we pursued that, what we learned, and what the data showed along the way.
The idea: past PRs are the best context data you have
The knowledge an experienced reviewer brings to a PR doesn’t come from a style guide. It comes from history: the last time this pattern came up, what was flagged then, what the team agreed to change. That knowledge lives in past pull requests that show what changed and the review conversations/comments that explain why it mattered.
So we built similar issue context via Atlassian Teamwork Graph Collection. When Code Reviewer processes a PR, it retrieves associated Jira issue and looks for semantically similar past issues from the same repository and provides two types of context to the model:
- PR diff context: the code changes from a similar past PR, showing what correct patterns look like in this codebase
- PR comment context: the review conversation from that PR, capturing what the team cared about and why
Diff context encodes technical correctness and comment context encodes team culture. Together they approximate institutional memory.
Experiment 1: Offline evaluation, establishing signal
We started with an offline evaluation on our internal benchmark of 3,492 issues. Of these, 2,620 were linked to valid Jira issues. It is important to note that the availability of similar issue PR context not only related to Jira issue linkage but also the ‘similarity’, where we set this to be 0.75. High selectivity of context causes the scarcity, which is an important property of the feature: similar issue context is high but sparse.
We ran three conditions, PR comment only, PR diff only, and both combined, measuring precision and recall against human-reviewed reference comments.
Similar Issue PR Diff (sample size: 190)
| Metric | Control (mean) | Treatment (mean) | Diff % (Treatment vs Control) | p-value |
|---|---|---|---|---|
| precision | 0.0551 | 0.0699 | +26.9% | 0.3692 |
| recall | 0.0798 | 0.0804 | +0.8% | 0.9753 |
| precisionPerfect | 0.0158 | 0.0316 | +100.0% | 0.2579 |
| recallPerfect | 0.0684 | 0.0737 | +7.7% | 0.7823 |
Similar Issue PR Comments (sample size: 204)
| Metric | Control (mean) | Treatment (mean) | Diff % (Treatment vs Control) | p-value |
|---|---|---|---|---|
| precision | 0.0708 | 0.0926 | +30.8% | 0.2902 |
| recall | 0.0770 | 0.0738 | -4.2% | 0.8550 |
| precisionPerfect | 0.0196 | 0.0490 | +150.0% | 0.0833 |
| recallPerfect | 0.0588 | 0.0686 | +16.7% | 0.5942 |
Similar Issue PR Comments + Diff (sample size: 220)
| Metric | Control (mean) | Treatment (mean) | Diff % (Treatment vs Control) | p-value |
|---|---|---|---|---|
| precision | 0.0592 | 0.0827 | +39.8% | 0.1749 |
| recall | 0.0731 | 0.0977 | +33.5% | 0.2125 |
| precisionPerfect | 0.0136 | 0.0455 | +233.3% | 0.0345 |
| recallPerfect | 0.0545 | 0.0909 | +66.7% | 0.0592 |
Two findings stood out. First, PR diff alone showed slight measurable impact, and the signal came from comments. Second, the combination of diff and comments produced the strongest results across all four metrics, with Precision Perfect reaching statistical significance (p=0.0345). This told us something important about how the two context types interact: comments tell the model what to look for; diffs tell it what correct looks like. Neither alone is sufficient.
A deliberate design decision: same-repo only
After the initial offline results, the natural question was whether we could extend coverage by pulling similar issues from other repositories (cross-repo context). More coverage would mean the feature applies to more PRs. We ran a dedicated experiment to find out.
Cross-repo context did increase coverage (+11.3%, from 220 to 245 issues). But the quality gains evaporated:
| Metric | Control | Cross-repo (diff + comment) | p-value |
|---|---|---|---|
| Precision | 0.056 | 0.080 | 0.13 (not significant) |
| Recall | 0.084 | 0.090 | 0.73 (not significant) |
| Precision Perfect | 0.020 | 0.032 | 0.40 (not significant) |
| Recall Perfect | 0.065 | 0.082 | 0.34 (not significant) |
The mean values were consistently elevated, but none reached significance. The reason became clear through case analysis: cross-repo diffs and comments reference entirely different file paths, class names, and framework. Rather than grounding the model in the current codebase’s conventions, they inject noise. In two case studies, cross-repo context either had no effect or actively degraded review quality by surfacing patterns irrelevant to the PR under review.
We decided to restrict to same-repo retrieval. The coverage trade-off is real, but the quality trade-off from cross-repo pollution is worse.
Experiment 2: Online internal, the first real-world signal
Moving from offline evaluation to live production revealed a new complexity: internal Atlassian repositories have a much more intricate issue graph than external ones, making Jira issue linking less reliable. To get a clean signal, we scoped the internal experiment to a single site, where linking was tractable.
This constrained the sample significantly. Across approximately 3,000 exposures, only 270 runs had a linked Jira issue (150 control, 120 test), and of those, 34 runs (out of 120 in test) actually received similar issue context in the prompt.
For those 34 runs, the resolution rate improved from 45.9% to 59.3%, a 13.4 percentage point lift. Resolution rate measures the fraction of generated comments that the PR author acted on with a code change; it excludes comments that were simply dismissed.
A secondary signal: the feedback rate (the fraction of runs where engineers chose to rate a comment at all) doubled from 4% to 8.8% when context was present. While the absolute feedback volume was too low to draw conclusions about quality direction, the engagement increase suggests the comments were resonating enough to prompt a response.
Experiment 3: Online external, validation at scale
We then ran a larger controlled experiment across external customer PRs (all of whom had opted in to Code Reviewer). The experiment ran from March 5–12, 2026 on claude-sonnet-4-5, covering 15,616 total runs across four groups.
| Group | Runs | Comments/run | Resolution rate |
|---|---|---|---|
| Control | 7,777 | 2.28 | 31.92% |
| Test [diff + PR comments] | 875 | 2.11 | 41.90% |
When both context types were present, resolution rate rose from 31.92% to 41.90%, a 10 percentage point absolute improvement. Diff context alone contributed about half that lift (to 36.86%); PR comment context added the remainder. The stacking effect confirmed what the offline experiment suggested: the two context types are complementary, not redundant.
One notable finding on the efficiency side: despite consuming 24% more input tokens (703K to 871K per run), the model made fewer LLM requests (27.6 to 26.7) and fewer function calls (40.4 to 37.2). Better context appears to make the model more decisive. It reaches conclusions faster rather than probing the codebase through additional tool calls. This is a meaningful benefit for customers who care about latency and cost.
What the context actually does: two case studies
Case study 1: Getting a type signature exactly right
Code Reviewer was reviewing a change to embedded sensor firmware. The similar past issue was a recently-merged PR from an adjacent sensor module. From that PR’s diff, the model generated a comment that the function signature was using the wrong const qualifier, specifically, const SensorIRTypes_Data_t* const instead of SensorIRTypes_Data_t* const.
This distinction is invisible without the historical diff. The model had no basis for knowing which signature was canonical from the current code alone. The similar PR had made exactly this change, making the correct form unambiguous. Two additional comments, spacing inconsistencies of the form isInWarmup = FALSE, were also sourced from the diff. All three comments were resolved by the author.
Case study 2: Framework patterns and team culture (100% resolution rate)
A test automation PR using Cucumber/Gherkin and Java. The similar issue was retrieved and five comments were generated; all five resolved by the author.
The context split here is instructive. Three comments came from diff context: the team’s @cleanupCloneScenario tag pattern, a JSON serialization approach for security, and a duplicate Scenario: keyword in Gherkin. Two came directly from PR comment context: one flagging that HTTP responses were never validated before storing, another identifying an unused class that a prior reviewer had called out as dead code.
The PR comment-sourced comments are the more interesting ones. They encode team standards that would never appear in a diff, and they exist only in the review conversation. This is the value that comment context adds over diff context alone.
What’s next: Extend our context to fully embrace Atlassian Context
Similar issue context is a first step, but it’s only one slice of what an AI agent reviewer could know. Past PRs tell the model about code patterns and team norms. What they don’t capture is the broader context that surrounds every change: the Jira issue that motivated the work, the Confluence design doc that shaped the approach, the acceptance criteria the team agreed on.
Atlassian is uniquely positioned here. Because Rovo Dev Code Reviewer lives inside the Atlassian ecosystem, it has access to a richer web of context than any standalone code review tool. A PR doesn’t exist in isolation and it’s linked to an issue, which connects to an epic, a design spec, a customer escalation, a product requirement. All of that is signal.
The diagram above shows how this fits together. At the centre is the pull request being reviewed. Radiating out from it are the layers of context we can draw on: similar past issues (what we’ve built and shipped today), the Jira issue that scoped the work, related Confluence pages that documented the design, and other connected artifacts across the Atlassian graph. Each layer adds a different kind of signal — technical, organizational, and strategic.
We are exploring how to progressively expand that aperture. Similar issues were the natural starting point because they’re the closest analogue to what an experienced human reviewer already does. But the opportunity to leverage the full Atlassian Teamwork Graph and to move code reviewer toward a more holistic view of every change is one of the most exciting directions ahead of us.
