Content moderation at Atlassian’s scale means making millions of real-time decisions every message, prompt, and query that flows through Rovo Chat, Rovo Search, and more has to be checked before a response is served. That puts moderation squarely on the critical path of every user request. Latency isn’t a nice-to-have. It’s a product requirement.
For a while, we ran a large off-the-shelf model — gpt-oss-safeguard-20B — as our Acceptable Use Policy (AUP) filter. It worked, but it came with real costs: it was slow by design (thinking mode generates reasoning tokens at inference), and after months of prompt engineering, recall plateaued at 50.9% on our internal dataset. We’d hit the ceiling of what you can accomplish without touching the model itself.
This is the story of how we broke through that ceiling by fine-tuning a 4B model to score 87.0% precision and 75.5% recall running as a first pass filter, the content moderator system runs 53% faster and increase system throughput by 42%.
Why prompt engineering wasn’t enough
It’s tempting to think that a sufficiently capable model with a well-crafted prompt can handle any classification task. In practice, AUP filtering is a narrow but genuinely hard domain. Atlassian’s acceptable use policies have nuance borderline cases that depend on context, intent, and subtle phrasing. You can write elaborate system prompts that cover the common cases, but adversarial inputs (paraphrasing, jailbreaks, even just adding a ? to the end of a query) expose the gaps quickly.
The 20B model reasons before every answer. It generates reasoning tokens at inference, and that step can be dialed down but not switched off. That’s what lets it handle nuanced cases, but it adds latency and cost to every request. We needed a model that learned from that kind of reasoning during training, so it can make those calls directly without generating it each time.
Our approach: teach a small model to reason like an expert, then deploy without the training wheels
We collaborated with NVIDIA on Nemotron3 Nano 4B a compact dense model with strong instruction-following capabilities and set out to transfer the policy reasoning capability into the model’s weights through fine-tuning.
Data strategy
Data quality was the hardest part of this project. Production traffic AUP traffic is highly skewed, way less than 1% is AUP violation, and lack of adversarial examples on all different types of violations. So, we built synthetic data:
- Training: synthesized violation and benign examples + adversarial variations (typos, paraphrasing, jailbreaks)
- Eval: a 2k balanced set (50:50 positive/negative) plus a 3k adversarial set to specifically stress-test robustness against jailbreaking, paraphrasing, and punctuation tricks
The training recipe
We used a two-stage approach using NVIDIA Megatron Bridge:
Stage 1 — Full Supervised Fine Tuning (20k+ examples): We trained the model to memorize correct decisions on borderline cases. The dataset included <prompt, label, reasoning_tokens> triples — the model learned not just the label, but the reasoning chain behind it.
Stage 2 — DPO (2k+ examples): We used Direct Preference Optimization to compare correct reasoning trace from the wrong ones to refine the model’s judgment on the hardest cases the ones where the SFT model struggles.
Metrics on evaluation datasets
Below is a comparison on the model’s performance, the f1 metrics increased from 0.66 (base model) to 0.81 (fine-tuned).
| Model Metrics | GPT-OSS-safeguard-20B MoE | Nemotron 3 Nano 4B | Nemotron 3 Nano 4B fine-tuned |
|---|---|---|---|
| precision | 94.0% | 78.9% | 87.0% |
| recall | 50.9% | 56.7% | 75.5% |
| f1 | 0.66 | 0.66 | 0.81 |
| model size | 21B total parameters with 3.6B active parameters | 4B dense | 4B dense |
AI-accelerated modeling experimentation
What would have taken months of manual modeling iteration happened in about a week because we used AI agents to drive experimentation autonomously. They ran 300+ experiments, varying dataset composition, training hyperparameters and recipe stages, and used earlier rounds’ eval results to choose the next. Each training run took roughly 5 hours on a single NVIDIA H200, with about 20 runs in parallel. Without autonomous iteration we’d never have found the right recipe at that pace.
Results
In production, the fine-tuned 4B model resolved most of the requests, cutting overall latency and compute, while allowing the 20B model to think harder on suspicious cases. Below is a table shows the end-to-end content moderator is more accurate and run faster. It improves the user experience and makes Atlassian’s AI safer.
| Content moderator Metric | Before (GPT-OSS-safeguard-20B MoE baseline) | After (Nemotron 3 Nano 4B fine-tuned as first pass) |
|---|---|---|
| Precision | 94.0% | 98.1% |
| Recall | 50.9% | 56.6% |
| Latency | 740ms | 350ms (53% faster) |
| Throughput | baseline | +42% |
What we learned
- Post-training unlocks what prompt engineering can’t reach. For narrow, high-stakes domains like AUP filtering, fine-tuning is the lever that moves quality past the ceiling. Prompt engineering and fine-tuning aren’t competing strategies they’re complementary.
- Train with reasoning, serve without it. Building reasoning traces into the training data even if you strip them at inference meaningfully improves the model’s judgment on borderline cases. The model doesn’t need to reason out loud to reason well.
- AI agents dramatically improve productivity. 20 experiments run in parallel, with 300+ experiments completed by a very lean ML team. Autonomous experimentation is no longer a research curiosity it’s a practical force multiplier for applied ML teams.
What’s next
- Expanding the model strategy. AUP filtering is one use case. The same pattern narrow domain, high-quality bar, latency constraints, adversarial robustness applies across Atlassian’s AI stack. We’re exploring where this approach generalizes.
- Deeper NVIDIA collaboration. Our collaboration with NVIDIA on Nemotron models continues. The same recipe that worked for AUP filtering is being extended to other model families and use cases, including the embedding fine-tuning work powering Rovo’s semantic search.
In conclusion
Atlassian’s AI Lab and NVIDIA fine-tuned NVIDIA Nemotron 3 Nano 4B to enforce Atlassian’s Acceptable Use Policy. It now runs as a fast first-pass filter ahead of a larger 20B model. Prompt engineering alone had stalled, so the team trained the model in two stages with NVIDIA Megatron Bridge, and AI agents ran more than 300 of the experiments. Content moderation is now 53% faster, handles 42% more requests, and is more accurate, with precision rising from 94.0% to 98.1%.
