Project Artemis: How We Hunt Security Bugs in Our Own Code

Dijkstra once joked with the following statement:

Every feature a team ships puts a few bugs in with it, and the only question that really matters is who finds them first.

We have run bug bounty programs for several years. Across our HackerOne bounty and vulnerability disclosure programs, submissions rose from 28 in Q2 2025 to 75 in Q2 2026, roughly two and a half times as many. Most of what the program surfaces follows the classic shape of web security: an outside party sends controlled data into an internet-facing service, that data flows to other internal services, and somewhere along the way an assumption that no longer holds turns into a vulnerability. Behind each vulnerability in our services is the same concern: protecting the customer data moving through our pipeline and the trust our customers place in us. Protecting that is why our bug bounty program exists (alongside regular security reviews with trusted external partners), and it is what pushed us to find these bugs ourselves, before anyone outside our organization does.

The logistics of running a bug bounty program, from the technical side, are rarely discussed from the defender’s perspective in the age of AI. Of those 75 reports submitted in Q2 2026, 31 had been marked as duplicates as of November 2026. A year earlier, 3 of 28 reports were duplicates. We also see AI-hallucinated write-ups, and only some reports are valid. We pay a bounty when a report is real and absorb the triage when it isn’t. But the triage is only the visible cost. The one that matters is exposure: until a report lands, a real bug sits in a live service with customer data moving through it, and the clock is running for whoever finds it first. The question that eventually gets asked is whether we could do the finding ourselves, before any of it reaches the queue.

We already maintain a map of our perimeter, our internet-facing attack surface seen from an attacker’s point of view. We wanted to turn that inventory into a source-level security investigation using a carefully prepared multi-agent pipeline: work out what an endpoint does, follow data across the services it touches, and challenge the assumptions along the way. Instead of stopping at a plausible bug, it works toward a finding backed by evidence, with reproduction where feasible and enough context for an engineer to act. That’s how Project Artemis was born.

What is Artemis?

Artemis is our AI-assisted red team. It is made up of specialized agents that self-organize as they investigate potential security weaknesses in our services.

A Scout gathers leads, an Analyst examines them and prepares test briefs, and an Operator tests selected hypotheses in a constrained environment. An independent Adjudicator reviews the evidence, reproduces proofs where available, and flags decisions that need a human.

For the first campaign, two security engineers ran Artemis on their laptops using Hermes and Claude Code. The campaign produced 112 findings. We have since replaced Claude Code with OpenCode because its base prompt uses fewer tokens and it is more customizable. Hermes remains part of the setup.

Artemis gives each agent a specific job, a curated set of tools and skills, and a shared set of Rules of Engagement. Controls are tailored to the role, with isolation and restricted egress for the execution lane. The aim is to give agents enough capability to investigate a hypothesis without treating that capability as permission to test anything they can reach.

Attack surface as a data problem

The project started with a scoped inventory of internet-facing endpoints across multiple services, exported as JSON that agents could read. Many routes included a pointer to the handler; for the rest, the pipeline had to locate the implementation. Those endpoints became the starting points for tracing attacker-controlled data through the system. The inventory included our own services and self-hosted third-party components; both count as attack surface. We then prioritized routes by exposure and potential impact, giving each campaign a manageable slice rather than trying to investigate everything at once.

Precision over false positives

Building the pipeline was the single most important step. Mechanically it’s a chain of stages, each one its own process that reads the previous stage’s output and writes the next. What it buys us is that intermediate results sit on disk where we can read it, and if an agent dies it can continue from where it left off. If a stage turns out to be fruitless, or an agent runs out of tokens or compacts too many times, we can rewrite that stage without unpicking the rest. That simplicity saved us time, because the need to replay a stage came up more than once.

One of those agents is the verifier, which is built to disbelieve. A candidate bug does not advance just because it sounds plausible. Other agents challenge it from different angles: is the code actually wrong, can an attacker reach it, and does the behavior create meaningful impact? Surviving those challenges is a qualification step, not proof by itself. We also run independent investigations in parallel to reduce elapsed time, while keeping their evidence separate. The goal is straightforward: fewer speculative reports and stronger findings for product teams to act on.

From nothing to something, Artemis’ first campaign

After reaching that point, we let the pipeline run for about a week. The results started coming in early, and we ended up at a striking number: 112 findings, of which 93 were valid and the rest were set aside as duplicates or invalid. For a single campaign, read entirely from source and confirmed against a local copy of the stack, that was well beyond what we expected, and it was only the first. Artemis fully exploited 80 of the 93 valid findings on its own. The remaining 13 were either impossible to exploit outright or needed a human in the loop.

Twelve entries were duplicates of already-tracked findings, and seven were refuted. Each report included more than an automated alert. It explained the mechanism, provided supporting evidence, and stated the limits of what had been demonstrated. This was a scoped campaign, not a benchmark against a human red team or a claim of complete coverage. It gave us a concrete reason to keep investing in the approach.

Looking at severity, the results are even more striking: graded on CVSS, Artemis found 4 Critical, 23 High, 25 Medium and 15 Low. And the vulnerabilities clustered exactly where we had predicted they would, authorization first, then command injection and IDORs.

Earning a place in the queue

Surviving the verifier isn’t the same as being worth an engineer’s afternoon. Two more things have to happen first:

1. Check whether we already know about it. Candidates are compared, read-only, with existing issues and prior reports to distinguish new findings from duplicates, related weaknesses, and regressions. A duplicate should not create another ticket or consume another round of reproduction work. A related finding or regression, however, may still deserve investigation. That distinction keeps the queue useful without throwing away new evidence.
2. Check whether the finding applies to the deployed system. We re-derive code locations from symbols, review the full middleware chain, and check the relevant deployment and configuration evidence. A weakness in source is not automatically reachable in a running service, and a local reproduction is not automatically proof of production impact. Each report should make those distinctions explicit, including any preconditions or uncertainty that remain.

The ticket is written so an engineer can start with the finding, not repeat the investigation. It carries the dataflow trace, the checks performed, the reproduction evidence where available, and a plain account of what was, and what was not, demonstrated. The root cause gives the fix a clear target, while links to related issues provide context. The goal is to shorten the distance between a security finding and an engineering change.

From reading to running

The agents do not investigate every class of bug equally. We prioritize security boundaries that matter to a multi-tenant platform: tenant isolation, authentication, authorization, server-side request forgery, injection, and unintended disclosure of secrets. Authorization receives particular attention because keeping one tenant’s actions and data separate from another’s is a fundamental requirement of our product.

Artemis separates reading code from testing a potential vulnerability. First, agents analyze the source and form a hypothesis. For selected hypotheses, they then test whether the behavior can be reproduced in an authorized environment. This second step is called dynamic testing. It requires the right scope, fixtures, and controls.

One of the pieces we are most proud of is a reproducible local environment spanning major parts of the RudderStack platform. Defined as code, it gives us a controlled place to exercise vulnerable behavior with test data and capture reproduction evidence. We can test a finding by sending the input to the local copy, observing whether the bug triggers, and capturing the proof. This avoids touching a live system or another tenant’s data. It turns dynamic confirmation from a risk we kept putting off into something safe to do on every serious finding, and it is fast becoming the bridge between what Artemis reads and what we can prove.

Models and Cost

The question that hangs over any initiative like this is cost, both the money and the time, and the usual assumption is that it has to be enormous. It was the thing we were most braced for, and it turned out to be the biggest surprise of the whole campaign.

For the first campaign, each engineer used an Anthropic Claude account accredited under the Cyber Verification Program (CVP). Between them, the campaign consumed at most a single week of usage on two Claude Max subscriptions. Priced out, that is a $200-a-month plan pro-rated across four weeks, for two people, for one week: ($200 / 4) × 2, or about $100. For the first campaign, we used Claude Opus 5 at different effort levels across auditing, review, validation, proof-of-concept work, and reporting. We later used Opus 4.8 when requests to Anthropic degraded after the Opus 5.5 release.

Artemis has since changed. It now uses Sol 6.1 for its tasks, except proof-of-concept work, which uses DeepSeek V4.1 Flash.

A hundred dollars and a week of mostly unattended compute, for 112 findings. The total cost was significantly less than we expected, and it pales in comparison to the value the Artemis initiative is driving.

The limits of an automated hunter

If one decision made this work, it was making the verifier hostile. An agent that’s happy to agree a bug is real doesn’t save you the triage. It hides it, moving the judgment call from a human staring at a queue to a model you now have to second-guess. The only reason the output is worth a security engineer’s time, instead of being one more firehose, is that every finding in it has survived agents built to destroy it.

The encouraging part and the discouraging part are the same fact. Reading code across service boundaries, the way the authorization findings did, turns up things that signature-based tools slide past. It is also loudest exactly where it’s most useful. Anything touching auth generates the most noise, and in the draft run a human reviewer found real bugs the agents had read and waved through. So we treat it as something that generates leads, not something that signs off on a service.

What bit us most often wasn’t false positives, it was stale code. The surface export pins every repo to a commit, and some of those commits were well behind what’s actually deployed, and in the AI era, with CI/CD, code moves faster than it ever used to. A finding pointing at a line that has since moved looks exactly like a hallucination, because the code it describes isn’t there anymore, when really the bug is fine and the pointer is old. Re-deriving anchors from symbols instead of line numbers fixed more wrong answers than any prompt change we made.

We also had to stop trusting our own scoreboard. For a while we graded the pipeline on whether it could re-find bugs we’d pulled from history, and it found all of them, which felt good and proved nothing; a test you can’t fail measures nothing. The number that turned out to matter was the opposite one. Run the pipeline against the commit that fixed each known bug, where the right behavior is silence, and count how often it raises its hand anyway. That’s the false-positive rate, and it’s what we watch now.

Our first approach to dynamic testing did not work as planned. We had intended to confirm the worst findings against a live staging environment, but testing a shared environment carried risks. In an early attempt, nearly all findings failed for environmental reasons. Most targeted code that existed only in open pull requests and had not been deployed to staging. And when we finally audited our own prompts against the real data feed, we found agents still asking for fields the feed didn’t carry, and not one of them reading the signal that said the code had drifted. The models were doing fine. We’d wired them up wrong.

So far, the honest read is that Artemis finds real exploitable bugs, sometimes ones our other tools missed.

What’s next

Artemis is already sandboxed and can connect to an authorized environment, a copy of production, in order to perform red team engagements. We want to expand coverage so more candidates can move from source-level reasoning to automated reproduction. We also want clearer evidence handoffs to the teams that own the code, along with automated re-testing of findings once fixes land in a new release.

Longer term, we want Artemis to investigate new changes soon after they ship, instead of waiting for the next campaign. We also plan to rerun hunts when new models bring security-related improvements.

We have a baseline of true positives and false positives pinned to specific source-code commits, which lets us quickly benchmark new models before a campaign.