Gabriel Bernadett-Shapiro

Reversing the Reverser

We ran a series of experiments building an autonomous malware reverse engineer, testing frontier models across multiple samples and agent configurations. The models analyzed code well, but their investigations were wildly inconsistent. Small changes to context or tools pushed them toward entirely different parts of the same binary, and repeated runs under identical conditions stopped in different places.When we compared their reasoning traces against the IDBs they produced, we found the deeper issue: conclusions the models reached during analysis frequently never made it into names, types, comments, or relationships. A model would work out what a function did and then fail to record it, leaving the next analyst a database that couldn't say what had been established or what was still open. What determined performance was the design of the tools, and how much those tools told the model about its own investigation.

That led us to build a reverse-engineering Agent Standard, an open spec and reference implementation for connecting agents to a disassembler. The agent works through purpose-built tools instead of improvising IDAPython and tracking its own state. A host-side harness reads results back from IDA, reports exactly what changed or failed, and keeps unfinished work visible across turns. The model decides what the code means while the host handles the mechanics. In live exercises this let models recover from stale state and produce work that survived independent readback.

We're releasing the standard, the IDA harness, model-facing tools, work and evidence formats, a reusable model skill, conformance tests, and sanitized traces from real investigations.