Brad Palm
Auto-Poking The Bear – Analytical Tradecraft In The AI Age
Analytical tradecraft and shared standards have transformed Cyber Threat Intelligence from a niche discipline into a collaborative industry-wide research endeavor. Researchers and analysts now routinely build on each other’s work, creating a foundation of trust and shared methodology.
AI is disrupting this ecosystem, as we increasingly delegate data preparation, analysis, and entire workflows to AI assistants. Doing so will make us more productive, but not without cost. While you may trust your own AI-assisted analysis, can you trust another researcher’s prompts/agent process? As questions about reliability and transparency persist, we will need to adapt our research methodology and develop a new joint understanding of the promises, pitfalls, and probabilities inherent in AI-assisted work.
We tackle these challenges through a concrete case study. We present our own LLM-based agentic system, developed to analyze Russian internet data leaked by Ukrainian cyber activists. We’ll walk through the system’s architecture and demonstrate its performance across tasks ranging from simple data collation to sophisticated analytical workflows to track adversaries.
Along the way, we outline how to understand the promises and limitations of this technology and more importantly, how to communicate them transparently to other researchers and audiences – so that we maintain transparency and accountability for published products.
Brad is the Chief Operating Officer at Dreadnode. Previously, he was a VP of Services and Technology for Pathfynder and the Managing Director of Software at Ascent, where he focused on SOC automation and the integration of CTI in the delivery of managed services. Brad served in the Marine Corps for over a decade, starting as a Combat Engineer Officer leading engineering teams in support of combat operations. He then studied Computer Science at the Naval Postgraduate school, specializing in networking and situational awareness analytics, and finished his service as the lead of a network analysis team that focused on testing network attached weapons and communication systems. Brad especially enjoys staying hands-on as a threat hunting analyst, as well as trying to keep up with all the awesome research at Dreadnode.
