Predicting Agent Failures with Finite Automata
Listen to the summary
Uses a voice available on your device
Audio options
On this page 5 sections
Related concepts 3 concepts
Key Takeaways
- Integrating FSM state information improves next step prediction cross entropy by 0.155 bits, representing a 21 percent gain over models without state context.
- Per state behavioral features allow for failure prediction with an AUROC up to 0.94.
- A single FSM can achieve perfect fitness across 4 different chat models when tested on the tau2 bench suite.
- The method enables an online monitor to effectively rank failing runs higher than passing runs using only partial trace information.
Summary & Methodology Analysis
The methodology transforms raw agent activity logs into structured Finite State Automata (FSM) to provide a transparent behavioral backbone. By inserting traces into a prefix tree and merging states via last activity right congruence, the system creates a compact representation of agent logic. The process then refines these transitions by removing infrequent paths, providing a reliable structure to condition future step predictions. This state conditioning specifically targets the transition probabilities, yielding a 0.155 bit (21 percent) improvement in cross entropy, a measure of prediction uncertainty, compared to methods lacking state awareness. For failure detection, the researchers extract behavioral features such as frequency and error rates from these states, achieving a held out AUROC of up to 0.94.
Interactive System Flowchart
Illustrative Implementation
A short sketch of the paper's core idea, not the authors' own code.
# Illustrative sketch (not from the paper)
import torch
from collections import defaultdict, Counter
# 1. Insert activity traces into a prefix tree (trie)
def build_trie(traces):
root = {}
for trace in traces:
node = root
for act in trace:
node = node.setdefault(act, {'_cnt': 0, '_next': {}})
node['_cnt'] += 1
return root
# 2. Merge states reached by the same incoming activity (last‑activity right congruence)
def merge_states(trie):
# Simple bottom‑up merge based on identical outgoing dictionaries
def recurse(node):
for act, child in list(node['_next'].items()):
recurse(child)
# merge identical children
sig_map = defaultdict(list)
for act, child in node['_next'].items():
sig = tuple(sorted(child['_next'].keys()))
sig_map[sig].append(act)
for acts in sig_map.values():
if len(acts) > 1:
master = node['_next'][acts[0]]
for dup in acts[1:]:
node['_next'][dup] = master
recurse(trie)
return trie
# 3. Remove transitions observed only once unless they are the sole continuation
def prune_transitions(trie):
def recurse(node):
for act, child in list(node['_next'].items()):
if child['_cnt'] == 1 and len(node['_next']) > 1:
del node['_next'][act]
else:
recurse(child)
recurse(trie)
return trie
# 4. Estimate P(a|q) using observed counts (FSM‑state conditioning)
def compute_transition_probs(trie):
probs = {}
def recurse(state, path):
total = sum(child['_cnt'] for child in state['_next'].values())
if total > 0:
probs[tuple(path)] = {act: torch.tensor(child['_cnt'] / total) for act, child in state['_next'].items()}
for act, child in state['_next'].items():
recurse(child, path + [act])
recurse(trie, [])
return probs
# 5. Extract per‑state behavioral features (frequency, avg length, error rate, drift placeholder)
def extract_features(trie):
feats = {}
def recurse(state, path):
freq = state['_cnt']
# placeholders for length, error, drift (computed elsewhere)
feats[tuple(path)] = {'visit_freq': freq, 'msg_len': None, 'error_rate': None, 'temporal_drift': None}
for act, child in state['_next'].items():
recurse(child, path + [act])
recurse(trie, [])
return feats
# Example usage with dummy traces
traces = [
['init', 'search', 'compile'],
['init', 'search', 'test'],
['init', 'debug', 'compile']
]
trie = build_trie(traces)
trie = merge_states(trie)
trie = prune_transitions(trie)
trans_probs = compute_transition_probs(trie)
state_feats = extract_features(trie)
// Illustrative sketch (not from the paper)
const torch = require('torch-js'); // placeholder for tensor ops
// 1. Build a prefix tree (trie) from activity traces
function buildTrie(traces) {
const root = {};
for (const trace of traces) {
let node = root;
for (const act of trace) {
if (!node[act]) node[act] = { _cnt: 0, _next: {} };
node = node[act];
node._cnt += 1;
}
}
return root;
}
// 2. Merge states with identical outgoing activities (last‑activity right congruence)
function mergeStates(trie) {
function recurse(node) {
for (const act in node._next) {
recurse(node._next[act]);
}
const sigMap = {};
for (const act in node._next) {
const child = node._next[act];
const sig = Object.keys(child._next).sort().join(',');
if (!sigMap[sig]) sigMap[sig] = [];
sigMap[sig].push(act);
}
for (const acts of Object.values(sigMap)) {
if (acts.length > 1) {
const master = node._next[acts[0]];
for (let i = 1; i < acts.length; i++) {
node._next[acts[i]] = master;
}
}
}
}
recurse(trie);
return trie;
}
// 3. Prune transitions seen only once unless they are the sole continuation
function pruneTransitions(trie) {
function recurse(node) {
for (const act in { ...node._next }) {
const child = node._next[act];
if (child._cnt === 1 && Object.keys(node._next).length > 1) {
delete node._next[act];
} else {
recurse(child);
}
}
}
recurse(trie);
return trie;
}
// 4. Compute P(a|q) from counts (FSM‑state conditioning)
function computeTransitionProbs(trie) {
const probs = {};
function recurse(state, path) {
const total = Object.values(state._next).reduce((s, c) => s + c._cnt, 0);
if (total > 0) {
probs[path.join('>')] = {};
for (const act in state._next) {
const child = state._next[act];
probs[path.join('>')][act] = torch.tensor(child._cnt / total);
}
}
for (const act in state._next) {
recurse(state._next[act], path.concat(act));
}
}
recurse(trie, []);
return probs;
}
// 5. Extract per‑state behavioral features (placeholders for length, error, drift)
function extractFeatures(trie) {
const feats = {};
function recurse(state, path) {
feats[path.join('>')] = {
visitFreq: state._cnt,
msgLen: null,
errorRate: null,
temporalDrift: null
};
for (const act in state._next) {
recurse(state._next[act], path.concat(act));
}
}
recurse(trie, []);
return feats;
}
// Example usage with dummy traces
const traces = [
['init', 'search', 'compile'],
['init', 'search', 'test'],
['init', 'debug', 'compile']
];
let trie = buildTrie(traces);
trie = mergeStates(trie);
trie = pruneTransitions(trie);
const transProbs = computeTransitionProbs(trie);
const stateFeats = extractFeatures(trie);
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary goal of this research?
The research aims to provide transparent behavioral structure to LLM based agents to assist with safety auditing and failure prediction during multi step tasks.
Q2. Can this method be used for real time monitoring?
Yes, the method enables an online monitor to rank failing runs above passing ones from a partial trace.
Q3. What kind of agents does this research apply to?
The research focuses on LLM based agents, specifically demonstrating results on the SWE agent framework.
Q4. How accurate is the failure prediction model?
The per state behavioral features reach a held out AUROC of up to 0.94.
Q5. Does the FSM perfectly capture the generating language of the agent?
No, the FSM accepts the directly follows closure of the observed traces rather than the true generating language, which means adversarial traces that preserve activity bigram statistics could potentially replay.
Q6. How does this approach improve next step prediction?
By using FSM state conditioning to estimate transition probabilities, the model improves cross entropy by 0.155 bits, or 21 percent, over methods without state.
Q7. What benchmarks were used to test the models?
The researchers utilized SWE agent, tau2 bench, and the trajectory level safety benchmark ATBench.
Q8. What is the status of cross architecture transferability?
The researchers measured cross model transfer with a mean cross AUROC of 0.786 on three tau2 bench suites, though they note that broader cross architecture and cross domain transfer remains future work.
Q9. How robust is the FSM fitness across different models?
Across 4 chat models tested on the tau2 bench, a single FSM was able to achieve perfect fitness.