Automata from Agent Traces: Failure and Next-Step Prediction

arXiv:2608.236702026-08-26

研究者把大量LLM智能体执行记录压缩成一个小型有限状态机,同时用来预测下一步动作和提前发现失败

基于大语言模型的智能体在完成任务时会留下冗长杂乱的执行记录,人工难以逐条审查安全性。作者把12个公开数据集的全部执行记录压缩进一个只有7到43个状态的紧凑有限状态机(FSM),并证明这个FSM同时能用于预测下一步动作和预测任务失败。该FSM在毫秒级时间内构建完成,对留出数据的回放拟合度达到0.997以上。

METAL LAB 解读图

从执行记录到一个FSM,再到两种预测

证据状态已报告实测结果

收集执行记录将12个公开智能体数据集中的工具调用、动作标签和消息转换为活动符号
合并为紧凑FSM把所有记录插入前缀树,再按最后进入的活动合并状态,得到7到43个状态的确定性FSM
下一步动作预测以当前FSM状态为上下文,在所有可对照真实标签的数据集上top-1准确率超过AWM
失败预测按状态提取访问频率、错误率等特征,留出测试集AUROC最高达0.94
运行时监控仅观察执行记录的32%即可提前叫停可能失败的运行,在SWE-agent上节省68%剩余计算量

这是 METAL LAB 制作的解读图，并非论文作者提供的原图。

他们做了什么

把智能体的每条消息(工具调用、动作标签或命令)映射成一个符号,将所有执行记录合并进一个前缀树,再按照进入状态的最后一个活动进行合并,得到一个无需任何学习超参数的确定性有限状态机。
在全部12个数据集上,构建出的FSM始终保持紧凑,状态数在7到43之间,对留出数据的回放拟合度不低于0.997,构建过程仅需毫秒级时间,复杂度随语料规模线性增长。
把FSM的当前状态作为预测下一步动作的上下文信息,在所有可与真实标签对照的数据集上都超过了此前的Agent Workflow Memory(AWM,一种只从成功记录中提取线性工作流的方法)。
在失败预测任务中,按状态提取的行为特征(访问频率、消息长度、错误率等)在留出测试集上的AUROC(0.5代表随机猜测,1.0代表完美区分)最高达到0.94;在线监控器仅看到执行记录的32%就能筛出可能失败的运行并提前终止。
研究观察到,智能体行为的结构模式更多是由部署系统(工具配置、运行环境)决定,而非所用的大语言模型本身:用四种不同大语言模型的记录合并训练出的同一个FSM,依然能几乎完美拟合每个模型各自的记录。

Figure 1: FSM evolution on SWE-agent. State count |Q| (red, left) and test fitness (blue, right) over training traces, with FSM snapshots at six milestones. The state space grows incrementally as new behavioral modes appear, while fitness saturates early (≥0.99 at 240 traces, 15% of training); construction completes in milliseconds.

Table 1: FSM extraction results on eight labeled real-trace datasets (excluding SWE-smith synthetic). |Q|: states. Fit: test replay fitness. †RPNI timeout at 120 s. Full baselines in Table 8; SWE-smith and the unlabeled datasets appear in Appendix D.1.
	Ours	RPNI	Alergia
Dataset	\|Q\|	Fit	\|Q\|	Fit	\|Q\|	Fit	Compr.
SWE-agent	25	0.999	59,510†	0.646	35	0.999	2,380×
WebArena	25	1.000	382	1.000	149	1.000	15×
AgentNet	25	1.000	62,495†	0.742	45	1.000	2,500×
tau2-bench (air)	18	1.000	6,506†	0.844	23	0.999	361×
tau2-bench (ret)	19	1.000	14,249†	0.837	25	1.000	750×
tau2-bench (tel)	43	1.000	63,897†	0.491	75	0.999	1,486×
ATBench	15	1.000	899†	0.984	15	1.000	60×
OSWorld	27	0.997	38,232†	0.706	31	0.999	1,416×

Figure 2: Next-step prediction cross-entropy (bits, ↓). (a) Per-dataset: FSM-conditioned methods (red) achieve 3–5× lower CE than baselines. (b) Average ranking: FSM-conditioned variants outperform their non-FSM counterparts on all four datasets. RPNI overfits worse than Uniform due to sparse transitions across thousands of states.

Table 2: Next-step prediction cross-entropy (bits, ↓). 5×5-fold CV across all datasets. Best per dataset in bold. The “FSM” columns use the FSM-state context format (ASG-minimal) selected on validation in Section 4.3.
Method	SWE-sm	SWE-ag	W&W	M2W	ATB	Avg
Uniform	3.170	4.585	3.000	2.807	3.807	3.474
Unigram	2.461	2.850	2.229	1.856	2.799	2.439
RPNI	3.851	4.284	2.855	3.549	2.448	3.397
Our FSM	0.638	1.071	0.963	0.756	1.243	0.934
FSM-PPM-AD	0.463	0.741	0.624	0.782	1.309	0.784
Ens(D0/3/5/7)	0.465	0.700	0.621	0.736	1.262	0.757
NGram-LR-K7	0.464	0.687	0.610	0.796	1.181	0.748
ESN-H64	0.476	0.691	0.580	0.746	1.184	0.735
FSM-LR-K7	0.460	0.686	0.582	0.755	1.162	0.729
FSM-ESN-H64	0.474	0.700	0.546	0.753	1.185	0.732

Figure 3: Failure prediction. (a) AUROC: FSM features (red) vs. raw trace statistics (blue) vs. fitness alone (gray). FSM features outperform raw features on SWE-agent (+7.9pp). (b) Early prediction: FSM features at 50% completion achieve 92% of final AUROC on SWE-agent. Solid: FSM; dashed: baseline.

Table 3: FSM context format ablation (tau2-bench retail, N=1,095, gpt-4.1-mini top-1 %).
Context format	Top-1 (%)
No memory (trace prefix only)	27.6
AWM (linear success workflows) (Wang et al., 2025d)	52.9
ASG-full (state + transitions + full graph)	52.2
ASG++ (ASG-full + multi-step continuations)	49.2
ASG-success (success-only ASG-full)	50.3
ASG (minimal: probabilities + top-15 continuations)	65.1

Figure 4: FSM-based runtime monitor. Cycle-rate over trace progress for one failing (red) vs. one successful (blue) SWE-agent run. The failing trace exceeds the cycle-rate threshold (0.778) at 32% of trace completion (vertical dashed line), triggering early termination. The successful trace stays below threshold and continues until natural completion.

Table 4: FSM vs. AWM as context for an LLM next-action predictor (gpt-4.1-mini, top-1 %). 6/8 gaps statsig at p<10−8. †tau2-retail used for FSM-context-format selection (§4.2); other 7 held out.
Dataset	N	AWM	FSM	Δ
WebArena	4,800	65.5	81.2	+15.7
SWE-smith	300	74.7	100.0	+25.3
SWE-agent	1,200	67.7	70.5	+2.8
tau2-bench (tel)	1,095	28.5	45.6	+17.1
tau2-bench (ret)†	1,095	52.9	65.1	+12.2
tau2-bench (air)	480	56.5	57.3	+0.8
ATBench	600	47.8	62.5	+14.7
OSWorld	1,286	55.0	70.7	+15.7

Figure 10: Per-dataset test fitness across all methods including Alergia. Our FSM achieves ≥0.997 fitness on all panels. Process mining baselines (Heur., Ind.) achieve competitive fitness but with low precision (Table 18).

Table 5: Evaluation datasets. |𝒜|: alphabet size. ∗7 primary + 17 rare. ‡4 LLMs. aTrivially separable.
Dataset	Domain	Traces	\|𝒜\|	Labels
Labeled (main results)
SWE-smith	Coding	500	9	✓
SWE-agent	Coding	2,000	24∗	✓
WebArena	Web nav.	8,337	24	✓
AgentNet	Desktop GUI	5,000	24	✓
tau2-bench (air)	Cust. svc.	800‡	17	✓
tau2-bench (ret)	Cust. svc.	1,824‡	18	✓
tau2-bench (tel)	Telecom	1,824‡	42	✓
ATBench	Safety	1,000	14	✓
OSWorld	Desktop OS	2,166	26	✓
Unlabeled (Appendix D.1)
Who_and_When	Multi-agent	184	8	✗
Mind2Web	Web nav.	500	7	✗
GUI-Odyssey	Mobile GUI	7,735	6	✓a

Figure 11: Runtime comparison. Left: FSM construction time (1–110 ms) vs. RPNI (7,000–36,000 ms) with speedup ratios annotated. Right: per-trace replay latency (0.003–0.015 ms), enabling real-time monitoring.

Table 6: Cross-model failure-prediction AUROC across three tau2-bench suites. Self-AUROC (diagonal) and mean cross-AUROC (off-diagonal); σ over the 4 self-AUROCs and 12 cross-pair AUROCs. Mean cross-AUROC across all 36 pairs is 0.786 vs. self mean 0.877 (0.091 gap). Self/cross fitness on airline 1.000/0.962, retail 1.000/0.972, telecom 1.000/0.990.
	Self-AUROC (diagonal)	Cross-AUROC (off-diagonal)
Suite	mean	σ	range	mean	σ	range	gap
τ2 airline	0.926	0.051	0.84–0.97	0.773	0.102	0.56–0.92	0.154
τ2 retail	0.749	0.052	0.68–0.82	0.681	0.075	0.54–0.79	0.068
τ2 telecom	0.956	0.028	0.92–0.99	0.905	0.045	0.82–0.99	0.051
Mean (3 suites)	0.877	–	–	0.786	–	–	0.091

Figure 13: Random and permuted trace rejection rates. Our FSM achieves near-100% rejection across most datasets, while RPNI shows poor permuted rejection on WebArena.

Table 7: Compression results on unlabeled datasets. Same methodology as Table 1.
	Ours	RPNI	Alergia
Dataset	\|Q\|	Fit	\|Q\|	Fit	\|Q\|	Fit	Compr.
Who_and_When	9	1.000	971	0.984	12	1.000	108×
Mind2Web	8	1.000	476	0.970	8	1.000	60×
GUI-Odyssey	7	1.000	21,255†	0.929	24	0.999	3,036×

Figure 14: FSM convergence: test fitness (right axis) and state count (left axis) as training traces are added. Fitness converges rapidly; the state space stabilizes later as rare patterns appear.

Table 12: k-Tails results. |Q|: states. Fit: test fitness. TO: timeout (300s). Our method requires no hyperparameter.
	Ours	k-Tails (k=1)	k-Tails (k=2)
Dataset	\|Q\|	Fit	\|Q\|	Fit	\|Q\|	Fit
SWE-smith	10	1.000	22	1.000	53	1.000
SWE-agent	25	0.999	74	0.996	332	0.993
WebArena	25	1.000	30	0.743	172	0.743
AgentNet	25	1.000	33	0.806	TO
tau2-bench (air)	18	1.000	37	0.973	731	0.773
tau2-bench (ret)	19	1.000	46	0.997	874	0.888
tau2-bench (tel)	43	1.000	210	0.930	TO
Who_and_When	9	1.000	14	0.541	20	0.541
Mind2Web	8	1.000	34	0.965	95	0.905
GUI-Odyssey	7	1.000	73	0.964	759	0.939

Figure 16: (a) State compression across labeled datasets: our FSM (10–43 states) vs. Alergia (10–149) and RPNI (382–63,897), with compression ratios annotated. (b) Cross-dataset fitness matrix: replaying traces from one dataset through another’s FSM. Diagonal entries (in-distribution) approach 1.0; off-diagonal entries (OOD) drop to near-zero for structurally distinct pairs (AUROC 1.000), except schema-sharing tau2-bench airline↔retail (near-1.0).

Table 14: Failure prediction AUROC (held-out). Main: full GBT pipeline. FSM-D ablation: training-free LR, Len+Ent vs. +per-state KL (Δ = gain). SWE-smith synthetic; tau2-bench aggregates 4 models.
		Main	FSM-D ablation
Dataset	\|𝒜\|	Holdout	Len+Ent	FSM-D	Δ
tau2-bench (tel)	42	0.941	0.725	0.857	+0.132
WebArena	24	0.903	0.718	0.773	+0.056
AgentNet	24	0.890	0.724	0.724	+0.000
ATBench	14	0.894	0.555	0.703	+0.147
tau2-bench (air)	17	0.864	0.625	0.777	+0.152
SWE-agent	24	0.799	0.690	0.689	−0.001
tau2-bench (ret)	18	0.779	0.596	0.660	+0.064
OSWorld	26	0.774	0.770	0.778	+0.008
SWE-smith	9	0.703	0.695	0.702	+0.006

Table 15: Failure prediction with Alergia FSMs under matched pipeline. Identical features, classifier, and CV protocol; FSM source varies. CV: 10×5-fold; Holdout: held-out test AUROC.
	CV AUROC	Holdout AUROC
Dataset	Ours	Alergia	Ours	Alergia
tau2-bench (tel)	0.923	0.748	0.915	0.752
WebArena	0.864	0.844	0.888	0.868
AgentNet	0.871	0.871	0.872	0.872
SWE-agent	0.790	0.709	0.805	0.714
tau2-bench (air)	0.792	0.664	0.826	0.764
tau-bench (ret)	0.789	0.606	0.666	0.498
tau-bench (air)	0.758	0.651	0.841	0.810
tau2-bench (ret)	0.713	0.576	0.743	0.575
SWE-smith	0.685	0.653	0.673	0.683

Table 17: Failure prediction: neural models on sequence vs. FSM features. CV AUROC (10×5-fold). Bold: FSM>Seq for same model.
	MLP	GRU	Transformer
Dataset	Seq	FSM	Seq	FSM	Seq	FSM
SWE-smith	0.663	0.699	0.683	0.700	0.649	0.665
SWE-agent	0.782	0.790	0.762	0.793	0.779	0.751
tau2-tel	0.962	0.969	0.958	0.970	0.964	0.967
tau2-air	0.816	0.839	0.793	0.815	0.819	0.845
tau2-ret	0.747	0.800	0.698	0.789	0.755	0.798
WebArena	0.848	0.861	0.818	0.835	0.860	0.868
AgentNet	0.878	0.920	0.877	0.917	0.876	0.917
FSM wins	7/7	7/7	6/7

Table 18: Precision analysis. Rand./Perm.: acceptance rate (lower is better).
	Ours	RPNI
Dataset	Rand.	Perm.	Rand.	Perm.
SWE-smith	0.0%	0.1%	0.0%	0.0%
SWE-agent	0.0%	0.05%	0.0%	1.6%
WebArena	0.0%	0.02%	0.2%	75.3%
AgentNet	0.0%	0.02%	0.2%	0.9%
tau2-bench (air)	0.0%	0.0%	0.0%	11.5%
tau2-bench (ret)	0.0%	0.0%	0.0%	3.1%
tau2-bench (tel)	0.0%	0.0%	0.0%	0.0%

Table 19: Adversarial trace rejection rate (%, ↑). Five mutations per trace.
Dataset	Subst.	Insert	Delete	Swap	Suffix
SWE-smith	81	100	98	100	100
SWE-agent	87	100	97	100	97
tau2-tel	77	96	61	93	100
tau2-air	78	90	77	93	97
tau2-ret	79	92	80	95	100
WebArena	100	96	84	100	68
AgentNet	100	100	97	100	99

Table 20: PM4Py miner results. Fit: replay fitness. Prec: precision from conformance checking. p/t: Petri net places/transitions.
	Alpha Miner	Heuristic Miner	Inductive Miner
Dataset	p/t	Fit	Prec	p/t	Fit	Prec	p/t	Fit	Prec
Who_and_When	3/8	0.37	0.19	10/23	1.00	0.31	16/24	1.00	0.27
SWE-smith	7/9	0.15	0.20	13/22	1.00	0.36	25/33	1.00	0.23
Mind2Web	2/7	0.63	0.29	15/29	0.96	0.45	23/30	1.00	0.32
SWE-agent	12/24	0.05	0.00	21/59	1.00	0.20	44/70	1.00	0.13
tau2-bench (air)	3/17	0.45	0.24	15/37	0.95	0.20	63/93	1.00	0.17
tau2-bench (ret)	3/18	0.46	0.23	12/32	0.95	0.21	36/58	1.00	0.14
tau2-bench (tel)	4/4	0.46	0.25	10/13	0.95	0.80	15/19	1.00	0.42
GUI-Odyssey	2/6	0.58	0.33	12/22	0.98	0.64	21/27	1.00	0.46
WebArena	23/24	0.24	0.11	35/71	0.98	0.55	17/42	1.00	0.10
AgentNet	174/24	0.30	0.00	35/84	0.99	0.45	18/39	1.00	0.13

Table 21: Convergence and generalization. Left: fraction at which fitness reaches 0.99. Right: train−test fitness gap at increasing fractions (all within ±0.003). Convergence behavior is similar across extraction levels (Appendix G.2).
	Convergence	Gen. gap (train−test)
Dataset	Train	0.99 at	Frac	\|Q\|	10%	30%	60%	100%
Who_and_When	147	8	5%	9	+0.002	+0.001	0	0
SWE-smith	400	40	10%	10	−0.002	−0.001	0	−0.001
Mind2Web	400	20	5%	8	−0.003	−0.001	0	0
SWE-agent	1,600	240	15%	25	−0.002	−0.001	−0.001	+0.001

Table 22: Conditional entropy (bits) by context order. The large drop from order 0→1 and convergence by order 2–3 shows strong sequential regularity.
Dataset	k=0	k=1	k=2	k=3	Drop 0→1
SWE-agent	2.16	1.06	0.80	0.79	51%
SWE-smith	1.98	0.63	0.55	0.55	68%
Mind2Web	1.69	0.74	0.79	0.68	56%

Table 23: Early prediction AUROC (holdout) by trace completion fraction. FSM vs. raw-statistic baseline.
Fraction	SWE-agent	SWE-smith
	FSM	Base	FSM	Base
10%	0.657	0.400	0.722	0.416
25%	0.656	0.456	0.687	0.693
50%	0.726	0.677	0.681	0.668
75%	0.726	0.672	0.688	0.685
100%	0.773	0.753	0.682	0.692

Table 24: Activity granularity robustness. |𝒜|: alphabet size. Fit: test fitness. AUROC: failure prediction (entropy-based).
Dataset	Level	\|𝒜\|	Fit	AUROC
SWE-smith	role-only	4	1.000	0.688
role-type	9	1.000	0.692
role-action	9	1.000	0.692
tool-only	9	1.000	0.692
SWE-agent	role-only	2	1.000	0.663
role-type	18	0.999	0.659
role-action	18	0.999	0.659
tool-only	18	0.999	0.659

Table 25: Best monitoring rules by F1 score. All rules use a single FSM-derived feature with a fixed threshold.
Dataset	Rule	Prec.	Recall	F1
SWE-agent	cycle-rate >0.885	0.956	0.451	0.613
SWE-smith	cycle-rate >0.878	0.267	0.750	0.393

Table 26: Failure prediction: FSM features vs. probabilistic baselines (holdout AUROC).
Dataset	FSM feat.	Trans. CE	Likelihood	N-gram
SWE-agent	0.813	0.452	0.626	0.317
SWE-smith	0.718	0.719	0.711	0.622

Table 27: ProbGuard (Wang et al., 2025b) vs. our FSM features for failure prediction. ProbGuard column = best AUROC over 20 configurations (Kunsafe× horizon, polarity-aware); FSM column = holdout AUROC from Table 14. Our FSM wins on every shared dataset by mean +17.6pp.
Dataset	ProbGuard (Wang et al., 2025b)	FSM features (ours)	Δ
τ2-bench (tel)	0.709	0.941	+0.232
τ2-bench (air)	0.723	0.864	+0.141
τ2-bench (ret)	0.566	0.779	+0.213
SWE-agent	0.683	0.799	+0.116
SWE-smith	0.525	0.703	+0.178
Mean (5 shared datasets)	0.641	0.817	+0.176

Table 28: Fitness under 10% perturbation intensity. Baseline fitness shown for reference.
Dataset	Subst.	Insert.	Swap	Trunc.	Base
SWE-smith	0.658	0.825	0.892	1.000	1.000
SWE-agent	0.818	0.885	0.938	0.999	0.999
Mind2Web	0.881	0.985	0.999	1.000	1.000
Who&When	0.828	0.865	1.000	1.000	1.000

Table 29: Statistical next-step prediction: top-1 accuracy (%, ↑) on full validation sets.
Dataset	Steps	Unigram	FSM	AWM	AWM cov.
SWE-smith	5,500	50.0	100.0	34.5	34.5%
WebArena	17,146	26.2	81.1	81.0	92.9%
SWE-agent	22,288	49.1	65.7	26.6	57.0%
tau2 (tel)	23,055	33.0	61.8	19.8	59.0%
tau2 (air)	4,020	30.8	69.2	55.9	91.3%
tau2 (ret)	10,207	27.4	72.6	63.5	91.4%
Mind2Web	886	80.2	80.2	0.0	0.0%

Table 30: Cross-dataset transfer: FSM vs. raw feature AUROC. Bold: FSM advantage > 3pp.
		Test dataset
Train	Features	SWE-sm	SWE-ag
SWE-smith	FSM	0.681	0.715
Raw	0.694	0.683
SWE-agent	FSM	0.648	0.780
Raw	0.682	0.720

研究结果

在全部12个公开数据集上,提取出的FSM始终保持7到43个状态的紧凑规模,对留出数据的回放拟合度不低于0.997,且构建时间仅为毫秒级。
在下一步动作预测中,以FSM状态为上下文的方法在所有可与真实标签对照的数据集上的top-1准确率都超过AWM(例如SWE-smith为100%对34.5%,tau2-telecom为61.8%对19.8%),受控实验显示FSM状态条件化平均带来0.155比特(21%)的交叉熵改善。
在失败预测中,按状态提取的特征在留出测试集上的AUROC最高达到0.94(tau2-bench telecom),在SWE-agent上比原始轨迹统计特征高出7.9个百分点,而仅用拟合度作特征的AUROC约为0.50。
仅使用执行记录前50%的内容,FSM特征就能达到完整记录时AUROC的92%;在SWE-agent上,监控器在执行完成32%时就能提前终止可能失败的运行,节省了68%的剩余计算量,精确率85.9%、召回率95.5%。
与自动机学习基线RPNI相比,FSM用少15到3036倍的状态数达到相当或更高的回放拟合度,构建速度也快得多(1到110毫秒对7000到36000毫秒)。

可应用场景

自动把智能体的执行历史压缩成小型结构,用于安全审计或定位导致故障的具体行为状态
在已部署的智能体旁运行轻量级实时监控器,在运行过程中发现失败迹象就提前终止
利用同时包含成功和失败记录的FSM上下文,作为只依赖成功记录的工作流记忆系统的替代或补充,辅助推荐下一步动作

局限与待验证事项

评测仅限于编程智能体、网页导航、客服对话等领域的12个公开数据集,活动符号表规模较小,为6到42个符号。
运行时监控器的规则(如周期率阈值)在状态种类丰富的数据集上表现良好,但在tau2-bench retail和SWE-smith等状态多样性较低的领域中区分能力不足,作者指出需要针对具体数据集调整规则。
作者明确指出,FSM恢复的只是观测到的表层活动顺序(directly-follows automaton),而非智能体背后真正的生成规则,后者从纯正例数据中原理上不可能被识别。
跨模型的失败预测特征迁移并不完美:模型自身平均AUROC为0.877,而跨模型平均AUROC为0.786,存在明显差距。
对于共享相同结构模式的领域(如tau2-bench的航空与零售套件),FSM难以清楚区分分布外与分布内的记录,这是一个例外情况。

为什么重要

如果冗长不透明的智能体执行日志能被压缩成一个小巧、可检视的结构,团队就无需逐条人工阅读即可对已部署的智能体进行安全审计、异常监控和提前叫停。由于该方法不依赖特定的大语言模型,它为多种不同的智能体系统提供了一个可复用的结构化基础工具。

本文术语

有限状态机(FSM) · 一种用一组可能情形(状态)及状态间转移规则来描述系统行为的数学模型
回放拟合度(replay fitness) · 把真实执行记录放入FSM中逐步回放时,FSM能够成功跟随的比例,越接近1.0说明FSM对真实行为的解释越准确
AUROC · 衡量模型区分两种结果(如成功与失败)能力的指标,0.5代表随机猜测,1.0代表完美区分
Agent Workflow Memory(AWM) · 一种此前的方法,只从成功的执行记录中提取线性工作流程,本文用作对比基线
交叉熵(cross-entropy) · 以比特为单位衡量模型预测的下一步动作概率与实际结果差距的指标,数值越低表示预测越准确

论文原文摘要（英文）

LLM-based agents execute multi-step tasks, but their behavioral structure remains opaque: long unstructured traces resist the safety auditing and runtime monitoring that deployment requires. Existing approaches operate per-trace or success-only, so they miss the cross-run topology that links next-step and failure prediction. To recover that shared structure, we collapse an entire trace corpus into a single, compact finite-state machine (FSM) that serves as a structural substrate for the otherwise unpredictable behavior of LLM agents. Across twelve public datasets, the FSMs are compact (7-43 states), replay held-out data at >=0.997 fitness with near-identical topology across splits, and build in milliseconds. This substrate addresses both prediction goals. For next-step prediction, FSM-state context outperforms Agent Workflow Memory on every ground-truth-matched dataset. For failure prediction, per-state behavioral features reach held-out AUROC up to 0.94, and an online monitor ranks failing runs above passing ones from a partial trace, triggering early stopping well before completion. Behavioral topology thus appears shaped more by the deployment harness than by the LLM, providing a model-agnostic structural primitive for safety auditing and runtime monitoring.

作者 · Seonglae Cho, Franklin Cardenoso Fernandez, Umar Mohammed, Zekun Wu, Kleyton Da Costa, Ilham Wicaksono, Adriano Koshiyam

在 arXiv 阅读

METAL LAB 最新报道

图片来源： Seonglae Cho et al., arXiv:2608.23670, CC BY 4.0