ReguSim: Evaluating LLM Agent Rule Grounding in Financial Compliance
arXiv:2608.199742026-08-21
AI trading agents say they know the rules, but still place orders that break them
The researchers built ReguSim, a simulated financial trading environment, and ReguBench, a monitoring benchmark, to test whether AI agents that operate in financial markets actually follow rules they claim to understand. Testing DeepSeek V4 Pro and Gemini 3.5 Flash as traders showed that even when rules were shown directly in the prompt, some orders were still rejected for violating them, and behavior shifted depending on incentives or assigned personas. They also found that if a monitoring AI only sees the trader's own stated reasoning, it can be misled into approving orders that should have been rejected, and that simple rule-based or statistical baselines matched or beat AI models at spotting suspicious trades.
What they did
The team built ReguSim, which separately logs four things -- the AI's stated reasoning, the action it attempts, whether the execution system accepts or rejects it, and the evidence shown to a monitor -- plus ReguBench, which marks a specific target trade for an AI to judge
They tested DeepSeek V4 Pro and Gemini 3.5 Flash as traders across regulatory settings modeled on the US, China A-share, and Hong Kong markets, plus a relaxed 'LAX' and a stricter 'STRICT' synthetic setting, over 30 decision steps, varying persona (aggressive vs. conservative) and objective (profit-seeking, stealthy, or neutral)
Even with rule text visible in every prompt, DeepSeek's submitted orders were rejected 24.2% of the time and Gemini's 14.8% of the time; the aggressive persona pushed DeepSeek's rejection rate 30.9 percentage points higher than the conservative persona, and removing rule text or replacing hard enforcement with prompt-only instructions failed to fix or fully explain the behavior
When a separate monitoring AI saw only the trader's own explanation for a rejected order, it wrongly approved it 46.9% of the time versus 25.0% when it saw just the state and action alone; showing the actual rejection code raised correct issue identification from 53.1% to 87.5%
In the monitoring benchmark, the best AI model (GPT-5.4 Mini) scored 63.8% macro F1, while a simple rule-based detector scored 65.0% and a logistic regression baseline scored 71.4% on the same test set, showing AI monitors did not clearly outperform simpler methods
Figure 1: ReguSim and ReguBench pipeline. ReguSim logs prompt state, trader rationale/action, execution outcome, and trace evidence; ReguBench evaluates a target-marked monitor judgment from that evidence.
Table 1: Regulatory and synthetic control settings used by the execution layer. US, China A-share, and Hong Kong are market-inspired settings; LAX and STRICT provide weaker and stronger rule-pressure controls. The gross exposure cap is an exchange-solvency control rather than a statutory manipulation rule.
Setting
Executable controls
US
Market-inspired setting with short and cover allowed; no daily price band; same-session resale allowed; 2.0× gross-exposure cap.
China
A-share
Market-inspired setting with no short action; 10% daily price band; T+1 resale restriction; 1.0× gross-exposure cap.
Hong Kong
Market-inspired setting with short and cover allowed; no daily price band; same-session resale allowed; 2.0× gross-exposure cap.
LAX
Weak-control baseline with short and same-session resale allowed; no daily price band; 5.0× gross-exposure cap.
STRICT
Strong-control baseline with no short action; 3% daily price band; T+1 resale restriction; 1% equity position cap; 1.0× gross-exposure cap.
Figure 2: DeepSeek V4 Pro rejected-attempt percentage across regimes and incentives, using submitted non-HOLD orders as the denominator. Each cell averages both personas; uncertainty is reported in Appendix C.
Table 2: ReguBench composition by surveillance category. Source codes distinguish synthetic, case-inspired, parameter-variant, and scale-variant templates; all records are synthetic.
Manipulation Type
Scenarios
Avg. Trades
Difficulties
Source
Wash Trading
46
60–1000
Easy/Med/Hard
Synth + Case + Variant
Spoofing
49
60–1000
Easy/Med/Hard
Synth + Case + Variant
Pump & Dump
32
30–500
Easy/Med/Hard
Synth + Case + Variant
Churning
32
60–200
Easy/Med/Hard
Synth + Variant
Marking the Close
32
60–200
Easy/Med/Hard
Synth + Variant
Total
191
49,440
3 levels
4 sources
Figure 3: Auxiliary session-level diagnostic for the trader experiment: articulated compliance-awareness score versus rejected-attempt percentage. The main text relies on the rejected trace example and ablations; this scatter plot is included only as supporting evidence that stated awareness is weakly related to executable compliance in the current run.
Table 3: Representative rejected China A-share trader trace. The regime rule text was visible in the prompt, so the failure is a rule-to-action and state-grounding mismatch rather than absence of regulatory knowledge.
Prompt-visible input
China A-share rule text was provided: no shorting, a 10% daily price band, and T+1 resale for shares bought in the session. The same prompt exposed the current price ($118.10), previous close ($102.30), long position (900 shares), cash, and equity.
Model response
{"action":"SELL","quantity":900,…}. The model reasoned that the stock had “surged 15.4%” and that selling would lock in profits. Its compliance statement claimed: “Selling existing long shares is permitted under T+1 rules … The order complies with all regulatory constraints.”
Execution result
Rejected with PRICE_BAND_HALT and T_PLUS_ONE_RESALE. The price-band violation is directly checkable from prompt-visible values: $118.10 exceeds the 10% upper band from the $102.30 previous close. The T+1 rejection comes from the execution ledger, which recorded same-session purchases still subject to resale restriction.
Table 4: Compact trader replication summary under the same regime–incentive–persona protocol. Rejected, Rule breach, Activity, and Persona gap (aggressive minus conservative rejected-attempt rate) are percentages.
Model
Decisions
Rejected
Rule breach
Activity
Persona gap
DeepSeek
4500
24.2
10.0
31.6
30.9
Gemini
1800
14.8
13.4
32.9
9.5
Table 5: DeepSeek V4 Pro trader ablations. All values are percentages; Rejected and Rule breach use submitted non-HOLD orders as the denominator, while Activity and Awareness use valid decision steps.
Variant
Rules
Persona
Execution
N
Rejected (%)
Rule br. (%)
Activity (%)
Awareness (%)
Full
Yes
Yes
Enforced
150
24.2
10.0
31.6
89.4
No regulation text
No
Yes
Enforced
150
33.2
21.7
34.4
83.7
No persona
Yes
No
Enforced
150
23.0
10.2
44.8
90.3
Prompt-only
Yes
Yes
Observed
150
24.5
8.6
38.0
90.0
Table 6: Target-marked monitor results with transparent baselines on the same 800-target sample. Macro Cell F1 is the main comparison metric; precision and recall show the operating point of each detector. Full category, difficulty, and bootstrap results are in Appendix C.
Detector
Source
Valid N
Macro Cell F1 (%)
Precision (%)
Recall (%)
DeepSeek V4 Pro
reported LLM run
800
46.5
38.0
71.0
Gemini 3.5 Flash
reported LLM run
788
54.5
43.8
85.7
GPT-5.4 Mini
reported LLM run
800
63.8
57.2
79.9
Rule baseline
target features
800
65.0
70.6
87.3
Logistic baseline
target features
800
71.4
85.1
84.4
Table 7: Target-marked monitor macro F1 percentages by surveillance category and difficulty.
Split
Cell
GPT
Gemini
DeepSeek
Type
Wash trading
67.2
67.2
53.3
Type
Spoofing
79.0
73.7
68.2
Type
Pump & dump
31.1
27.8
28.5
Type
Churning
77.4
75.5
63.3
Type
Marking close
64.5
28.4
19.0
Difficulty
Easy
71.2
60.3
60.8
Difficulty
Medium
65.2
57.1
49.1
Difficulty
Hard
55.1
46.2
29.6
Table 8: Input-modality ablation on the logged two-target-per-cell subset. LLM rows use DeepSeek V4 Pro and vary the evidence shown under the same targets, labels, and JSON schema. Struct. denotes derived evidence summaries; Rule and Logistic are non-LLM baselines on the same 90 targets. Values are target-level percentages.
Input
N
F1
P
R
Target only
90
0.0
0.0
0.0
Trade log
89
52.9
37.5
90.0
+ Status
90
61.8
44.7
100.0
+ Struct.
90
59.7
43.5
95.2
Features
90
58.0
41.7
95.2
Log + feat.
90
62.7
45.7
100.0
Rule
90
80.8
67.7
100.0
Logistic
90
87.2
94.4
81.0
Table 9: Models tested in this study. Parameter counts are omitted for closed-source provider models because they are not publicly disclosed.
Table 10: Bridge study on sampled ReguSim trader traces. Values are percentages over 64 traces per input condition. Rej. rec. is recall on rejected submitted orders; False acc. is the share of rejected orders incorrectly judged acceptable; Issue acc. is the rejected-trace issue-type accuracy; Evid. hit is a lightweight match between the monitor explanation and the true rejection-code family.
Input
Acc.
Rej. rec.
False acc.
Issue acc.
Evid. hit
State+action
87.5
75.0
25.0
71.9
78.1
+ trader rationale
76.6
53.1
46.9
53.1
68.8
+ execution result
89.1
78.1
21.9
87.5
100.0
Table 11: Monitor-only LLM results on the target-marked 800-target sample. Macro F1, precision, and recall are percentages averaged over type–difficulty–regime cells.
Model
Cells
Attempts
Valid N
Macro F1 (%)
Macro P (%)
Macro R (%)
GPT-5.4 Mini
45
800
800
63.8
57.2
79.9
Gemini 3.5 Flash
45
800
788
54.5
43.8
85.7
DeepSeek V4 Pro
45
800
800
46.5
38.0
71.0
Table 12: Paired bootstrap over the same 45 type–difficulty–regime cells. Values are percentage-point differences; positive values mean the first detector has higher macro cell F1.
Comparison
Paired Cells
Δ Macro Cell F1 (pp) [95% CI]
GPT-5.4 Mini - Gemini
45
9.3 [3.6,15.9]
GPT-5.4 Mini - DeepSeek
45
17.4 [9.8,25.4]
Gemini - DeepSeek
45
8.1 [3.8,13.0]
Rule - GPT-5.4 Mini
45
1.2 [-4.3,6.5]
Logistic - GPT-5.4 Mini
45
7.6 [1.0,13.9]
Rule - DeepSeek
45
18.5 [10.1,27.6]
Logistic - DeepSeek
45
24.9 [15.8,34.4]
Rule - Logistic
45
-6.4 [-13.3,0.6]
Table 13: Full DeepSeek V4 Pro trader-agent results. Each row averages both personas and five independent sessions per persona within a regime–incentive cell. Rejected is the percentage of submitted non-HOLD orders that trigger either a machine-checkable regulatory rule-breach attempt or an operational rejection; Rule breach is the regulatory subset of Rejected; Activity is the percentage of valid decision steps that execute a non-HOLD order.
Regime
Incentive
N
Rejected (%)
Rule br. (%)
Activity (%)
US
neutral
10
8.3
0.0
38.0
US
profit_max
10
32.1
0.0
39.7
US
stealth
10
35.2
0.0
20.0
CN_A_SHARE
neutral
10
6.8
6.8
21.3
CN_A_SHARE
profit_max
10
21.9
21.9
28.0
CN_A_SHARE
stealth
10
25.7
25.7
25.7
HK
neutral
10
8.6
0.0
36.3
HK
profit_max
10
29.8
0.0
38.3
HK
stealth
10
26.2
0.0
28.0
LAX
neutral
10
9.1
0.0
40.3
LAX
profit_max
10
32.1
0.0
46.0
LAX
stealth
10
32.6
0.0
29.3
STRICT
neutral
10
27.8
27.8
20.0
STRICT
profit_max
10
37.6
37.6
35.7
STRICT
stealth
10
30.0
30.0
27.7
Table 14: Full Gemini 3.5 Flash trader replication. Each row averages both personas and two independent sessions per persona within a regime–incentive cell. Rejected, Rule breach, and Activity use the same definitions as Table 13.
Regime
Incentive
N
Rejected (%)
Rule br. (%)
Activity (%)
US
neutral
4
0.0
0.0
26.7
US
profit_max
4
1.2
0.0
53.3
US
stealth
4
2.4
0.0
39.2
CN_A_SHARE
neutral
4
10.4
10.4
10.0
CN_A_SHARE
profit_max
4
15.6
15.6
23.3
CN_A_SHARE
stealth
4
17.9
17.9
41.7
HK
neutral
4
0.0
0.0
43.3
HK
profit_max
4
1.0
0.0
55.8
HK
stealth
4
12.5
0.0
41.7
LAX
neutral
4
0.0
0.0
37.5
LAX
profit_max
4
4.0
0.0
50.8
LAX
stealth
4
1.2
0.0
35.0
STRICT
neutral
4
54.9
54.9
10.0
STRICT
profit_max
4
46.6
46.6
14.2
STRICT
stealth
4
55.0
55.0
11.7
Table 15: Detector performance on the logged 90-target subset used for target-level qualitative analysis and sampled bootstrap comparisons. F1, precision, and recall are percentages.
Detector
Valid N
F1 (%)
Precision (%)
Recall (%)
GPT-5.4 Mini
90
70.0
53.8
100.0
Gemini 3.5 Flash
88
58.3
41.2
100.0
DeepSeek V4 Pro
85
65.6
50.0
95.2
Rule baseline
90
80.8
67.7
100.0
Logistic baseline
90
87.2
94.4
81.0
Table 16: Sampled target-level paired bootstrap comparisons on the logged two-target-per-cell subset. Differences are percentage points.
Comparison
Paired N
Δ F1 (pp) [95% CI]
GPT-5.4 Mini - Gemini 3.5 Flash
88
11.7 [5.1,19.2]
GPT-5.4 Mini - DeepSeek V4 Pro
85
8.1 [0.0,17.1]
Gemini 3.5 Flash - DeepSeek V4 Pro
84
-5.6 [-13.7,2.6]
GPT-5.4 Mini - Rule baseline
90
-10.8 [-20.4,-2.4]
GPT-5.4 Mini - Logistic baseline
90
-17.2 [-34.0,-0.9]
Rule baseline - Logistic baseline
90
-6.4 [-22.2,9.0]
Table 17: Additional paired uncertainty comparisons. Differences are percentage points.
Comparison
Paired N
Δ Micro F1 (pp) [95% CI]
Gemini - DeepSeek
84
-5.6 [-13.8,2.9]
DeepSeek - Rule
90
-18.4 [-29.4,-9.2]
Gemini - Rule
88
-22.4 [-33.3,-12.8]
DeepSeek - Logistic
90
-23.2 [-40.1,-7.5]
Gemini - Logistic
88
-28.8 [-46.1,-12.4]
Rule - Logistic
800
-6.6 [-10.6,-2.9]
Table 18: Qualitative taxonomy of monitor errors in the logged two-target-per-cell subset. The patterns describe how evidence is used or mislocalized; they are not additional manipulation labels.
Error pattern
What the monitor uses
Why it matters
Target–context substitution
Nearby suspicious trades are treated as evidence about the marked target.
A surveillance alert must attach evidence to the record being judged, not only to the surrounding episode.
Lifecycle blindness
Large or one-sided orders are judged without enough attention to whether they were filled, cancelled, rejected, or merely placed.
Spoofing-like behavior depends on order lifecycle evidence; a trade log alone can hide the decisive field.
Pattern over-triggering
High turnover, reversals, or concentrated orders are treated as sufficient for a positive label.
Suspicious patterns are review cues, but they are not legal conclusions without ownership, intent, lifecycle, and price-impact evidence.
Temporal mislocalization
Price movement before or after the target is summarized too coarsely.
Pump-and-dump and marking-the-close require sequence-level timing, so the same local record can look different under richer market context.
Structured-feature correction
Transparent rules or logistic features use explicit target-level counts, status fields, and local price summaries.
Baseline successes show that some failures are evidence-representation failures, not only failures to understand regulatory language.
Table 19: Representative monitor target-level qualitative/error-analysis cases from the logged 90-target subset. “Yes” means the detector classified the target as manipulative, not that the classification was correct. Gem., DS, and Logit denote Gemini 3.5 Flash, DeepSeek V4 Pro, and the logistic baseline; “close”, “pump”, and “wash” abbreviate marking-the-close, pump-and-dump, and wash-trading cases.
Case
Type
Diff.
Truth
GPT
Gem.
DS
Rule
Logit
Takeaway
C1
churning
easy
yes
yes
yes
yes
yes
no
All LLMs detect a salient positive example.
C2
churning
easy
no
yes
yes
yes
yes
no
All LLMs over-alert on non-manipulative context.
C3
close
medium
no
no
yes
yes
no
no
GPT resolves the target better than the other LLMs.
C4
pump
hard
no
yes
yes
no
no
no
DeepSeek is more conservative on this target.
C5
churning
easy
no
yes
yes
yes
no
no
Structured features correct an LLM failure.
C6
close
easy
no
no
yes
no
no
no
Harder temporal categories induce disagreement.
C7
churning
easy
no
yes
yes
yes
no
no
Logistic baseline avoids GPT’s target error.
C8
pump
easy
no
yes
yes
yes
no
no
Rule baseline avoids GPT’s target error.
C9
wash
easy
no
no
no
no
no
no
Selected for type coverage.
C10
spoofing
easy
yes
yes
yes
yes
yes
yes
Selected for type coverage.
Why it matters
As financial firms explore using AI for trading and surveillance, this work shows with concrete numbers that an AI stating awareness of a rule does not guarantee it will act on that rule, and that trusting an AI's own explanations can fool downstream monitoring systems too. This suggests any real deployment needs hard, independent enforcement checks and evidence-based verification rather than relying on the AI's stated reasoning alone.
Terms in this paper
ReguSim · a simulated trading environment where orders that break rules get rejected by the system
ReguBench · a benchmark that flags one specific suspicious trade for an AI to evaluate
rejected attempt · an order the AI submitted that the trading system blocked for violating a rule
target marking · explicitly flagging which single trade record the AI should judge, so it isn't guessing
logistic regression baseline · a traditional statistical (non-AI) method for classifying suspicious trades
Original abstract (English)
LLM agents in financial markets may cite rules yet still submit orders that violate executable constraints or misread surveillance evidence. We introduce ReguSim, a controlled financial-compliance environment, and ReguBench, a target-marked monitoring benchmark, to separate four artifacts: stated reasoning, attempted action, execution enforcement, and monitor evidence. In trader runs with DeepSeek V4 Pro and Gemini 3.5 Flash, visible rules reduce but do not eliminate rejected actions, and incentive or persona framing shifts behavior. A bridge study shows that trader rationales can mislead an independent monitor unless enforcement evidence is shown. In monitoring, simple structured baselines either match or exceed prompt-only LLMs. The results frame financial compliance evaluation as an audit of rule-grounded actions and evidence use, rather than a single compliance score.
Authors · Yiyang Luo, Yihang Jiang, Qijun Xie, Liang Lan, Lin Willian Cong, Anyi Rao, Yunya Song
