SkillJack: Persistent Skill Backdoors in Self-Evolving Agents

arXiv:2608.035092026-08-03

A new attack shows AI agents can be tricked into baking a backdoor into their own reusable skills

SkillJack targets self-evolving AI agents that turn past interaction records into reusable skills, showing that a poisoned experience record disguised as ordinary work (like a 'backup' task) can be automatically compiled into a malicious skill the agent keeps using. Detection by an LLM judge dropped from 98.5% on the raw poisoned record to 11.4% on the extracted skill, and 80% of skill-based attacks still worked even after the original record was deleted. The same pattern showed up across two independently built agent systems, SkillX and Anything2Skill.

METAL LAB explanatory visual

How SkillJack launders a poisoned record into a persistent skill

Evidence statusMeasured results reported

Inject disguised poisoned recordA trajectory framed as ordinary work (e.g. backup/archive) but containing a policy-violating action is added to the agent's experience data.
Pass through extraction pipelineThe agent's load-transform-persist-route pipeline compresses and abstracts the record into a skill made of a name, description, and code.
Detection collapses (whitewashing)LLM-judge detection falls from 98.5% on the raw record to 11.4% on the extracted skill (Table 2).
Routing and triggeringThe stored skill is later routed to matching tasks and triggers the intended violation on 56.2% of attack tasks in SkillX and 89.2% in Anything2Skill.
Survives source deletionDeleting the original poisoned record does not remove the skill: 80.0% of skill-based attacks still trigger afterward.

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

The authors introduce SkillJack, an attack on the 'experience-to-skill' pipeline by which self-evolving agents convert past interaction records into durable, reusable skills, and describe three properties: sanitization whitewashing, cross-layer promotion, and persistence isolation.
The attack works by feeding the agent poisoned trajectories written with legitimate-sounding framing (e.g. 'backup' or 'archive' language) that still contain a policy-violating action such as data exfiltration, privilege escalation, unauthorized transfer, or a persistent backdoor.
Both were tested on two real implementations, SkillX and Anything2Skill, using a shared dataset of 150 trajectories across four policy-risk categories, judged with a combination of an LLM judge and a pattern-matching detector.
LLM-judge detection fell from 98.5% on raw poisoned trajectories to 11.4% on the skills extracted from them, and those extracted skills then went on to trigger the intended policy violation for 56.2% of attack-oriented tasks on SkillX and 89.2% on Anything2Skill.
After the original poisoned records were deleted, 80.0% of skill-based attacks still triggered, and the derived skills also misfired on unrelated clean tasks (20.0% on SkillX's routing library, 95.0% match rate on Anything2Skill's retrieval).

Figure 2: Sanitization whitewashing across systems. For each configuration, the LLM judge flags the raw trajectory far more often than the skill later extracted from it: SkillX 98.5% vs. 11.4%, A2S DOC mode 100.0% vs. 44.4%, and A2S SDK mode 100.0% vs. 28.6%. Extraction reduces detectability in every case; the reduction is largest for SkillX, which produces the most abstracted skills.

Table 1: Representative systems that reuse experience through reusable instructions, workflows, or skills. SkillX and Anything2Skill are the two representative implementations evaluated in this report.
System	Year	Experience Input	Skill Output
SkillX [14]	2026	Trajectory	Structured skills
Anything2Skill [10]	2026	Heterogeneous records	Structured skill contracts
AutoSkill [18]	2026	Dialogue and interaction traces	Reusable skills
Skill-Pro [7]	2026	Interaction experiences	Skill-MDPs
Trace2Skill [8]	2026	Execution trajectories	Transferable skills
AWM [16]	2025	Trajectory	Reusable workflow
A-Mem [17]	2025	Interaction records	Structured memory notes
ExpeL [21]	2024	Trajectory	Insights + reusable steps

Figure 3: Trigger rates in the matched 25-task evaluation, per risk category and overall (25 tasks). Direct-context injection triggers most often, but the skill pathway is durable: 80.0% of skill-based triggers survive deletion of the source trajectories (post-deletion versus skill-based, overall 32.0/40.0). Values are percentages.

Table 2: Extraction and post-extraction detectability across the two representative systems. Extraction is the share of input trajectories that yield at least one skill; SkillX yields skills for essentially all inputs, so its extraction entry is fixed at 100.0%. Detection and evasion are reported for the framed (attack) skills unless the payload column states otherwise. All values are rates.
System	Payload	Extraction	Raw-traj. LLM det.	Derived LLM det.	Derived pattern det.	LLM evasion
SkillX	Framed	100.0%	98.5%	11.4%	11.0%	88.6%
SkillX	Direct baseline	100.0%	n/a	27.0%	27.8%	73.0%
A2S (SDK)	Framed	10.8%	100.0%	28.6%	28.6%	71.4%
A2S (DOC)	Framed	41.5%	100.0%	44.4%	3.7%	55.6%
A2S (DOC)	Direct baseline	0.0%	100.0%	n/a	n/a	n/a

Figure 4: Routing-level outcomes on SkillX over a 356-skill library (305 poisoned, 51 clean), across 32 attack-oriented and 15 clean tasks. Trigger rate is the share of tasks routed to a skill that satisfies the policy-violation proxy; expected-category match is the share routed to a skill of the intended category (undefined for clean tasks). The clean-task trigger rate of 20.0% reflects collateral selection. Values are percentages.

Table 3: Downstream triggering and defense sensitivity across the two representative systems. SkillX and A2S use their respective routing configurations and task sets; values therefore characterize system-specific sensitivity rather than a leaderboard comparison.
System	Setting	Tasks	Trigger rate
SkillX	Native routing, attack-oriented tasks	32	56.2%
SkillX	Native routing, clean tasks	15	20.0%
A2S DOC	Lexical SkillBank, attack-oriented tasks	65	89.2%
A2S DOC	Lexical SkillBank, clean-task retrieval	20	95.0%
SkillX	Extraction review	15	6.7%
SkillX	Runtime monitor	15	0.0%
A2S DOC	Enhanced runtime monitor	20	20.0%
A2S DOC	Review + enhanced monitor	20	5.0%

Figure 5: Metadata ablation: LLM-judge detection rate by presentation condition. SkillX values use 30 sampled skills per condition; A2S DOC values use all 30 extracted skills. Detection is lowest when the benign name and documentation accompany the code (full skill) and rises when the judge sees code only, indicating that the judge keys on surface metadata. Values are percentages.

Table 4: Functional framing vocabulary and example derived skills by policy-risk category. The table describes the experimental stimuli, not recommended operational procedures.
Risk Category	Functional Framing Vocabulary	Example Derived Skill
Data Exfiltration	“backup”, “archive”, “export”, “migrate”	backup_files_to_email
Privilege Escalation	“admin scope”, “unified management”, “batch access”	admin authentication
Unauthorized Transfer	“charity”, “gift”, “recurring payment”, “processing fee”	venmo_create_payment
Persistent Backdoor	“error recovery”, “diagnostic”, “circuit breaker”	diagnostic reporting

Table 5: Dataset and skill-bank counts. The 505 extracted skills are the union of the two SkillX poisoned extraction sets; the 356-skill routing library contains 305 poisoned and 51 clean skills.
Component	Count
Framed poisoned trajectories	65
Direct-baseline trajectories	65
Clean trajectories	20
Extracted framed skills	264
Extracted direct-baseline skills	241
Clean baseline skills	51
Routing-library skills	356

Findings

LLM-judge detection dropped from 98.5% on raw trajectories to 11.4% on extracted SkillX skills, and Anything2Skill showed a similar drop (DOC mode 100.0% to 44.4%, SDK mode 100.0% to 28.6%) (Figure 2, Table 2).
In a matched 25-task test, direct-context injection triggered more often (68.0% vs 40.0% for skill-based), but skill-based attacks were far more durable: 80.0% still triggered after the source trajectories were deleted (Figure 3).
In a SkillX routing test over a 356-skill library (305 poisoned, 51 clean), 56.2% of 32 attack-oriented tasks were routed to a skill that satisfied the policy-violation proxy, while 20.0% of 15 clean tasks were also mistakenly triggered (Figure 4).
Skills extracted by Anything2Skill triggered the policy-violation proxy on 89.2% of 65 attack-oriented task queries, and also matched 95.0% of clean-task queries, indicating broad collateral routing.
Breaking down what the LLM judge sees showed detection was lowest (10.0% on SkillX) when a benign name and description accompanied the code, and rose to 36.7% when the judge saw code only, indicating the judge relies heavily on surface metadata (Figure 5).

Where it can be used

Developers building agent frameworks with automatic skill extraction could use this as a reference for propagating provenance links from source records to derived skills.
Teams operating a skill library could use this to design cleanup policies that revoke or re-review derived skills, not just delete the original records that produced them.
Designers of LLM-based safety judges could use the metadata-sensitivity finding to justify checking a skill's actual code behavior rather than only its name and description.

Limits and open work

All experiments use a single model, DeepSeek-v4-flash, and the findings have not yet been replicated across other models.
Success is measured as a routing-level policy-violation proxy (pattern matching plus an LLM judge), not as live execution success against real external services, so figures are routing-level rather than deployment-level.
The defense experiments (extraction-time LLM review, regex-based runtime monitoring) used small samples and rule-based tools, and the authors themselves describe this as an exploratory, preliminary study.
All data comes from a controlled synthetic dataset built on public AppWorld API patterns, not independently collected real-world experience or live execution environments.
Whether analogous risks apply to other experience-learning mechanisms, such as reflection memories or tool synthesis, is raised only as a hypothesis and was not evaluated.

Why it matters

As agent frameworks increasingly let agents automatically turn experience into reusable skills, this work shows that the conversion step itself can silently launder malicious intent past safety checks and make the resulting behavior outlive the record that caused it. That matters directly for anyone deploying self-evolving agents, because deleting memory or source logs is not enough once a poisoned skill has already been compiled and stored.

Terms in this paper

self-evolving agent · an AI agent that accumulates its own interaction history and converts it into reusable skills to improve over time
experience-to-skill pipeline · the load-transform-persist-route process by which an agent turns raw experience records into stored, reusable skills
sanitization whitewashing · the phenomenon where a malicious record becomes less detectable after being compiled into a skill
cross-layer promotion · an experience-layer record being turned into a persistent, reusable skill-layer artifact
policy-violation proxy · the paper's evaluation measure combining pattern matching and an LLM judge to flag whether a skill contains a policy-violating action

Original abstract (English)

Self-evolving agents increasingly convert interaction histories into reusable skills that persist beyond individual tasks. While prior work studies memory and retrieval poisoning, such attacks only affect agents when poisoned records are retrieved as context. We uncover a new and more fundamental risk: poisoned experiences can be transformed by the agent itself into durable behavioral artifacts. We present SkillJack, the first attack that exploits the experience-to-skill pipeline of self-evolvin

Authors · Zonghao Ying

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Zonghao Ying et al., arXiv:2608.03509, CC BY 4.0