Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation
arXiv:2608.192992026-08-21
Researchers put AI in the air traffic controller's seat and found it sounds right but often gives the wrong instructions
Using a hand-transcribed real radio exchange from a general-aviation flight over San Francisco's Bay Tour as ground truth, researchers tested how well nine large language models (LLMs) could play air traffic controller across a full 36-turn conversation. Making the prompt more detailed and rule-heavy actually hurt performance, and when models kept feeding their own past replies back into the conversation, early mistakes snowballed. In the end, the models were good at sounding like controllers but poor at making operationally correct decisions.
What they did
A real 36-turn pilot-controller radio transcript from a Bay Area tour flight was used as ground truth, and nine models (six open-weight, three closed-source) generated controller replies while conditioning on the growing dialogue history, just as a real controller would
Five prompts were tested from a bare-bones role assignment up to a heavily rule-laden version; the lightest prompts performed best, while the most over-specified prompt caused a 35% relative drop in similarity to the ground truth
When models conditioned on their own prior replies, early errors (like defaulting to a repetitive 'Roger' acknowledgment) compounded and dragged performance down over the conversation, but feeding in the true controller history instead fixed this collapse
Adding a worked example transcript into the prompt (in-context learning) improved surface similarity to ground truth for most models, but did not improve an LLM-judge's assessment of operational correctness
Models scored about 4.1 out of 5 on callsign handling and conciseness but only 2.1 out of 5 on operational correctness, and bigger open-source models did not close this gap
Figure 1: Overall architecture. (1) Generation: pilot transmissions transcribed from an experimental Bay Area tour flight are issued one turn at a time to the language model, which is conditioned on a system prompt and on the communication history accumulated so far; each pseudo-ATC response is appended back into that history. (2) Teacher-forced generation: the same scenario is replayed with the controller side of the history replaced by the ground truth transmissions. The example contrasts the two regimes at turn 24. Conditioned on its own replies (left), the model has fallen into an Acknowledgment loop and answers “Roger,” missing the required radar-service termination and Hayward handoff. Given the true history (right), the same turn is answered with the correct facility, the correct “keep the code” instruction, and the correct frequency 120.2. (3) Evaluation: every turn is scored by automatic similarity metrics, by an LLM-as-judge, and—on a stratified subsample—by a human expert.Figure 2: Prompt × (ICL × history) interaction, pooled over the nine models (mean ± 95% CI across models). Color encodes the dialogue history the model conditions on, line style encodes whether a worked in-context example was supplied. Under the light prompts C1–C3 the two history regimes are indistinguishable; under the over-specified C5 the self-history condition collapses and ground-truth grounding recovers it.
Table 1: Per-prompt similarity to the C0 ground truth, by condition (ICL × history), averaged over the 9 models and 3 attempts. Best per column in bold.
ROUGE-L
Semantic cosine
Prompt
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
C1
0.216
0.209
0.244
0.244
0.464
0.449
0.497
0.490
C2
0.217
0.214
0.243
0.230
0.482
0.472
0.492
0.477
C3
0.219
0.212
0.243
0.224
0.487
0.470
0.493
0.467
C4
0.209
0.220
0.216
0.238
0.430
0.445
0.447
0.470
C5
0.167
0.218
0.159
0.224
0.302
0.383
0.298
0.396
Figure 3: Cumulative running-mean similarity over the 36-turn conversation (ICL on, pooled over models). Under C1–C3 the self- and ground-truth-history trajectories track each other; under C5 self-history drifts steadily downward as early errors propagate through the dialogue history, while ground-truth injection holds the trajectory flat.Figure 4: LLM-judge score per rubric dimension, pooled over the nine models. Color encodes the dialogue history the model conditions on; hatching marks the in-context learning condition. Models score highly on callsign handling and conciseness but poorly on operational correctness.
Table 2: Per-model similarity to the C0 ground truth, by condition (ICL × history), averaged over prompts C1–C5 and 3 attempts. Best per column in bold.
ROUGE-L
Semantic cosine
Model
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
Open-source
Qwen2.5-7B
0.197
0.215
0.212
0.234
0.413
0.458
0.443
0.483
Llama-3.1-8B
0.193
0.181
0.208
0.203
0.437
0.403
0.452
0.418
Gemma-2-9B
0.188
0.218
0.216
0.248
0.405
0.441
0.438
0.487
Qwen2.5-14B
0.203
0.200
0.215
0.195
0.481
0.464
0.483
0.456
Qwen2.5-32B
0.215
0.213
0.240
0.234
0.460
0.453
0.476
0.474
Mixtral-8x7B
0.123
0.149
0.116
0.150
0.336
0.361
0.310
0.373
Average
0.186
0.196
0.201
0.211
0.422
0.430
0.434
0.449
Closed-source
GPT-5.4
0.246
0.236
0.255
0.247
0.470
0.450
0.470
0.449
GPT-5.4-mini
0.246
0.237
0.253
0.267
0.447
0.469
0.463
0.482
Claude-Sonnet-4.6
0.240
0.282
0.275
0.310
0.447
0.495
0.474
0.519
Average
0.244
0.252
0.261
0.275
0.455
0.471
0.469
0.483
Figure 5: Per-dimension scores from the LLM judge (a) and the human expert (b) on the same 180 annotated outputs, broken out by condition (color = dialogue history, hatching = in-context example). Both raters produce the same cross-dimension profile, peaking on callsign handling and conciseness and dipping on operational correctness, but the human sits about one point higher on every dimension. Because both panels use the identical 180-sample subset, the vertical offset between them is the genuine human–judge leniency gap.(b) Human expert
Table 3: Hypothesis tests for the three experimental contrasts, computed on per-model means across the nine models. Paired contrasts use an exact Wilcoxon signed-rank test (W); the closed-versus-open contrast uses a one-sided Mann–Whitney U test. Δ is reported in the direction named in each group heading, wins counts models favoring that direction, and the last column is the effect size (dz for the paired contrasts, Cohen’s d for the unpaired one). Bold indicates p<0.05.
Measure
Δ
Wins
Statistic
p
Effect
In-context example vs. none (paired, W, n=9)
ROUGE-L (pooled)
0.016
8/9
1.0
0.008
1.46
Normalized BLEU (pooled)
0.009
8/9
1.0
0.008
1.47
LLM-judge overall
0.036
5/9
13.0
0.547
0.25
Ground-truth vs. self dialogue history (paired, W, n=9)
ROUGE-L (pooled)
0.010
5/9
14.0
0.359
0.49
BLEU-1 (pooled)
0.044
9/9
0.0
0.004
1.98
ROUGE-L (prompt P1)
−0.003
4/9
20.0
0.820
−0.11
ROUGE-L (prompt P5)
0.057
9/9
0.0
0.004
1.72
LLM-judge overall
−0.078
3/9
14.5
0.371
−0.40
Closed- vs. open-source (unpaired, U, 3 vs. 6)
ROUGE-L
0.059
—
18.0
0.012
2.03
LLM-judge overall
1.050
—
18.0
0.012
4.43
Figure 6: Model-level agreement between each automatic similarity metric and the LLM judge. Each point is one model, plotted as its mean metric score against its mean judge rating, with a fitted line and 95% confidence band. ROUGE-L and TF-IDF track expert judgment closely; BLEU-1 and sentence-embedding similarity do not. The open/closed legend in (a) applies to all four panels.(b) TF-IDF cosine
Table 4: LLM-judge scores (1–5) per model, averaged over conditions and the top-25 judged outputs per cell. Best per column in bold.
Model
Phraseology
Correctness
Callsign
Conciseness
Safety
Halluc.-free
Overall
Open-source
Qwen2.5-7B
2.46
1.75
4.36
3.97
2.63
2.68
1.99
Llama-3.1-8B
2.40
1.74
3.65
3.88
2.51
2.22
1.89
Gemma-2-9B
2.80
1.94
4.14
4.34
2.80
2.37
2.12
Qwen2.5-14B
2.35
1.70
4.44
3.17
2.47
2.00
1.84
Qwen2.5-32B
2.81
2.00
4.29
3.83
2.83
2.33
2.20
Mixtral-8x7B
2.00
1.32
3.54
3.42
2.11
1.82
1.45
Average
2.47
1.74
4.07
3.77
2.56
2.24
1.91
Closed-source
GPT-5.4
3.84
2.86
4.24
4.78
3.81
3.77
3.13
GPT-5.4-mini
3.59
2.57
4.15
4.61
3.54
3.20
2.82
Claude-Sonnet-4.6
3.72
2.68
4.35
4.75
3.60
3.31
2.95
Average
3.71
2.71
4.25
4.71
3.65
3.43
2.96
(c) BLEU-1(d) Semantic cosine
Table 5: Spearman rank-correlation matrix over all six evaluation measures, computed on the 180 human-annotated outputs so that every measure is available for every observation. The two reference-overlap metrics (ROUGE-L, TF-IDF) are the only automatic measures that track either human-facing judgement.
ROUGE-L
TF-IDF
BLEU-1
Semantic
LLM judge
TF-IDF
0.73
BLEU-1
0.58
0.36
Semantic
0.32
0.47
0.30
LLM judge
0.53
0.49
0.02
0.10
Human
0.34
0.38
-0.05
0.09
0.67
Figure 7: Mean overall rating assigned by the LLM judge and by the human expert, split by the judge’s own score band (mean ± 95% CI). The human is systematically more lenient, and the gap grows as output quality falls.Figure 8: Per-model similarity by history mode (ICL on), grouped into open- and closed-source models. Bars are self history (hatched //) versus ground-truth history (hatched \\); panels are the four similarity metrics.
Table 6: Agreement between the three evaluation layers. Sample-level statistics treat each scored output as one observation; model-level statistics use per-model means (n=9). κw is quadratic-weighted Cohen’s kappa, defined only when both measures share the 1–5 ordinal scale.
Comparison
Level
Pearson r
Spearman ρ
κw
Automatic metric vs. LLM-judge overall (sample n=4,500)
ROUGE-L
sample
0.51
0.52
–
ROUGE-L
model
0.90
0.93
–
TF-IDF
sample
0.50
0.48
–
TF-IDF
model
0.91
0.90
–
BLEU-1
sample
0.12
0.06
–
BLEU-1
model
0.43
0.22
–
Semantic
sample
0.08
0.09
–
Semantic
model
0.67
0.50
–
Automatic metric vs. human overall (sample n=180)
ROUGE-L
sample
0.31
0.34
–
ROUGE-L
model
0.73
0.81
–
TF-IDF
sample
0.32
0.38
–
TF-IDF
model
0.72
0.79
–
BLEU-1
sample
0.02
-0.05
–
BLEU-1
model
0.17
-0.03
–
Semantic
sample
0.09
0.09
–
Semantic
model
0.39
0.28
–
LLM judge vs. human annotator (sample n=180)
Overall rating
sample
0.71
0.67
0.60
Overall rating
model
0.69
0.86
–
Table 7: Per-dimension and per-stratum agreement between the LLM judge and the human annotator (n=180). Δ is the human mean minus the judge mean; positive values mean the human is more lenient. Within-stratum correlations are attenuated by range restriction and κw is uninformative there, so it is reported only for the pooled sample.
Dimension
Judge mean
Human mean
Δ
Spearman ρ
κw
Phraseology
3.24
4.30
+1.06
0.45
0.36
Correctness
2.89
3.79
+0.90
0.63
0.52
Callsign
4.43
4.86
+0.43
0.35
0.43
Conciseness
4.34
4.51
+0.17
0.41
0.59
Safety
3.63
3.91
+0.28
0.53
0.59
Halluc.-free
3.61
3.96
+0.34
0.48
0.46
Overall
3.12
3.91
+0.79
0.67
0.60
Overall rating, split by judge stratum
Low (n=45)
1.13
2.58
+1.44
0.44
–
Mid (n=45)
2.87
3.78
+0.91
0.36
–
High (n=90)
4.24
4.64
+0.40
0.20
–
Why it matters
As industry explores AI assistance for safety-critical tasks like air traffic control, this study offers concrete evidence of a persistent gap between sounding correct and being correct in current LLMs. It supports using AI for supporting roles like drafting routine messages, training simulators, or reviewing recordings, rather than placing it directly in the control loop.
Terms in this paper
LLM-as-judge · using a separate large language model to score the quality of generated outputs instead of a human
in-context learning (ICL) · giving a model a worked example directly in its prompt so it can imitate the pattern
ROUGE-L, BLEU, TF-IDF · automatic text-similarity metrics that measure how closely generated text matches a reference in wording and order
teacher forcing · replacing a model's own past outputs with the true correct history when generating the next turn
quadratic-weighted Cohen's kappa (κw) · a statistic measuring how closely two raters agree on ordinal scores
Original abstract (English)
Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco "Bay Tour" route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.
Authors · Mahyar Ghazanfari, Matthias Casanova, Jordan Kam, Alex Zongo, Peng Wei, Torsten Darrell, Alexandre Bayen
