Figure 1: Overall architecture. (1) Generation: pilot transmissions transcribed from an experimental Bay Area tour flight are issued one turn at a time to the language model, which is conditioned on a system prompt and on the communication history accumulated so far; each pseudo-ATC response is appended back into that history. (2) Teacher-forced generation: the same scenario is replayed with the controller side of the history replaced by the ground truth transmissions. The example contrasts the two regimes at turn 24. Conditioned on its own replies (left), the model has fallen into an Acknowledgment loop and answers “Roger,” missing the required radar-service termination and Hayward handoff. Given the true history (right), the same turn is answered with the correct facility, the correct “keep the code” instruction, and the correct frequency 120.2. (3) Evaluation: every turn is scored by automatic similarity metrics, by an LLM-as-judge, and—on a stratified subsample—by a human expert.Figure 2: Prompt × (ICL × history) interaction, pooled over the nine models (mean ± 95% CI across models). Color encodes the dialogue history the model conditions on, line style encodes whether a worked in-context example was supplied. Under the light prompts C1–C3 the two history regimes are indistinguishable; under the over-specified C5 the self-history condition collapses and ground-truth grounding recovers it.
Table 1: Per-prompt similarity to the C0 ground truth, by condition (ICL × history), averaged over the 9 models and 3 attempts. Best per column in bold.
ROUGE-L
Semantic cosine
Prompt
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
C1
0.216
0.209
0.244
0.244
0.464
0.449
0.497
0.490
C2
0.217
0.214
0.243
0.230
0.482
0.472
0.492
0.477
C3
0.219
0.212
0.243
0.224
0.487
0.470
0.493
0.467
C4
0.209
0.220
0.216
0.238
0.430
0.445
0.447
0.470
C5
0.167
0.218
0.159
0.224
0.302
0.383
0.298
0.396
Figure 3: Cumulative running-mean similarity over the 36-turn conversation (ICL on, pooled over models). Under C1–C3 the self- and ground-truth-history trajectories track each other; under C5 self-history drifts steadily downward as early errors propagate through the dialogue history, while ground-truth injection holds the trajectory flat.Figure 4: LLM-judge score per rubric dimension, pooled over the nine models. Color encodes the dialogue history the model conditions on; hatching marks the in-context learning condition. Models score highly on callsign handling and conciseness but poorly on operational correctness.
Table 2: Per-model similarity to the C0 ground truth, by condition (ICL × history), averaged over prompts C1–C5 and 3 attempts. Best per column in bold.
ROUGE-L
Semantic cosine
Model
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
Open-source
Qwen2.5-7B
0.197
0.215
0.212
0.234
0.413
0.458
0.443
0.483
Llama-3.1-8B
0.193
0.181
0.208
0.203
0.437
0.403
0.452
0.418
Gemma-2-9B
0.188
0.218
0.216
0.248
0.405
0.441
0.438
0.487
Qwen2.5-14B
0.203
0.200
0.215
0.195
0.481
0.464
0.483
0.456
Qwen2.5-32B
0.215
0.213
0.240
0.234
0.460
0.453
0.476
0.474
Mixtral-8x7B
0.123
0.149
0.116
0.150
0.336
0.361
0.310
0.373
Average
0.186
0.196
0.201
0.211
0.422
0.430
0.434
0.449
Closed-source
GPT-5.4
0.246
0.236
0.255
0.247
0.470
0.450
0.470
0.449
GPT-5.4-mini
0.246
0.237
0.253
0.267
0.447
0.469
0.463
0.482
Claude-Sonnet-4.6
0.240
0.282
0.275
0.310
0.447
0.495
0.474
0.519
Average
0.244
0.252
0.261
0.275
0.455
0.471
0.469
0.483
Figure 5: Per-dimension scores from the LLM judge (a) and the human expert (b) on the same 180 annotated outputs, broken out by condition (color = dialogue history, hatching = in-context example). Both raters produce the same cross-dimension profile, peaking on callsign handling and conciseness and dipping on operational correctness, but the human sits about one point higher on every dimension. Because both panels use the identical 180-sample subset, the vertical offset between them is the genuine human–judge leniency gap.(b) Human expert
Table 3: Hypothesis tests for the three experimental contrasts, computed on per-model means across the nine models. Paired contrasts use an exact Wilcoxon signed-rank test (W); the closed-versus-open contrast uses a one-sided Mann–Whitney U test. Δ is reported in the direction named in each group heading, wins counts models favoring that direction, and the last column is the effect size (dz for the paired contrasts, Cohen’s d for the unpaired one). Bold indicates p<0.05.
Measure
Δ
Wins
Statistic
p
Effect
In-context example vs. none (paired, W, n=9)
ROUGE-L (pooled)
0.016
8/9
1.0
0.008
1.46
Normalized BLEU (pooled)
0.009
8/9
1.0
0.008
1.47
LLM-judge overall
0.036
5/9
13.0
0.547
0.25
Ground-truth vs. self dialogue history (paired, W, n=9)
ROUGE-L (pooled)
0.010
5/9
14.0
0.359
0.49
BLEU-1 (pooled)
0.044
9/9
0.0
0.004
1.98
ROUGE-L (prompt P1)
−0.003
4/9
20.0
0.820
−0.11
ROUGE-L (prompt P5)
0.057
9/9
0.0
0.004
1.72
LLM-judge overall
−0.078
3/9
14.5
0.371
−0.40
Closed- vs. open-source (unpaired, U, 3 vs. 6)
ROUGE-L
0.059
—
18.0
0.012
2.03
LLM-judge overall
1.050
—
18.0
0.012
4.43
Figure 6: Model-level agreement between each automatic similarity metric and the LLM judge. Each point is one model, plotted as its mean metric score against its mean judge rating, with a fitted line and 95% confidence band. ROUGE-L and TF-IDF track expert judgment closely; BLEU-1 and sentence-embedding similarity do not. The open/closed legend in (a) applies to all four panels.(b) TF-IDF cosine
Table 4: LLM-judge scores (1–5) per model, averaged over conditions and the top-25 judged outputs per cell. Best per column in bold.
Model
Phraseology
Correctness
Callsign
Conciseness
Safety
Halluc.-free
Overall
Open-source
Qwen2.5-7B
2.46
1.75
4.36
3.97
2.63
2.68
1.99
Llama-3.1-8B
2.40
1.74
3.65
3.88
2.51
2.22
1.89
Gemma-2-9B
2.80
1.94
4.14
4.34
2.80
2.37
2.12
Qwen2.5-14B
2.35
1.70
4.44
3.17
2.47
2.00
1.84
Qwen2.5-32B
2.81
2.00
4.29
3.83
2.83
2.33
2.20
Mixtral-8x7B
2.00
1.32
3.54
3.42
2.11
1.82
1.45
Average
2.47
1.74
4.07
3.77
2.56
2.24
1.91
Closed-source
GPT-5.4
3.84
2.86
4.24
4.78
3.81
3.77
3.13
GPT-5.4-mini
3.59
2.57
4.15
4.61
3.54
3.20
2.82
Claude-Sonnet-4.6
3.72
2.68
4.35
4.75
3.60
3.31
2.95
Average
3.71
2.71
4.25
4.71
3.65
3.43
2.96
(c) BLEU-1(d) Semantic cosine
Table 5: Spearman rank-correlation matrix over all six evaluation measures, computed on the 180 human-annotated outputs so that every measure is available for every observation. The two reference-overlap metrics (ROUGE-L, TF-IDF) are the only automatic measures that track either human-facing judgement.
ROUGE-L
TF-IDF
BLEU-1
Semantic
LLM judge
TF-IDF
0.73
BLEU-1
0.58
0.36
Semantic
0.32
0.47
0.30
LLM judge
0.53
0.49
0.02
0.10
Human
0.34
0.38
-0.05
0.09
0.67
Figure 7: Mean overall rating assigned by the LLM judge and by the human expert, split by the judge’s own score band (mean ± 95% CI). The human is systematically more lenient, and the gap grows as output quality falls.Figure 8: Per-model similarity by history mode (ICL on), grouped into open- and closed-source models. Bars are self history (hatched //) versus ground-truth history (hatched \\); panels are the four similarity metrics.
Table 6: Agreement between the three evaluation layers. Sample-level statistics treat each scored output as one observation; model-level statistics use per-model means (n=9). κw is quadratic-weighted Cohen’s kappa, defined only when both measures share the 1–5 ordinal scale.
Comparison
Level
Pearson r
Spearman ρ
κw
Automatic metric vs. LLM-judge overall (sample n=4,500)
ROUGE-L
sample
0.51
0.52
–
ROUGE-L
model
0.90
0.93
–
TF-IDF
sample
0.50
0.48
–
TF-IDF
model
0.91
0.90
–
BLEU-1
sample
0.12
0.06
–
BLEU-1
model
0.43
0.22
–
Semantic
sample
0.08
0.09
–
Semantic
model
0.67
0.50
–
Automatic metric vs. human overall (sample n=180)
ROUGE-L
sample
0.31
0.34
–
ROUGE-L
model
0.73
0.81
–
TF-IDF
sample
0.32
0.38
–
TF-IDF
model
0.72
0.79
–
BLEU-1
sample
0.02
-0.05
–
BLEU-1
model
0.17
-0.03
–
Semantic
sample
0.09
0.09
–
Semantic
model
0.39
0.28
–
LLM judge vs. human annotator (sample n=180)
Overall rating
sample
0.71
0.67
0.60
Overall rating
model
0.69
0.86
–
Table 7: Per-dimension and per-stratum agreement between the LLM judge and the human annotator (n=180). Δ is the human mean minus the judge mean; positive values mean the human is more lenient. Within-stratum correlations are attenuated by range restriction and κw is uninformative there, so it is reported only for the pooled sample.
Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco "Bay Tour" route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.
作者 · Mahyar Ghazanfari, Matthias Casanova, Jordan Kam, Alex Zongo, Peng Wei, Torsten Darrell, Alexandre Bayen
