Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation
arXiv:2608.192992026-08-21
AI에게 관제탑 역할을 맡겨봤더니, 말투는 그럴듯한데 정작 중요한 지시는 자꾸 틀렸다
연구진은 샌프란시스코 상공을 실제로 비행한 경비행기의 조종사-관제사 무전 기록을 정답 삼아, 9개의 거대언어모델(LLM)이 관제사 역할을 얼마나 잘 해내는지 실험했다. 프롬프트(지시문)를 다섯 단계로 정교하게 만들수록 오히려 성능이 떨어졌고, 모델이 자기가 뱉은 대답을 계속 참고하면 초반 실수가 대화 내내 눈덩이처럼 불어났다. 결과적으로 모델들은 관제사처럼 '말하는 형식'은 곧잘 흉내 냈지만 '운항상 올바른 판단'은 하지 못했다.
무엇을 했나
샌프란시스코 베이 투어를 비행한 경비행기의 실제 무전 기록 36턴을 정답(ground truth)으로 만들고, 이전 대화를 계속 누적해서 참고하는 방식으로 9개 모델(오픈소스 6개, 상용 3개)을 테스트했다
가벼운 프롬프트(역할만 부여)부터 규칙을 잔뜩 넣은 프롬프트까지 5단계로 만들어 비교한 결과, 가장 단순한 프롬프트가 가장 좋은 성적을 냈고 규칙을 과하게 넣은 프롬프트는 성능이 35%나 떨어졌다
모델이 자기 이전 답변을 계속 참고하는 조건에서는 초반의 사소한 실수(예: 무조건 '알겠습니다'라고만 답하는 패턴)가 대화 후반까지 이어지며 성능이 계속 떨어졌지만, 실제 관제사의 정답 대화를 대신 넣어주면 이 문제가 해결됐다
예시 대화 하나를 프롬프트에 넣어주면(인컨텍스트 러닝) 겉보기 유사도는 좋아졌지만, GPT-5.5를 심사위원으로 활용한 평가에서는 실제 운항 판단력이 좋아지지는 않았다
모델들은 호출부호 처리나 간결함에서는 5점 만점에 4.1점을 받았지만, 실제로 맞는 지시를 내리는 '운항상 정확성'에서는 2.1점에 그쳤고, 이는 모델 크기를 키워도 해결되지 않았다
Figure 1: Overall architecture. (1) Generation: pilot transmissions transcribed from an experimental Bay Area tour flight are issued one turn at a time to the language model, which is conditioned on a system prompt and on the communication history accumulated so far; each pseudo-ATC response is appended back into that history. (2) Teacher-forced generation: the same scenario is replayed with the controller side of the history replaced by the ground truth transmissions. The example contrasts the two regimes at turn 24. Conditioned on its own replies (left), the model has fallen into an Acknowledgment loop and answers “Roger,” missing the required radar-service termination and Hayward handoff. Given the true history (right), the same turn is answered with the correct facility, the correct “keep the code” instruction, and the correct frequency 120.2. (3) Evaluation: every turn is scored by automatic similarity metrics, by an LLM-as-judge, and—on a stratified subsample—by a human expert.Figure 2: Prompt × (ICL × history) interaction, pooled over the nine models (mean ± 95% CI across models). Color encodes the dialogue history the model conditions on, line style encodes whether a worked in-context example was supplied. Under the light prompts C1–C3 the two history regimes are indistinguishable; under the over-specified C5 the self-history condition collapses and ground-truth grounding recovers it.
Table 1: Per-prompt similarity to the C0 ground truth, by condition (ICL × history), averaged over the 9 models and 3 attempts. Best per column in bold.
ROUGE-L
Semantic cosine
Prompt
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
C1
0.216
0.209
0.244
0.244
0.464
0.449
0.497
0.490
C2
0.217
0.214
0.243
0.230
0.482
0.472
0.492
0.477
C3
0.219
0.212
0.243
0.224
0.487
0.470
0.493
0.467
C4
0.209
0.220
0.216
0.238
0.430
0.445
0.447
0.470
C5
0.167
0.218
0.159
0.224
0.302
0.383
0.298
0.396
Figure 3: Cumulative running-mean similarity over the 36-turn conversation (ICL on, pooled over models). Under C1–C3 the self- and ground-truth-history trajectories track each other; under C5 self-history drifts steadily downward as early errors propagate through the dialogue history, while ground-truth injection holds the trajectory flat.Figure 4: LLM-judge score per rubric dimension, pooled over the nine models. Color encodes the dialogue history the model conditions on; hatching marks the in-context learning condition. Models score highly on callsign handling and conciseness but poorly on operational correctness.
Table 2: Per-model similarity to the C0 ground truth, by condition (ICL × history), averaged over prompts C1–C5 and 3 attempts. Best per column in bold.
ROUGE-L
Semantic cosine
Model
no ICL, self
no ICL, GT
ICL, self
ICL, GT
no ICL, self
no ICL, GT
ICL, self
ICL, GT
Open-source
Qwen2.5-7B
0.197
0.215
0.212
0.234
0.413
0.458
0.443
0.483
Llama-3.1-8B
0.193
0.181
0.208
0.203
0.437
0.403
0.452
0.418
Gemma-2-9B
0.188
0.218
0.216
0.248
0.405
0.441
0.438
0.487
Qwen2.5-14B
0.203
0.200
0.215
0.195
0.481
0.464
0.483
0.456
Qwen2.5-32B
0.215
0.213
0.240
0.234
0.460
0.453
0.476
0.474
Mixtral-8x7B
0.123
0.149
0.116
0.150
0.336
0.361
0.310
0.373
Average
0.186
0.196
0.201
0.211
0.422
0.430
0.434
0.449
Closed-source
GPT-5.4
0.246
0.236
0.255
0.247
0.470
0.450
0.470
0.449
GPT-5.4-mini
0.246
0.237
0.253
0.267
0.447
0.469
0.463
0.482
Claude-Sonnet-4.6
0.240
0.282
0.275
0.310
0.447
0.495
0.474
0.519
Average
0.244
0.252
0.261
0.275
0.455
0.471
0.469
0.483
Figure 5: Per-dimension scores from the LLM judge (a) and the human expert (b) on the same 180 annotated outputs, broken out by condition (color = dialogue history, hatching = in-context example). Both raters produce the same cross-dimension profile, peaking on callsign handling and conciseness and dipping on operational correctness, but the human sits about one point higher on every dimension. Because both panels use the identical 180-sample subset, the vertical offset between them is the genuine human–judge leniency gap.(b) Human expert
Table 3: Hypothesis tests for the three experimental contrasts, computed on per-model means across the nine models. Paired contrasts use an exact Wilcoxon signed-rank test (W); the closed-versus-open contrast uses a one-sided Mann–Whitney U test. Δ is reported in the direction named in each group heading, wins counts models favoring that direction, and the last column is the effect size (dz for the paired contrasts, Cohen’s d for the unpaired one). Bold indicates p<0.05.
Measure
Δ
Wins
Statistic
p
Effect
In-context example vs. none (paired, W, n=9)
ROUGE-L (pooled)
0.016
8/9
1.0
0.008
1.46
Normalized BLEU (pooled)
0.009
8/9
1.0
0.008
1.47
LLM-judge overall
0.036
5/9
13.0
0.547
0.25
Ground-truth vs. self dialogue history (paired, W, n=9)
ROUGE-L (pooled)
0.010
5/9
14.0
0.359
0.49
BLEU-1 (pooled)
0.044
9/9
0.0
0.004
1.98
ROUGE-L (prompt P1)
−0.003
4/9
20.0
0.820
−0.11
ROUGE-L (prompt P5)
0.057
9/9
0.0
0.004
1.72
LLM-judge overall
−0.078
3/9
14.5
0.371
−0.40
Closed- vs. open-source (unpaired, U, 3 vs. 6)
ROUGE-L
0.059
—
18.0
0.012
2.03
LLM-judge overall
1.050
—
18.0
0.012
4.43
Figure 6: Model-level agreement between each automatic similarity metric and the LLM judge. Each point is one model, plotted as its mean metric score against its mean judge rating, with a fitted line and 95% confidence band. ROUGE-L and TF-IDF track expert judgment closely; BLEU-1 and sentence-embedding similarity do not. The open/closed legend in (a) applies to all four panels.(b) TF-IDF cosine
Table 4: LLM-judge scores (1–5) per model, averaged over conditions and the top-25 judged outputs per cell. Best per column in bold.
Model
Phraseology
Correctness
Callsign
Conciseness
Safety
Halluc.-free
Overall
Open-source
Qwen2.5-7B
2.46
1.75
4.36
3.97
2.63
2.68
1.99
Llama-3.1-8B
2.40
1.74
3.65
3.88
2.51
2.22
1.89
Gemma-2-9B
2.80
1.94
4.14
4.34
2.80
2.37
2.12
Qwen2.5-14B
2.35
1.70
4.44
3.17
2.47
2.00
1.84
Qwen2.5-32B
2.81
2.00
4.29
3.83
2.83
2.33
2.20
Mixtral-8x7B
2.00
1.32
3.54
3.42
2.11
1.82
1.45
Average
2.47
1.74
4.07
3.77
2.56
2.24
1.91
Closed-source
GPT-5.4
3.84
2.86
4.24
4.78
3.81
3.77
3.13
GPT-5.4-mini
3.59
2.57
4.15
4.61
3.54
3.20
2.82
Claude-Sonnet-4.6
3.72
2.68
4.35
4.75
3.60
3.31
2.95
Average
3.71
2.71
4.25
4.71
3.65
3.43
2.96
(c) BLEU-1(d) Semantic cosine
Table 5: Spearman rank-correlation matrix over all six evaluation measures, computed on the 180 human-annotated outputs so that every measure is available for every observation. The two reference-overlap metrics (ROUGE-L, TF-IDF) are the only automatic measures that track either human-facing judgement.
ROUGE-L
TF-IDF
BLEU-1
Semantic
LLM judge
TF-IDF
0.73
BLEU-1
0.58
0.36
Semantic
0.32
0.47
0.30
LLM judge
0.53
0.49
0.02
0.10
Human
0.34
0.38
-0.05
0.09
0.67
Figure 7: Mean overall rating assigned by the LLM judge and by the human expert, split by the judge’s own score band (mean ± 95% CI). The human is systematically more lenient, and the gap grows as output quality falls.Figure 8: Per-model similarity by history mode (ICL on), grouped into open- and closed-source models. Bars are self history (hatched //) versus ground-truth history (hatched \\); panels are the four similarity metrics.
Table 6: Agreement between the three evaluation layers. Sample-level statistics treat each scored output as one observation; model-level statistics use per-model means (n=9). κw is quadratic-weighted Cohen’s kappa, defined only when both measures share the 1–5 ordinal scale.
Comparison
Level
Pearson r
Spearman ρ
κw
Automatic metric vs. LLM-judge overall (sample n=4,500)
ROUGE-L
sample
0.51
0.52
–
ROUGE-L
model
0.90
0.93
–
TF-IDF
sample
0.50
0.48
–
TF-IDF
model
0.91
0.90
–
BLEU-1
sample
0.12
0.06
–
BLEU-1
model
0.43
0.22
–
Semantic
sample
0.08
0.09
–
Semantic
model
0.67
0.50
–
Automatic metric vs. human overall (sample n=180)
ROUGE-L
sample
0.31
0.34
–
ROUGE-L
model
0.73
0.81
–
TF-IDF
sample
0.32
0.38
–
TF-IDF
model
0.72
0.79
–
BLEU-1
sample
0.02
-0.05
–
BLEU-1
model
0.17
-0.03
–
Semantic
sample
0.09
0.09
–
Semantic
model
0.39
0.28
–
LLM judge vs. human annotator (sample n=180)
Overall rating
sample
0.71
0.67
0.60
Overall rating
model
0.69
0.86
–
Table 7: Per-dimension and per-stratum agreement between the LLM judge and the human annotator (n=180). Δ is the human mean minus the judge mean; positive values mean the human is more lenient. Within-stratum correlations are attenuated by range restriction and κw is uninformative there, so it is reported only for the pooled sample.
Dimension
Judge mean
Human mean
Δ
Spearman ρ
κw
Phraseology
3.24
4.30
+1.06
0.45
0.36
Correctness
2.89
3.79
+0.90
0.63
0.52
Callsign
4.43
4.86
+0.43
0.35
0.43
Conciseness
4.34
4.51
+0.17
0.41
0.59
Safety
3.63
3.91
+0.28
0.53
0.59
Halluc.-free
3.61
3.96
+0.34
0.48
0.46
Overall
3.12
3.91
+0.79
0.67
0.60
Overall rating, split by judge stratum
Low (n=45)
1.13
2.58
+1.44
0.44
–
Mid (n=45)
2.87
3.78
+0.91
0.36
–
High (n=90)
4.24
4.64
+0.40
0.20
–
왜 중요한가
항공관제처럼 사람의 생명이 걸린 안전 필수 업무에 AI를 투입하려는 시도가 늘고 있는데, 이 연구는 현재 LLM이 '그럴듯하게 말하는 것'과 '올바르게 판단하는 것' 사이에 큰 간극이 있음을 구체적 수치로 보여준다. AI를 관제 보조, 훈련 시뮬레이터, 녹음 데이터 검토 등 사람을 대체하지 않는 보조 역할로 먼저 활용해야 한다는 실무적 시사점을 준다.
이 논문의 용어
LLM-as-judge · 사람 대신 또 다른 거대언어모델이 결과물의 품질을 채점하게 하는 평가 방식
인컨텍스트 러닝(ICL) · 모델에게 정답 예시를 프롬프트 안에 함께 넣어주어 참고하게 하는 기법
ROUGE-L, BLEU, TF-IDF · 생성된 문장이 정답 문장과 단어·순서 면에서 얼마나 비슷한지 재는 자동 채점 지표들
티처 포싱(teacher forcing) · 모델의 이전 답변 대신 실제 정답 대화를 강제로 넣어 다음 답을 생성하게 하는 방법
쿼드러틱 가중 코헨 카파(κw) · 두 채점자의 점수가 순위 척도에서 얼마나 일치하는지 재는 통계 지표
논문 원문 초록 (영문)
Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco "Bay Tour" route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.
저자 · Mahyar Ghazanfari, Matthias Casanova, Jordan Kam, Alex Zongo, Peng Wei, Torsten Darrell, Alexandre Bayen
