A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queries
arXiv:2608.198752026-08-21
환자가 증상을 다 말하지 않아 애매한 질문에, AI가 먼저 되물어서 답을 맞히는 방법
환자가 헬스케어 챗봇에 짧게 던지는 질문은 임신 여부, 복용 약, 알레르기 같은 개인 정보가 빠져 있어 정답이 여러 개일 수 있다. 연구진은 챗봇과 최종 답변 모델 사이에 지식그래프 기반 에이전트를 끼워 넣어, 부족한 정보를 콕 집어 되묻고 답을 모은 뒤 명확해진 질문을 최종 모델에 넘기는 방식을 만들었다. 5개 언어모델과 두 가지 벤치마크(증상 기반 진단 찾기, 식이 안전 판정)로 시험한 결과, 그냥 답하게 하거나 질문만 다시 표현시키는 것보다 정확도가 크게 올랐다.
무엇을 했나
환자 질문이 문법적으로는 멀쩡해도 임신, 알레르기, 복용약 같은 숨은 정보에 따라 정답이 달라지는 '환자 맥락 모호성' 문제를 정의했다
질문 해석 → 지식그래프로 가능한 후보(가설) 만들기 → 후보를 구분할 결정적 질문 고르기 → 환자 답변으로 후보 좁히기, 이 과정을 반복하는 에이전트를 설계했다 (Neo4j 지식그래프와 내부 언어모델, 규칙 기반 컨트롤러 조합)
증상 마스킹한 1,034건 진단 문제에서 에이전트는 정확 진단명 맞히기 정확도를 직접 답변 대비 최소 57.1퍼센트포인트, 상위5개 안에 정답 포함 비율을 최소 77.7퍼센트포인트 끌어올렸다
건강 맥락을 뺀 487건 식이 안전 판정에서도 5개 모델 전부 정확도가 올랐고, 5개 중 4개 모델에서 종합 성능 지표(MCC)가 가장 좋았다
같은 질문을 여러 번 반복시켜도 에이전트를 거친 답변이 그렇지 않은 경우보다 답이 훨씬 일관되게 나왔다(불확실성 감소)
Figure 1: Mitigating patient-context ambiguity before downstream language-model generation. a, In direct interaction with a healthcare chatbot, the downstream large language model (LLM) can use knowledge encoded during training and, when available, external evidence. However, decision-relevant patient-specific context may remain unavailable because it has not been elicited. The response is therefore generated from incomplete patient context. b, The knowledge-graph-guided ambiguity-mitigation framework operates between the patient-facing interface and an otherwise unchanged downstream LLM. The framework grounds the query in a task-specific knowledge graph, retrieves graph-supported hypotheses and selects the most informative clarification question. The patient’s answer is used to update the active hypotheses, and the cycle repeats until a stopping condition is met. The resulting clarified record is then passed to the downstream LLM, which generates a response informed by the elicited context.
Table 1: Illustrative effect of patient-specific context on an otherwise unchanged health query. The examples are conceptual and show why the same linguistically clear query may require different response strategies as additional patient context becomes available. They are not intended as treatment recommendations or as outputs from the evaluated models.
Patient query
Patient-specific context
Implication for the response
What should I take for a headache?
Not provided
Several common options may appear plausible, but the information needed to distinguish among them is absent.
What should I take for a headache?
Pregnancy
The response must account for pregnancy-related medication constraints and may differ from general over-the-counter guidance.
What should I take for a headache?
Use of an anticoagulant
The response must consider medication interactions and bleeding risk before suggesting an option.
What should I take for a headache?
Liver disease
The response must consider whether common medications are appropriate given impaired liver function.
What should I take for a headache?
Multiple relevant conditions or warning signs
A specific medication recommendation may be inappropriate, and the response may instead require escalation or professional assessment.
Figure 2: Between-model performance and variation under each configuration. Markers denote the mean across the five evaluated downstream language models, and horizontal error bars denote ±1 population standard deviation across the five model-specific values. Diagnosis metrics were computed over all 1,034 cases, with abstentions counted as incorrect, and dietary metrics were computed over all 487 queries. Higher values indicate better performance for category Top-1 accuracy, ICD Top-1 accuracy, exact Recall@5, and dietary MCC, whereas a lower value indicates better performance for the dietary false-negative rate, as shown by the arrows above each panel. For every displayed metric, the Agent attained both a better mean and a smaller between-model standard deviation than either baseline, indicating stronger average performance and less dependence on the selected downstream model. MCC is unitless; all other quantities are expressed in percentage points. Exact Top-1 diagnosis accuracy was omitted because its near-zero baseline means made dispersion alone misleading.
Table 2: Overview of the two ambiguity-mitigation evaluation sets. Both evaluation sets were curated so that the intended output could not be uniquely determined from the information initially provided. Missing patient-specific information was subsequently made available through the clarification process.
Symptom-based diagnosis retrieval
Dietary safety classification
Data source
Curated synthetic patient records
Curated food and health-condition resources
Evaluation instances
1,034
487
Output space
32 diagnosis labels
OK or Not OK
Information initially provided
Partial symptom profile
Food-related question without complete health context
Information initially withheld
Approximately 49.5% of the symptoms recorded in the complete case
Patient-specific health conditions relevant to the queried food
Information initially visible
Approximately 50.5% of the symptoms recorded in the complete case
Queried food and the user’s stated request
Reference information used during clarification
Complete symptom profile
Complete patient health context
Health conditions represented
–
38
Label distribution
32 diagnosis classes
301 OK and 186 Not OK
Primary evaluation measures
Category-level, ICD-level, and exact diagnosis retrieval
MCC, recall, precision, and classification error rates
Figure 3: Knowledge-graph-guided ambiguity-mitigation framework. a, The framework operates between the user-facing Interface and an unchanged downstream large language model (LLM). The Interface exchanges the initial query, clarification questions and user answers. The Orchestrator combines an inner LLM with a deterministic controller and accesses a task-specific knowledge graph implemented in Neo4j. After clarification, the Orchestrator passes the clarified query and retained graph evidence to the downstream LLM for response generation. b, The inner LLM interprets the query, and graph grounding retrieves the supported candidate hypotheses. When the available context is insufficient, the controller selects the eligible question with the greatest expected information gain. The inner LLM expresses the selected variable as a user-facing question, and the controller uses the answer to remove incompatible hypotheses. The cycle repeats until the remaining evidence supports a task output, no informative question remains or the interaction limit is reached. Orange denotes inner-LLM operations, blue denotes deterministic-controller operations, green denotes knowledge-graph operations and purple denotes the clarified output.
Table 3: Symptom-based diagnosis retrieval on the curated evaluation set (n=1,034). Performance was evaluated at three levels of specificity: broad diagnostic category, ICD-level grouping, and exact diagnosis name. Selective results include only cases for which the framework returned an answer. Overall results include all evaluation cases, with abstentions counted as incorrect. Because the Basic and RaR configurations did not abstain, their selective and overall values are identical. Bold values indicate the best configuration within each downstream model and metric column. Underlined values indicate the highest value in the corresponding column across all models and configurations. Category and ICD values rely on reference mappings that were not independently clinician-adjudicated, whereas exact-name matching does not require these mappings. All values are percentages.
Top-1, selective
Recall@5, selective
Top-1, overall
Recall@5, overall
Model
Configuration
Abstain
Category
ICD
Exact
Category
ICD
Exact
Category
ICD
Exact
Category
ICD
Exact
GPT-5.5
Agent
3.4
92.1
82.2
64.3
96.9
95.1
94.1
89.0
79.4
62.1
93.6
91.9
90.9
GPT-5.5
RaR
0.0
80.3
27.5
1.7
95.6
60.5
2.2
80.3
27.5
1.7
95.6
60.5
2.2
GPT-5.5
Basic
0.0
80.9
27.2
1.6
95.6
60.1
2.0
80.9
27.2
1.6
95.6
60.1
2.0
Claude Opus 4.8
Agent
3.4
94.0
81.6
72.6
96.7
94.9
93.9
90.8
78.8
70.1
93.4
91.7
90.7
Claude Opus 4.8
RaR
0.0
79.2
33.3
4.5
94.8
63.4
10.9
79.2
33.3
4.5
94.8
63.4
10.9
Claude Opus 4.8
Basic
0.0
75.2
36.3
1.5
92.9
67.2
4.0
75.2
36.3
1.5
92.9
67.2
4.0
Gemini 3.1 Pro
Agent
3.4
92.1
82.3
59.8
96.9
95.1
94.1
89.0
79.5
57.7
93.6
91.9
90.9
Gemini 3.1 Pro
RaR
0.0
81.2
29.3
0.8
96.6
56.3
3.3
81.2
29.3
0.8
96.6
56.3
3.3
Gemini 3.1 Pro
Basic
0.0
83.8
28.6
0.6
96.8
67.5
5.7
83.8
28.6
0.6
96.8
67.5
5.7
LLaMA 3.3 70B
Agent
3.4
92.2
83.4
73.6
97.2
94.9
93.9
89.1
80.6
71.1
93.9
91.7
90.7
LLaMA 3.3 70B
RaR
0.0
81.7
44.5
10.0
94.9
62.7
15.7
81.7
44.5
10.0
94.9
62.7
15.7
LLaMA 3.3 70B
Basic
0.0
81.1
45.9
11.0
93.9
62.1
16.2
81.1
45.9
11.0
93.9
62.1
16.2
Mistral Large
Agent
3.4
93.0
82.1
71.2
97.5
94.8
93.8
89.8
79.3
68.8
94.2
91.6
90.6
Mistral Large
RaR
0.0
76.3
37.1
7.4
94.7
66.4
12.3
76.3
37.1
7.4
94.7
66.4
12.3
Mistral Large
Basic
0.0
74.7
38.9
2.7
93.7
66.3
5.1
74.7
38.9
2.7
93.7
66.3
5.1
Table 4: Dietary safety classification on 487 queries. The positive class is Not OK. A false negative denotes an unsafe food classified as safe, whereas a false positive denotes an unnecessary restriction. MCC is reported because the dataset contains 301 OK and 186 Not OK queries. Bold values indicate the best result within each downstream-model block and metric column; ⋆ marks the best configuration for each model, ranked by MCC. Higher values are better for accuracy, MCC, F1, recall, and precision, whereas lower values are better for false-negative rate, false-positive rate, and false-negative count. Accuracy, recall, precision, and error rates are percentages.
Model
Configuration
Accuracy
MCC
F1
Recall
Precision
FN rate
FP rate
FN (n)
GPT-5.5
Agent ⋆
89.3
0.783
0.869
92.5
81.9
7.5
12.6
14
GPT-5.5
RaR
83.2
0.640
0.747
65.1
87.7
34.9
5.6
65
GPT-5.5
Basic
88.3
0.757
0.852
88.2
82.4
11.8
11.6
22
Claude Opus 4.8
Agent ⋆
91.8
0.837
0.900
96.8
84.1
3.2
11.3
6
Claude Opus 4.8
RaR
83.4
0.650
0.741
62.4
91.3
37.6
3.7
70
Claude Opus 4.8
Basic
81.3
0.607
0.760
77.4
74.6
22.6
16.3
42
Gemini 3.1 Pro
Agent
90.3
0.795
0.872
86.0
88.4
14.0
7.0
26
Gemini 3.1 Pro
RaR
88.3
0.751
0.846
83.9
85.2
16.1
9.0
30
Gemini 3.1 Pro
Basic ⋆
89.5
0.799
0.877
97.3
79.7
2.7
15.3
5
LLaMA 3.3 70B
Agent ⋆
86.9
0.750
0.848
95.7
76.1
4.3
18.6
8
LLaMA 3.3 70B
RaR
85.6
0.726
0.834
94.6
74.6
5.4
19.9
10
LLaMA 3.3 70B
Basic
83.8
0.701
0.818
95.7
71.5
4.3
23.6
8
Mistral Large
Agent ⋆
86.4
0.741
0.843
95.2
75.6
4.8
18.9
9
Mistral Large
RaR
84.2
0.718
0.826
98.4
71.2
1.6
24.6
3
Mistral Large
Basic
85.0
0.718
0.829
95.2
73.4
4.8
21.3
9
Table 5: Diagnosis-output uncertainty under repeated sampling. Two randomly selected diagnosis queries were evaluated using 500 independent repetitions for each model and configuration. K denotes the number of distinct exact diagnosis strings returned, Hexact denotes predictive entropy over exact diagnosis strings, and Hcat denotes entropy after mapping outputs to their clinical categories. Item 0 had the reference diagnosis acute viral pharyngitis, and Item 98 had the reference diagnosis ischemic heart disease. In the Agent condition, the symbolic clarification trajectory and terminal hypothesis were held fixed, so the repetitions resampled only downstream language generation. Outputs that could not be mapped to a clinical category were retained as distinct categories rather than excluded. Bold values identify the lowest-uncertainty configuration within each model block, and ⋆ marks that configuration. Lower values indicate less variation across repeated outputs. Entropy measures the dispersion of the generated answers and does not, by itself, indicate whether the resulting diagnosis was correct. Entropy is reported in bits.
Item 0: acute viral pharyngitis
Item 98: ischemic heart disease
Model
Configuration
K
Hexact
Hcat
K
Hexact
Hcat
GPT-5.5
Agent ⋆
1
0.000
0.000
1
0.000
0.000
GPT-5.5
RaR
1
0.000
0.000
6
1.006
0.988
GPT-5.5
Basic
1
0.000
0.000
5
1.232
1.128
Claude Opus 4.8
Agent ⋆
1
0.000
0.000
2
0.335
0.000
Claude Opus 4.8
RaR
11
2.092
0.000
9
2.044
0.511
Claude Opus 4.8
Basic
1
0.000
0.000
16
1.161
0.932
Gemini 3.1 Pro
Agent ⋆
1
0.000
0.000
2
0.094
0.094
Gemini 3.1 Pro
RaR
3
0.058
0.000
17
2.703
0.930
Gemini 3.1 Pro
Basic
2
0.053
0.000
15
1.968
1.056
LLaMA 3.3 70B
Agent ⋆
3
0.074
0.021
2
0.053
0.053
LLaMA 3.3 70B
RaR
6
1.897
0.000
7
1.208
0.205
LLaMA 3.3 70B
Basic
5
1.530
0.000
5
0.847
0.021
Mistral Large
Agent ⋆
1
0.000
0.000
3
1.145
0.000
Mistral Large
RaR
3
0.236
0.000
4
1.411
0.741
Mistral Large
Basic
1
0.000
0.000
3
0.344
0.067
Table 6: Dietary-safety uncertainty under repeated sampling. One Not OK-labeled query and one OK-labeled query were each evaluated using 500 independent repetitions per model and configuration, producing 1,000 binary decisions per row. The positive class was Not OK. Hunsafe and Hsafe denote predictive entropy for the Not OK- and OK-labeled queries, respectively. Zero bits indicates a fully consistent output, while one bit indicates an approximately even split between the two labels. In the Agent condition, the confirmed patient context and final safety decision were held fixed across repetitions, so the repetitions resampled only downstream language generation. Bold values indicate the best result within each model block and metric column. The ⋆ symbol marks the best-performing configuration for each model, ranked first by MCC, then by accuracy and mean entropy. Exact ties are marked for both configurations. Low entropy indicates consistency, not necessarily correctness. Accuracy, recall, precision, and false-positive rate are percentages; MCC is unitless.
Classification over 1,000 repetitions
Predictive entropy (bits)
Model
Configuration
Accuracy
MCC
Recall
Precision
FP rate
Hunsafe
Hsafe
Mean H
GPT-5.5
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
GPT-5.5
RaR
96.8
0.938
93.6
100.0
0.0
0.343
0.000
0.172
GPT-5.5
Basic ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Claude Opus 4.8
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Claude Opus 4.8
RaR
99.2
0.984
98.4
100.0
0.0
0.118
0.000
0.059
Claude Opus 4.8
Basic ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Gemini 3.1 Pro
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Gemini 3.1 Pro
RaR
99.8
0.996
99.6
100.0
0.0
0.038
0.000
0.019
Gemini 3.1 Pro
Basic
59.2
0.318
100.0
55.1
81.6
0.000
0.689
0.344
LLaMA 3.3 70B
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
LLaMA 3.3 70B
RaR
50.6
0.078
100.0
50.3
98.8
0.000
0.094
0.047
LLaMA 3.3 70B
Basic
50.0
0.000
100.0
50.0
100.0
0.000
0.000
0.000
Mistral Large
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Mistral Large
RaR
64.5
0.412
100.0
58.5
71.0
0.000
0.869
0.434
Mistral Large
Basic
76.1
0.594
100.0
67.7
47.8
0.000
0.999
0.499
Supplementary Table 1: Flow of synthetic patients and case–diagnosis instances into the diagnosis benchmark.
Stage or partition
Patients
Instances
Generated by Synthea
4,884
–
Excluded: no retained benchmark instance
366
–
Included in the frozen benchmark
4,518
10,374
Training partition
3,163
7,258
Validation partition
904
2,082
Test partition
451
1,034
Supplementary Table 2: DxSxKG artifact used in the reported evaluation.
Component
Count
Diagnosis nodes
129,815
Symptom nodes
340,842
HAS_SYMPTOM relations
82,813
IS_A relations
1,400
Supplementary Table 3: FoodSafetyKG components used in the reported evaluation.
Component
Count
USDA food-item nodes
2,085,340
Nutrient nodes
477
Food-phrase nodes
298
Ingredient nodes
130
Food-property nodes
60
Condition nodes
67
Condition-family nodes
16
Condition-alias nodes
138
HAS_NUTRIENT relations
27,094,028
Derived food-item HAS_PROPERTY relations
2,542,284
Materialized food-item risk relations
21,030,444
Curated food-phrase–ingredient relations
398
Curated ingredient–property relations
341
Property–condition RISKY_FOR relations
126
Property–condition SAFE_FOR relations
67
Supplementary Table 4: Model routing used for the frozen evaluation.
Display label
Developer
API provider
Requested identifier
GPT-5.5
OpenAI
OpenAI
gpt-5.5
Claude Opus 4.8
Anthropic
Anthropic
claude-opus-4-8
Gemini 3.1 Pro
Google
OpenRouter
google/gemini-3.1-pro-preview
Llama 3.3 70B Instruct
Meta
OpenRouter
meta-llama/llama-3.3-70b-instruct
Mistral Large
Mistral AI
Mistral
mistral-large-2512
Supplementary Table 5: Mental-health domain: example transformations produced by AIM masking, lexical generalization, and one illustrative vague user paraphrase. Latent indicates what is retained as hidden structure and is not shown to the user at inference time.
Mechanism
Seed (specific)
Surface after AIM
Latent
Severity masking
I’m feeling extremely anxious and having panic attacks for 2 weeks. I don’t know what to do.
I’m feeling anxious and having panic attacks for 2 weeks. I don’t know what to do.
full seed
Temporal masking
I’ve been depressed for 2 weeks.
I’ve been depressed for a while.
full seed
Euphemism mapping (risk-preserving vagueness)
I’ve been having thoughts of hurting myself.
I’ve been having thoughts.a
full seed
Diagnostic phrase removal
I was diagnosed with major depressive disorder.
I was diagnosed with.b
full seed
Ontological generalization (domain map + WordNet)
After masking: … and having panic attacks…
… and having mental condition… (domain hypernym map where applicable)
masked intermediate
LLM persona rewrite (illustrative)
(same seed as row 1, after rule-based steps)
“I’ve been kind of on edge and my body does that thing where it freaks out, it’s been going on for a bit.”
full seed + strategy tags
Supplementary Table 6: Food/dietary domain: examples of omission, abstraction, and generalization that force clarification of health context (e.g., diabetes, allergy), amounts, or ingredients. The Latent column names the hidden structure that must be recovered before a safety decision.
Mechanism
Seed (specific)
Surface after AIM
Latent
Disease/condition masking
Can I eat white rice with chicken curry? I have type 2 diabetes.
Can I eat white rice with chicken curry?
condition + safety label
Allergen masking
I need a recipe with 2 cups flour and chicken, but I have a peanut allergy.
I need a recipe with 2 cups flour and chicken, but.
allergen + full seed
Quantity removal
Add 2 cups of flour to the bowl.
Add some of flour to the bowl.
quantities + full seed
Ingredient omission
I need chicken, onions, and garlic for the stir-fry.
I need onions, and garlic for the stir-fry. (one ingredient dropped)
omitted slots + full seed
Generalization only
I want grilled salmon for dinner.
I want grilled fish for dinner.
full seed
LLM persona rewrite (illustrative)
(masked recipe-style request)
“I want to make that creamy spicy thing, maybe with some meat? I can’t remember what I bought.”
full seed + variables
Surface perturbation (illustrative)
Can I eat oatmeal with honey in the morning?
Can I eat oatmea with honney in the morning?
same as clean twin
왜 중요한가
챗봇 모델 자체를 바꾸거나 재학습하지 않고도, 중간에 질문을 던지는 계층만 추가해 의료 챗봇의 오답 위험을 줄일 수 있음을 보여준다. 이는 실제 서비스에 이미 쓰이는 상용 언어모델에 그대로 얹을 수 있는 구조라는 점에서 실용적 의미가 크다.
이 논문의 용어
지식그래프(Knowledge Graph) · 개념과 개념 사이의 관계를 노드와 연결선으로 정리한 구조화된 데이터베이스
환자 맥락 모호성(patient-context ambiguity) · 질문 문장은 명확하지만 환자 개인정보가 빠져서 정답이 여러 개가 될 수 있는 상태
Top-1 정확도 / Recall@5 · 모델이 내놓은 첫 번째 답이 맞는 비율 / 상위 5개 후보 안에 정답이 포함된 비율
매튜상관계수(MCC) · 이진 분류 성능을 정답과 오답 4가지 경우를 모두 반영해 종합적으로 평가하는 지표
예측 엔트로피(predictive entropy) · 같은 질문을 반복했을 때 답변이 얼마나 흩어져 나오는지를 재는 값, 낮을수록 일관됨
논문 원문 초록 (영문)
Patients often submit short, underspecified queries to healthcare chatbots that lack the patient-specific information needed to determine an appropriate response. Although these queries may be linguistically clear, they can support multiple plausible answers depending on undisclosed factors such as symptoms, diagnoses, medications, allergies, or dietary restrictions. A language model answering such a query directly may therefore rely on unsupported assumptions about the patient. We introduce a knowledge-guided agentic framework for mitigating patient-context ambiguity before final response generation. The framework operates between the patient and an otherwise unchanged downstream language model. It interprets the initial query, uses a task-specific knowledge graph to construct a set of plausible hypotheses, identifies the missing patient-context variables needed to distinguish among them, and asks targeted follow-up questions. The original query and the acquired context are then combined into a clarified prompt for the downstream model. We evaluated the framework across five language models using two controlled ambiguity-mitigation benchmarks: diagnosis retrieval from 1,034 symptom queries with clinically relevant evidence systematically masked, and dietary-safety classification from 487 queries with decisive health context omitted. The framework was compared with direct answering of the underspecified query and with rephrasing the same query without acquiring new patient information. In diagnosis retrieval, it increased overall exact Top-1 accuracy by at least 57.1 percentage points and selective exact Recall@5 by at least 77.7 percentage points across the five evaluated models compared with direct prompting. In dietary-safety classification, it improved accuracy across all five models and achieved the highest Matthews correlation coefficient for four...
저자 · Mahyar Abbasian, Saba A. Farahani, Arshia Ilaty, Hung Cao, Ramesh Jain, Amir M. Rahmani
