A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queries
arXiv:2608.198752026-08-21
Getting AI to ask the right follow-up questions before answering vague health queries
Short health chatbot queries often hide personal details like pregnancy, medications, or allergies, so the same question can have different correct answers for different patients. The researchers built an agent that sits between the patient and the answering language model, using a knowledge graph to figure out what's missing, ask a targeted follow-up question, and then hand a clarified prompt to the final model. Tested across five language models on two benchmarks (symptom-based diagnosis and dietary-safety classification), this clarification step produced large accuracy gains over answering directly or just rephrasing the question.
What they did
Defined 'patient-context ambiguity': queries that are grammatically clear but whose correct answer depends on undisclosed personal facts such as pregnancy, allergies, or medications
Built an agent that interprets the query, builds candidate hypotheses from a task-specific knowledge graph (implemented in Neo4j), picks the most informative missing-information question, and repeats until enough evidence is gathered, combining an inner language model with a rule-based controller
On 1,034 diagnosis queries with key symptoms masked, the agent raised exact Top-1 diagnosis accuracy by at least 57.1 percentage points and exact Recall@5 by at least 77.7 percentage points versus direct answering, across five models
On 487 dietary-safety queries missing critical health context, the agent improved accuracy for all five models and got the best Matthews correlation coefficient (MCC) for four of them
When the same query was repeated many times, agent-clarified answers were far more consistent (lower entropy) than answers from direct or rephrased prompting
Figure 1: Mitigating patient-context ambiguity before downstream language-model generation. a, In direct interaction with a healthcare chatbot, the downstream large language model (LLM) can use knowledge encoded during training and, when available, external evidence. However, decision-relevant patient-specific context may remain unavailable because it has not been elicited. The response is therefore generated from incomplete patient context. b, The knowledge-graph-guided ambiguity-mitigation framework operates between the patient-facing interface and an otherwise unchanged downstream LLM. The framework grounds the query in a task-specific knowledge graph, retrieves graph-supported hypotheses and selects the most informative clarification question. The patient’s answer is used to update the active hypotheses, and the cycle repeats until a stopping condition is met. The resulting clarified record is then passed to the downstream LLM, which generates a response informed by the elicited context.
Table 1: Illustrative effect of patient-specific context on an otherwise unchanged health query. The examples are conceptual and show why the same linguistically clear query may require different response strategies as additional patient context becomes available. They are not intended as treatment recommendations or as outputs from the evaluated models.
Patient query
Patient-specific context
Implication for the response
What should I take for a headache?
Not provided
Several common options may appear plausible, but the information needed to distinguish among them is absent.
What should I take for a headache?
Pregnancy
The response must account for pregnancy-related medication constraints and may differ from general over-the-counter guidance.
What should I take for a headache?
Use of an anticoagulant
The response must consider medication interactions and bleeding risk before suggesting an option.
What should I take for a headache?
Liver disease
The response must consider whether common medications are appropriate given impaired liver function.
What should I take for a headache?
Multiple relevant conditions or warning signs
A specific medication recommendation may be inappropriate, and the response may instead require escalation or professional assessment.
Figure 2: Between-model performance and variation under each configuration. Markers denote the mean across the five evaluated downstream language models, and horizontal error bars denote ±1 population standard deviation across the five model-specific values. Diagnosis metrics were computed over all 1,034 cases, with abstentions counted as incorrect, and dietary metrics were computed over all 487 queries. Higher values indicate better performance for category Top-1 accuracy, ICD Top-1 accuracy, exact Recall@5, and dietary MCC, whereas a lower value indicates better performance for the dietary false-negative rate, as shown by the arrows above each panel. For every displayed metric, the Agent attained both a better mean and a smaller between-model standard deviation than either baseline, indicating stronger average performance and less dependence on the selected downstream model. MCC is unitless; all other quantities are expressed in percentage points. Exact Top-1 diagnosis accuracy was omitted because its near-zero baseline means made dispersion alone misleading.
Table 2: Overview of the two ambiguity-mitigation evaluation sets. Both evaluation sets were curated so that the intended output could not be uniquely determined from the information initially provided. Missing patient-specific information was subsequently made available through the clarification process.
Symptom-based diagnosis retrieval
Dietary safety classification
Data source
Curated synthetic patient records
Curated food and health-condition resources
Evaluation instances
1,034
487
Output space
32 diagnosis labels
OK or Not OK
Information initially provided
Partial symptom profile
Food-related question without complete health context
Information initially withheld
Approximately 49.5% of the symptoms recorded in the complete case
Patient-specific health conditions relevant to the queried food
Information initially visible
Approximately 50.5% of the symptoms recorded in the complete case
Queried food and the user’s stated request
Reference information used during clarification
Complete symptom profile
Complete patient health context
Health conditions represented
–
38
Label distribution
32 diagnosis classes
301 OK and 186 Not OK
Primary evaluation measures
Category-level, ICD-level, and exact diagnosis retrieval
MCC, recall, precision, and classification error rates
Figure 3: Knowledge-graph-guided ambiguity-mitigation framework. a, The framework operates between the user-facing Interface and an unchanged downstream large language model (LLM). The Interface exchanges the initial query, clarification questions and user answers. The Orchestrator combines an inner LLM with a deterministic controller and accesses a task-specific knowledge graph implemented in Neo4j. After clarification, the Orchestrator passes the clarified query and retained graph evidence to the downstream LLM for response generation. b, The inner LLM interprets the query, and graph grounding retrieves the supported candidate hypotheses. When the available context is insufficient, the controller selects the eligible question with the greatest expected information gain. The inner LLM expresses the selected variable as a user-facing question, and the controller uses the answer to remove incompatible hypotheses. The cycle repeats until the remaining evidence supports a task output, no informative question remains or the interaction limit is reached. Orange denotes inner-LLM operations, blue denotes deterministic-controller operations, green denotes knowledge-graph operations and purple denotes the clarified output.
Table 3: Symptom-based diagnosis retrieval on the curated evaluation set (n=1,034). Performance was evaluated at three levels of specificity: broad diagnostic category, ICD-level grouping, and exact diagnosis name. Selective results include only cases for which the framework returned an answer. Overall results include all evaluation cases, with abstentions counted as incorrect. Because the Basic and RaR configurations did not abstain, their selective and overall values are identical. Bold values indicate the best configuration within each downstream model and metric column. Underlined values indicate the highest value in the corresponding column across all models and configurations. Category and ICD values rely on reference mappings that were not independently clinician-adjudicated, whereas exact-name matching does not require these mappings. All values are percentages.
Top-1, selective
Recall@5, selective
Top-1, overall
Recall@5, overall
Model
Configuration
Abstain
Category
ICD
Exact
Category
ICD
Exact
Category
ICD
Exact
Category
ICD
Exact
GPT-5.5
Agent
3.4
92.1
82.2
64.3
96.9
95.1
94.1
89.0
79.4
62.1
93.6
91.9
90.9
GPT-5.5
RaR
0.0
80.3
27.5
1.7
95.6
60.5
2.2
80.3
27.5
1.7
95.6
60.5
2.2
GPT-5.5
Basic
0.0
80.9
27.2
1.6
95.6
60.1
2.0
80.9
27.2
1.6
95.6
60.1
2.0
Claude Opus 4.8
Agent
3.4
94.0
81.6
72.6
96.7
94.9
93.9
90.8
78.8
70.1
93.4
91.7
90.7
Claude Opus 4.8
RaR
0.0
79.2
33.3
4.5
94.8
63.4
10.9
79.2
33.3
4.5
94.8
63.4
10.9
Claude Opus 4.8
Basic
0.0
75.2
36.3
1.5
92.9
67.2
4.0
75.2
36.3
1.5
92.9
67.2
4.0
Gemini 3.1 Pro
Agent
3.4
92.1
82.3
59.8
96.9
95.1
94.1
89.0
79.5
57.7
93.6
91.9
90.9
Gemini 3.1 Pro
RaR
0.0
81.2
29.3
0.8
96.6
56.3
3.3
81.2
29.3
0.8
96.6
56.3
3.3
Gemini 3.1 Pro
Basic
0.0
83.8
28.6
0.6
96.8
67.5
5.7
83.8
28.6
0.6
96.8
67.5
5.7
LLaMA 3.3 70B
Agent
3.4
92.2
83.4
73.6
97.2
94.9
93.9
89.1
80.6
71.1
93.9
91.7
90.7
LLaMA 3.3 70B
RaR
0.0
81.7
44.5
10.0
94.9
62.7
15.7
81.7
44.5
10.0
94.9
62.7
15.7
LLaMA 3.3 70B
Basic
0.0
81.1
45.9
11.0
93.9
62.1
16.2
81.1
45.9
11.0
93.9
62.1
16.2
Mistral Large
Agent
3.4
93.0
82.1
71.2
97.5
94.8
93.8
89.8
79.3
68.8
94.2
91.6
90.6
Mistral Large
RaR
0.0
76.3
37.1
7.4
94.7
66.4
12.3
76.3
37.1
7.4
94.7
66.4
12.3
Mistral Large
Basic
0.0
74.7
38.9
2.7
93.7
66.3
5.1
74.7
38.9
2.7
93.7
66.3
5.1
Table 4: Dietary safety classification on 487 queries. The positive class is Not OK. A false negative denotes an unsafe food classified as safe, whereas a false positive denotes an unnecessary restriction. MCC is reported because the dataset contains 301 OK and 186 Not OK queries. Bold values indicate the best result within each downstream-model block and metric column; ⋆ marks the best configuration for each model, ranked by MCC. Higher values are better for accuracy, MCC, F1, recall, and precision, whereas lower values are better for false-negative rate, false-positive rate, and false-negative count. Accuracy, recall, precision, and error rates are percentages.
Model
Configuration
Accuracy
MCC
F1
Recall
Precision
FN rate
FP rate
FN (n)
GPT-5.5
Agent ⋆
89.3
0.783
0.869
92.5
81.9
7.5
12.6
14
GPT-5.5
RaR
83.2
0.640
0.747
65.1
87.7
34.9
5.6
65
GPT-5.5
Basic
88.3
0.757
0.852
88.2
82.4
11.8
11.6
22
Claude Opus 4.8
Agent ⋆
91.8
0.837
0.900
96.8
84.1
3.2
11.3
6
Claude Opus 4.8
RaR
83.4
0.650
0.741
62.4
91.3
37.6
3.7
70
Claude Opus 4.8
Basic
81.3
0.607
0.760
77.4
74.6
22.6
16.3
42
Gemini 3.1 Pro
Agent
90.3
0.795
0.872
86.0
88.4
14.0
7.0
26
Gemini 3.1 Pro
RaR
88.3
0.751
0.846
83.9
85.2
16.1
9.0
30
Gemini 3.1 Pro
Basic ⋆
89.5
0.799
0.877
97.3
79.7
2.7
15.3
5
LLaMA 3.3 70B
Agent ⋆
86.9
0.750
0.848
95.7
76.1
4.3
18.6
8
LLaMA 3.3 70B
RaR
85.6
0.726
0.834
94.6
74.6
5.4
19.9
10
LLaMA 3.3 70B
Basic
83.8
0.701
0.818
95.7
71.5
4.3
23.6
8
Mistral Large
Agent ⋆
86.4
0.741
0.843
95.2
75.6
4.8
18.9
9
Mistral Large
RaR
84.2
0.718
0.826
98.4
71.2
1.6
24.6
3
Mistral Large
Basic
85.0
0.718
0.829
95.2
73.4
4.8
21.3
9
Table 5: Diagnosis-output uncertainty under repeated sampling. Two randomly selected diagnosis queries were evaluated using 500 independent repetitions for each model and configuration. K denotes the number of distinct exact diagnosis strings returned, Hexact denotes predictive entropy over exact diagnosis strings, and Hcat denotes entropy after mapping outputs to their clinical categories. Item 0 had the reference diagnosis acute viral pharyngitis, and Item 98 had the reference diagnosis ischemic heart disease. In the Agent condition, the symbolic clarification trajectory and terminal hypothesis were held fixed, so the repetitions resampled only downstream language generation. Outputs that could not be mapped to a clinical category were retained as distinct categories rather than excluded. Bold values identify the lowest-uncertainty configuration within each model block, and ⋆ marks that configuration. Lower values indicate less variation across repeated outputs. Entropy measures the dispersion of the generated answers and does not, by itself, indicate whether the resulting diagnosis was correct. Entropy is reported in bits.
Item 0: acute viral pharyngitis
Item 98: ischemic heart disease
Model
Configuration
K
Hexact
Hcat
K
Hexact
Hcat
GPT-5.5
Agent ⋆
1
0.000
0.000
1
0.000
0.000
GPT-5.5
RaR
1
0.000
0.000
6
1.006
0.988
GPT-5.5
Basic
1
0.000
0.000
5
1.232
1.128
Claude Opus 4.8
Agent ⋆
1
0.000
0.000
2
0.335
0.000
Claude Opus 4.8
RaR
11
2.092
0.000
9
2.044
0.511
Claude Opus 4.8
Basic
1
0.000
0.000
16
1.161
0.932
Gemini 3.1 Pro
Agent ⋆
1
0.000
0.000
2
0.094
0.094
Gemini 3.1 Pro
RaR
3
0.058
0.000
17
2.703
0.930
Gemini 3.1 Pro
Basic
2
0.053
0.000
15
1.968
1.056
LLaMA 3.3 70B
Agent ⋆
3
0.074
0.021
2
0.053
0.053
LLaMA 3.3 70B
RaR
6
1.897
0.000
7
1.208
0.205
LLaMA 3.3 70B
Basic
5
1.530
0.000
5
0.847
0.021
Mistral Large
Agent ⋆
1
0.000
0.000
3
1.145
0.000
Mistral Large
RaR
3
0.236
0.000
4
1.411
0.741
Mistral Large
Basic
1
0.000
0.000
3
0.344
0.067
Table 6: Dietary-safety uncertainty under repeated sampling. One Not OK-labeled query and one OK-labeled query were each evaluated using 500 independent repetitions per model and configuration, producing 1,000 binary decisions per row. The positive class was Not OK. Hunsafe and Hsafe denote predictive entropy for the Not OK- and OK-labeled queries, respectively. Zero bits indicates a fully consistent output, while one bit indicates an approximately even split between the two labels. In the Agent condition, the confirmed patient context and final safety decision were held fixed across repetitions, so the repetitions resampled only downstream language generation. Bold values indicate the best result within each model block and metric column. The ⋆ symbol marks the best-performing configuration for each model, ranked first by MCC, then by accuracy and mean entropy. Exact ties are marked for both configurations. Low entropy indicates consistency, not necessarily correctness. Accuracy, recall, precision, and false-positive rate are percentages; MCC is unitless.
Classification over 1,000 repetitions
Predictive entropy (bits)
Model
Configuration
Accuracy
MCC
Recall
Precision
FP rate
Hunsafe
Hsafe
Mean H
GPT-5.5
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
GPT-5.5
RaR
96.8
0.938
93.6
100.0
0.0
0.343
0.000
0.172
GPT-5.5
Basic ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Claude Opus 4.8
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Claude Opus 4.8
RaR
99.2
0.984
98.4
100.0
0.0
0.118
0.000
0.059
Claude Opus 4.8
Basic ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Gemini 3.1 Pro
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Gemini 3.1 Pro
RaR
99.8
0.996
99.6
100.0
0.0
0.038
0.000
0.019
Gemini 3.1 Pro
Basic
59.2
0.318
100.0
55.1
81.6
0.000
0.689
0.344
LLaMA 3.3 70B
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
LLaMA 3.3 70B
RaR
50.6
0.078
100.0
50.3
98.8
0.000
0.094
0.047
LLaMA 3.3 70B
Basic
50.0
0.000
100.0
50.0
100.0
0.000
0.000
0.000
Mistral Large
Agent ⋆
100.0
1.000
100.0
100.0
0.0
0.000
0.000
0.000
Mistral Large
RaR
64.5
0.412
100.0
58.5
71.0
0.000
0.869
0.434
Mistral Large
Basic
76.1
0.594
100.0
67.7
47.8
0.000
0.999
0.499
Supplementary Table 1: Flow of synthetic patients and case–diagnosis instances into the diagnosis benchmark.
Stage or partition
Patients
Instances
Generated by Synthea
4,884
–
Excluded: no retained benchmark instance
366
–
Included in the frozen benchmark
4,518
10,374
Training partition
3,163
7,258
Validation partition
904
2,082
Test partition
451
1,034
Supplementary Table 2: DxSxKG artifact used in the reported evaluation.
Component
Count
Diagnosis nodes
129,815
Symptom nodes
340,842
HAS_SYMPTOM relations
82,813
IS_A relations
1,400
Supplementary Table 3: FoodSafetyKG components used in the reported evaluation.
Component
Count
USDA food-item nodes
2,085,340
Nutrient nodes
477
Food-phrase nodes
298
Ingredient nodes
130
Food-property nodes
60
Condition nodes
67
Condition-family nodes
16
Condition-alias nodes
138
HAS_NUTRIENT relations
27,094,028
Derived food-item HAS_PROPERTY relations
2,542,284
Materialized food-item risk relations
21,030,444
Curated food-phrase–ingredient relations
398
Curated ingredient–property relations
341
Property–condition RISKY_FOR relations
126
Property–condition SAFE_FOR relations
67
Supplementary Table 4: Model routing used for the frozen evaluation.
Display label
Developer
API provider
Requested identifier
GPT-5.5
OpenAI
OpenAI
gpt-5.5
Claude Opus 4.8
Anthropic
Anthropic
claude-opus-4-8
Gemini 3.1 Pro
Google
OpenRouter
google/gemini-3.1-pro-preview
Llama 3.3 70B Instruct
Meta
OpenRouter
meta-llama/llama-3.3-70b-instruct
Mistral Large
Mistral AI
Mistral
mistral-large-2512
Supplementary Table 5: Mental-health domain: example transformations produced by AIM masking, lexical generalization, and one illustrative vague user paraphrase. Latent indicates what is retained as hidden structure and is not shown to the user at inference time.
Mechanism
Seed (specific)
Surface after AIM
Latent
Severity masking
I’m feeling extremely anxious and having panic attacks for 2 weeks. I don’t know what to do.
I’m feeling anxious and having panic attacks for 2 weeks. I don’t know what to do.
full seed
Temporal masking
I’ve been depressed for 2 weeks.
I’ve been depressed for a while.
full seed
Euphemism mapping (risk-preserving vagueness)
I’ve been having thoughts of hurting myself.
I’ve been having thoughts.a
full seed
Diagnostic phrase removal
I was diagnosed with major depressive disorder.
I was diagnosed with.b
full seed
Ontological generalization (domain map + WordNet)
After masking: … and having panic attacks…
… and having mental condition… (domain hypernym map where applicable)
masked intermediate
LLM persona rewrite (illustrative)
(same seed as row 1, after rule-based steps)
“I’ve been kind of on edge and my body does that thing where it freaks out, it’s been going on for a bit.”
full seed + strategy tags
Supplementary Table 6: Food/dietary domain: examples of omission, abstraction, and generalization that force clarification of health context (e.g., diabetes, allergy), amounts, or ingredients. The Latent column names the hidden structure that must be recovered before a safety decision.
Mechanism
Seed (specific)
Surface after AIM
Latent
Disease/condition masking
Can I eat white rice with chicken curry? I have type 2 diabetes.
Can I eat white rice with chicken curry?
condition + safety label
Allergen masking
I need a recipe with 2 cups flour and chicken, but I have a peanut allergy.
I need a recipe with 2 cups flour and chicken, but.
allergen + full seed
Quantity removal
Add 2 cups of flour to the bowl.
Add some of flour to the bowl.
quantities + full seed
Ingredient omission
I need chicken, onions, and garlic for the stir-fry.
I need onions, and garlic for the stir-fry. (one ingredient dropped)
omitted slots + full seed
Generalization only
I want grilled salmon for dinner.
I want grilled fish for dinner.
full seed
LLM persona rewrite (illustrative)
(masked recipe-style request)
“I want to make that creamy spicy thing, maybe with some meat? I can’t remember what I bought.”
full seed + variables
Surface perturbation (illustrative)
Can I eat oatmeal with honey in the morning?
Can I eat oatmea with honney in the morning?
same as clean twin
Why it matters
This shows a way to cut down risky guesswork in health chatbots without retraining or modifying the underlying language model, just by adding a question-asking layer in front of it. That makes it practically deployable on top of existing commercial models already used in real chatbot products.
Terms in this paper
Knowledge graph · a structured database of concepts and the relationships connecting them, represented as nodes and links
Patient-context ambiguity · a situation where a query's wording is clear but its correct answer depends on undisclosed patient-specific information
Top-1 accuracy / Recall@5 · how often the model's single best answer is correct / how often the correct answer appears among its top five guesses
Matthews correlation coefficient (MCC) · a summary score for binary classification that accounts for all four outcome types (true/false positives/negatives)
Predictive entropy · a measure of how scattered repeated answers to the same question are; lower means more consistent
Original abstract (English)
Patients often submit short, underspecified queries to healthcare chatbots that lack the patient-specific information needed to determine an appropriate response. Although these queries may be linguistically clear, they can support multiple plausible answers depending on undisclosed factors such as symptoms, diagnoses, medications, allergies, or dietary restrictions. A language model answering such a query directly may therefore rely on unsupported assumptions about the patient. We introduce a knowledge-guided agentic framework for mitigating patient-context ambiguity before final response generation. The framework operates between the patient and an otherwise unchanged downstream language model. It interprets the initial query, uses a task-specific knowledge graph to construct a set of plausible hypotheses, identifies the missing patient-context variables needed to distinguish among them, and asks targeted follow-up questions. The original query and the acquired context are then combined into a clarified prompt for the downstream model. We evaluated the framework across five language models using two controlled ambiguity-mitigation benchmarks: diagnosis retrieval from 1,034 symptom queries with clinically relevant evidence systematically masked, and dietary-safety classification from 487 queries with decisive health context omitted. The framework was compared with direct answering of the underspecified query and with rephrasing the same query without acquiring new patient information. In diagnosis retrieval, it increased overall exact Top-1 accuracy by at least 57.1 percentage points and selective exact Recall@5 by at least 77.7 percentage points across the five evaluated models compared with direct prompting. In dietary-safety classification, it improved accuracy across all five models and achieved the highest Matthews correlation coefficient for four...
Authors · Mahyar Abbasian, Saba A. Farahani, Arshia Ilaty, Hung Cao, Ramesh Jain, Amir M. Rahmani
