Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents

arXiv:2608.197942026-08-21

말만 잘하는 AI를 몸을 가진 로봇으로 만들려면 무엇이 더 필요한가를 정리한 서베이 논문

이 논문은 대형언어모델(LLM), 지식베이스, 추론능력, 로봇 같은 물리적 몸체(임바디드 지능)를 결합해 진짜 범용 지능형 로봇을 만드는 길을 정리한 리뷰 논문이다. 저자들은 기존 연구들을 표로 정리하며 각 분야의 성과와 한계를 짚고, 이 네 요소를 하나로 묶는 개념적 틀을 제시했다. 실제로 구현된 로봇 시스템이 아니라, 앞으로 나아가야 할 방향을 보여주는 지도 역할을 하는 논문이다.

무엇을 했나

GPT-4, PaLM, Gemini, Claude 같은 LLM은 언어는 잘하지만 물리 세계를 직접 경험하지 못해 '몸이 없는' 한계를 가진다는 문제의식에서 출발한다
LLM의 구조와 학습 방식, 지식베이스 연결 방법, 추론 프레임워크(예: 단계별로 생각을 풀어내는 Chain-of-Thought, 여러 갈래로 생각을 뻗는 Tree-of-Thought), 로봇이 실제 환경에서 움직이고 배우는 임바디드 지능 연구를 각각 정리했다
이 네 가지를 하나로 합친 개념적 프레임워크를 제시해 인식-추론-행동이 어떻게 연결되는지 보여주지만, 실제로 작동하는 엔지니어링 시스템은 아니다
앞으로 풀어야 할 다섯 가지 과제로 LLM을 가볍게 만들어 로봇에 실을 수 있게 하는 경량화 배포, 지식을 실시간으로 갱신하는 폐쇄루프 지식통합, 규칙 기반과 신경망 기반을 섞는 하이브리드 추론, 언어 명령을 실제 동작으로 연결하는 인식-행동 접지, 잊어버리지 않고 계속 배우는 지속학습을 꼽았다
기존 리뷰 논문들과 비교해 언어모델·지식·추론·몸체 네 요소를 통합적으로 다룬 최초의 체계적 로드맵이라고 주장한다

Fig. 1: Schematic diagram of a unified multimodal large language model architecture.

TABLE I: Comparative Summary of Related Embodied Intelligence Review Articles
Authors	Year	Core Contributions	Limitations
Yao et al. [151]	2025	Reviews multimodal embodied intelligence tasks and models; explores LLM/MLM integration with robots; proposes PPA paradigm and evaluates MLMs.	Classical AI lacks physical interaction; DRL is data-hungry and less robust; VLN training is costly and generalization is poor; LLMs lack causal reasoning; Sim2Real transfer is weak; evaluation standards are lacking.
Dong et al. [135]	2025	Analyzes LLM-enabled Human-Robot Symbiotic Manufacturing (HRSM), including interaction, collaboration, and execution.	Privacy and data sensitivity risks; hallucination and bias in LLMs; absence of unified evaluation standards; integration difficulties in diverse environments; token length limits document processing.
Tan et al. [70]	2025	Proposes CEIM to fuse multi-source data and LLM reasoning for semantic integration and decision-making in customized manufacturing.	Data fusion issues; low semantic alignment; real-time control accuracy is lacking; high computation cost; insufficient standardization of AI-device interfaces.
Jeong et al. [59]	2024	Surveys five directions of LLM/VLM in robotics: reward design, control, planning, manipulation, and scene understanding.	Limited by embedded compute; weak sensor fusion; prompt bias and poor interpretability; lacks domain-specific knowledge; Sim2Real generalization is weak.
Wang et al. [68]	2025	Discusses LLMs in robotics tasks and evaluates GPT-4V; explores future applications in agriculture and medicine.	Plan robustness is lacking; long prompts require expert input; fixed action sets reduce flexibility; API is closed-source; RGBD-based 3D reasoning is limited.
Our Survey	2025	Systematically analyzed the synergy among LLMs, knowledge bases, reasoning capabilities, and embodied intelligence; proposed a unified framework and five future directions for GEI.	Limited empirical examples; insufficient discussion on multilingual settings and real-world deployment.

Fig. 2: Evolution and Taxonomy of Large Language Models.

TABLE II: Comparison of Research on Large Language Models (LLMs)
Author(s)	Core Improvement	Advantages	Application Scenarios
Cummins et al. [25]	Compiler optimization framework	Enhances disassembly processes	Compiler optimization
Hu et al. [174]	Prompting strategies incorporating context to enhance VQA performance	Strengthens contextual understanding	Knowledge-based applications
Hu et al. [166]	LLM-TIKG: Threat Intelligence Knowledge Graph framework	Enhances cybersecurity analysis	Threat Intelligence Analysis
Qian et al. [80]	Infrastructure for large-scale LLM training	Efficiently handles large-scale training tasks	LLM training infrastructure
Sun et al. [121]	UniCoder: Intermediate representation to improve downstream NLP task processing	Enhances the model’s ability to handle complex tasks	Model scaling
Zhang et al. [156]	BB-GeoGPT: Framework tailored for Geographic Information Science	Highly adaptable to domain-specific knowledge	Geographic Information Science (GIS)
Wang et al. [139]	VideoAgent: Vision-language integrated system for long-form video interpretation	Improves multimodal reasoning capabilities	Multimodal reasoning
Xia et al. [19]	LLMGA: LLM-based generation assistant combining image generation and editing tools	Precise control over image synthesis,	Image generation and editing
Zhou et al. [172]	Weak-to-Strong Search: Greedy search for aligning large models with smaller counterparts	Compute-efficient, enhances model generalization	Model alignment and fine-tuning
Land et al. [110]	Glitch token analysis and detection method	Improves tokenizer robustness	tokenizer optimization

Fig. 3: Schematic architecture of the Transformer model.

TABLE III: Summary of Key Contributions in Embodied Intelligence Research
Author	Year	Research Topic	Key Contribution	Application Domain
Zardini et al. [46]	2020	Co-Design Framework	Structured co-design for integration	General systems
Ma et al. [18]	2021	Soft Robotics Actuators	Integrated sensing & actuation	Soft robotics
Gupta et al. [4]	2021	DERL for Adaptation	Co-evolution of body and control	Locomotion & control
Roy et al. [96]	2021	ML in Embodied Systems	Energy-aware physical learning	Robotics & ML
Mengaldo et al. [45]	2022	Physics Modeling	Physics-based modeling	Soft robotics
Iida et al. [43]	2022	Temporal Dynamics	Timescale-based self-organization	Adaptive systems
Long et al. [158]	2023	Human-in-the-loop Learning	Interactive simulation for training	Surgical robotics
Zeng et al. [39]	2023	LLMs for Robotics	Survey of LLM-enhanced robotics	Human-robot interaction
Fan et al. [54]	2024	LLMs in Industry	LLM-based autonomous control	Industrial robots
Feng et al. [69]	2025	Spatial AI Reasoning	Spatial AI via LLMs	Multiscale AI systems

Fig. 4: Construction Process of Large Language Models.

TABLE IV: LLM and Knowledge Graph Integration
Author(s)	Innovation	Advantages	Drawbacks
Daniel et al. [30]	Verifying RDF triples using LLMs by comparing RDF statements to external documents	High precision (88%) in RDF verification	Requires human oversight, method may be slower
Feng et al. [155]	Knowledge Graph–based thought framework to improve LLM responses with verified KG information	Reduces factual errors, enhances drug–cancer associations	Limited to biomedical, requires cross-domain testing
Dai et al. [140]	Empirical study on LLMs’ comprehension of KG input formats and prompt organization	Improves LLM performance in KG-based tasks	Varying effectiveness across different models, noisy subgraphs
Erickson et al. [75]	Enhancing LLM workflows with RAG and KGs to ensure traceability and flexibility	More accurate and explainable LLM outputs	Integration with external tools can be complex
Xiang et al. [138]	LLM-based generative retrieval framework focusing on trustworthy search results	Improves relevance, responsibility, and trustworthiness	Works better with certain query types, not universally applicable
Sequeda et al. [77]	LLM-powered enterprise question answering, supported by KGs	Provides formal validation and trusted data for queries	May not scale well for large or complex queries in diverse enterprises

Fig. 5: Description of the execution process of the intelligent document processing and knowledge question answering system.

TABLE V: LLM for Domain-Specific Applications
Authors	Innovation	Advantages	Drawbacks
Wang et al. [41]	LLM-KGMQA: Multi-hop question-answering system for the medical field, addressing entity linking and reasoning	Improved accuracy (99.80%) and robust performance	Requires significant computational resources for entity linking and multi-hop reasoning
Jiang et al. [72]	Knowledge-guided agricultural LLM for domain-specific decision-making	State-of-the-art performance in agriculture, improves accuracy in agricultural queries	Data scarcity in the domain, challenges in precision retrieval
Benjira et al. [137]	LLM-augmented KG for SDG indicator mapping, integrating open data	High precision in mapping open data to SDGs	Requires access to high-quality and diverse metadata
Freire et al. [109]	LLM-powered knowledge-sharing system for manufacturing, retrieving information from factory documentation	Enhances quick information retrieval and resolution of issues	Preference for human experts still prevails, open-source models may have privacy concerns
Xie et al. [122]	Theme-based lecture summary system using LLMs and graph-based segmentation	Enhances knowledge recall and student satisfaction	Effectiveness varies with lecture structure
Sun et al. [88]	SF-GPT: A training-free method for enhancing LLM-based KG construction	Improved recall and F1 score, no training required	May not work well with all types of datasets or knowledge fusion

Fig. 6: Flowchart of knowledge base construction and use in large language model.

TABLE VI: Category 1: Research Based on Reasoning Frameworks and Methods
Authors	Year	Innovation	Advantages	Limitations
Wu et al. [51]	2025	Proposed Long-to-Short (L2S) reasoning and model merging.	Increases reasoning efficiency while maintaining performance.	Merging efficiency depends on model size, adds computational complexity.
Tung et al. [90]	2025	Introduced GreenMind, using strategy optimization for reasoning.	Outperforms previous works on Vietnamese reasoning tasks.	Limited to specific languages, struggles with low-resource languages.
Ma et al. [100]	2025	Introduced SQL-R1, a reinforcement learning-based model for NL2SQL.	Improves accuracy in complex database tasks.	Reinforcement learning training may suffer from cold start issues.
Costas [28]	2024	GNN-RAG combines GNN reasoning with LLMs in a retrieval-augmented generation style.	Optimizes graph data processing, improves knowledge graph QA.	Performance depends on graph data quality, limited in complex reasoning.
Sui et al. [162]	2024	Proposed FiDeLiS for factuality-enhanced LLM reasoning.	Improves factuality and interpretability.	Dependency on reasoning steps increases computational cost.
Luo et al. [116]	2025	Introduced RALLRec+ for enhancing recommendation system reasoning.	Enhances recommendation reasoning, handles dynamic user preferences.	Requires continuous data updates, may still be less effective in real-time applications.

Fig. 7: Summary of inference optimization techniques in large language models.

TABLE VII: Category 2: Application-Specific Reasoning Research
Authors	Year	Innovation	Advantages	Limitations
Yang et al. [57]	2025	Introduced Reasoning-based Bias Detector (RBD).	Reduces bias in evaluations, improves consistency.	Needs significant supervision for bias detection.
Zhang et al. [73]	2024	Designed step-wise multimodal models for document reasoning.	Enhances multimodal understanding, improves complex question answering.	Relies heavily on high-quality labeled data.
Beary et al. [14]	2025	Introduced hybrid fine-tuning to enhance medical domain reasoning.	Improves reasoning accuracy in radiation oncology.	Domain-specific, limited transferability to other medical fields.
Tan et al. [141]	2025	Proposed Hydra framework for cross-source enhanced reasoning.	Excels in multi-hop and multi-entity reasoning tasks.	Requires diverse data sources, computationally intensive.
Kunat et al. [81]	2025	Merged language-specific LLMs with advanced reasoning models.	Enhances reasoning in low-resource languages.	Can be computationally expensive for merging models.
Lan et al. [95]	2024	Proposed LLM4QA for graph-based reasoning with SPARQL queries.	Improves knowledge graph reasoning efficiency.	Relies on structured knowledge graphs, limited in unstructured data.
Li et al. [66]	2024	Proposed LINKED for filtering and integrating knowledge in reasoning tasks.	Improves commonsense reasoning accuracy.	Performance may degrade with noisy knowledge sources.
Inoue et al. [160]	2025	Introduced DrugAgent, a multi-agent LLM system for drug-target prediction.	Enhances reliability and transparency in drug-target predictions.	Multi-agent approach may face challenges in information integration.
Phuc et al. [127]	2025	Proposed QUERY2TREE for reasoning over knowledge graphs.	Improves logical query accuracy with knowledge graph embeddings.	Dependent on knowledge graph quality, may struggle with complex queries.

Fig. 8: The core components and development path of embodied intelligence.

TABLE VIII: LLM-driven Embodied Task Execution and Planning
Authors	Year	Innovation	Advantages	Limitations
Fan et al. [55]	2025	Proposed a three-stage framework for industrial robot task execution using LLMs	Outstanding GPT-4 performance in industrial tasks	Limited ability in 3D spatial planning
Chen et al. [152]	2025	Designed RoboGPT with long-term decomposition and feedback adjustment	Adaptive subgoal updating	Requires large datasets and complex modules
Ruaridh et al. [106]	2025	Introduced ELLMER with RAG for multi-step reasoning	Strong sensorimotor adaptation	Tested mainly in home-like tasks
Li et al. [91]	2024	Proposed multi-modal LLMs for autonomous mining driving	Applicable to realistic mining scenarios	Conceptual work, lacks implementation
Song et al. [23]	2023	Introduced LLM-Planner for few-shot planning	Strong performance with limited data	Weak environmental state modeling
Andrew et al. [11]	2025	Unified GEA model across embodied tasks via MLLM	High cross-domain generalization	Data-hungry and complex training
Vishnu et al. [128]	2023	Proposed LGX for zero-shot navigation	Large improvement in success rate	Validated mostly in simulation
Schumann et al. [102]	2024	VELMA verbalizes visual context for navigation	Greatly improves city-scale VLN	Requires explicit visual-language mapping
Wu et al. [173]	2023	TaPA plans executable actions with scene grounding	Outperforms GPT-3.5 in planning	Object recognition impacts execution
Danny et al. [31]	2023	PaLM-E integrates vision, state, and language	Versatile across modalities and tasks	Costly training and deployment
Mower et al. [26]	2024	ROS-LLM enables non-expert robot programming via chat	Supports multiple behavior modes	Relies on full ROS ecosystem

Fig. 9: The development of embodied intelligence.

TABLE IX: Multi-Modal Fusion and Perception Enhancement in Embodied Systems
Authors	Year	Innovation	Advantages	Limitations
Sun et al. [153]	2025	Multi-level LLM framework for coal mine sensor analysis	Efficient risk prediction and learning	Requires simulated and expert knowledge
Abhay et al. [1]	2024	EnvGen: LLM-generated adaptive RL training environments	Boosts training efficiency	Depends on feedback quality
Zheng et al. [118]	2023	Steve-Eye combines LLM and visual encoders	Enables end-to-end multimodal interaction	Requires large datasets
Ma et al. [169]	2025	Perspective on LLMs in autonomous driving	Offers future research outlook	Lacks technical solutions
Liu et al. [48]	2024	EAI-SIM simulation platform with ROS	Controls UAVs and arms in photo-realistic sim	Needs powerful simulation hardware
Song et al. [111]	2024	Guide-LLM aids visually impaired via text maps	Combines commonsense and path planning	Small sample evaluation
Micol et al. [93]	2025	VITA for adaptive mental well-being coaching	Personalized multi-modal robot coaching	Limited experimental scale
Kolby et al. [79]	2023	DECKARD agent with Dream/Wake exploration	Greatly improves RL efficiency	Relies on LLM hypothesized subgoals
Yang et al. [157]	2024	EMMA distilled from text-world LLM to visual world	Successfully transfers to vision tasks	Needs dual-modality synchronization

Fig. 10: Different categories of Embodied Robots: (a) Quadruped Robots, (b) Fixed-base Robots, (c) Tracked Robots, (d) Wheeled Robots, and (e) Humanoid Robots.

TABLE X: Multi-Agent Collaboration and Organizational Strategies
Authors	Year	Innovation	Advantages	Limitations
Zhang et al. [145]	2024	ReAd mechanism improves multi-agent LLM efficiency	Fewer LLM queries and agent steps	Evaluation mainly in simulation
Liu et al. [65]	2024	FaGeL uses smart fabric for implicit feedback	Learns user preference non-intrusively	High system complexity
Guo et al. [142]	2024	Organizational prompting reduces LLM agent conflict	Boosts teamwork and efficiency	Validation in real-world pending

Fig. 11: Example diagram of the integration of embodied intelligence and large language model.

TABLE XI: Security and Robustness Challenges in Embodied LLMs
Authors	Year	Innovation	Advantages	Limitations
Liu et al. [5]	2025	Contextual backdoor attacks in LLM-based agents	Dual-modal triggers for covert actions	No defense strategy yet in real use
Zhang et al. [52]	2024	Investigated LLM-based robot jailbreak threats	Systematic exposure of jailbreak scenarios	No concrete safety mechanisms
Li et al. [92]	2024	Universal benchmarking interface for LLM agents	Fine-grained error taxonomy	No coverage of real multimodal scenes
Kovalev et al. [7]	2022	Reviewed LLM-based instruction planners without training	Works in zero-shot scenarios	Lack of real task validations
Szot et al. [12]	2023	LLaRP: RL-trained LLM policy with vision	Superior multi-task performance	Needs intensive training and environment
Leon et al. [84]	2024	Studied virtual agent personality effects in VR	Extrovert agents yield better experience	No embodiment in physical robots
Liu et al. [115]	2024	EIRAD dataset and BLIP2 for adversarial attack test	Reveals attack success in LLM agents	Evaluation mainly on textual input

Fig. 12: Embodied Intelligence, Large Language Model, Knowledge Base and Reasoning Ability together constitute the core architecture of the intelligent system.

TABLE XII: Summary of Studies on LLM-driven Embodied Control
Authors	Year	Innovation	Advantages	Limitations
Ruaridh et al. [107]	2025	Proposed ELLMER framework combining GPT-4 with RAG for task planning in unpredictable environments.	Enables long-horizon task execution via visual/force feedback; adaptable planning.	Generalization scope not fully validated.
Fan et al. [56]	2024	Introduced LLM agents in manufacturing for autonomous design, planning, and control.	Achieved 81.88% task completion with GPT-4 in complex scenarios.	Challenges remain in 3D spatial task handling and real-time integration.
Liang et al. [62]	2023	Introduced “Code as Policies” using LLMs to write robot policy code from commands.	Enables reactive and trajectory-based control via few-shot prompting.	Limited physical-world validation.
Xu et al. [71]	2025	Reviewed LLM-integrated CPSS frameworks for human-centered Industry 5.0.	Connects LLMs, perception, scheduling, and swarm intelligence into unified framework.	Lacks empirical task-specific evaluations.
Zhao et al. [134]	2024	Developed MultiBotGPT system using GPT-3.5 for UAV/UGV task execution.	Outperforms BERT in assignment success rate; improves operator experience.	Focused on limited command types and single-modal input.
Shen et al. [124]	2024	Discussed LLMs as foundation for embodied agents in physical environments.	Conceptual bridge from virtual to real-world through embodiment.	Lacks concrete framework or task-based implementation details.
Jiang et al. [78]	2024	Proposed KoMA: LLM-driven multi-agent system for autonomous driving.	Achieves robust decision-making via planning, shared memory, and reflection.	Complexity of real-world deployment and scalability remains a challenge.

왜 중요한가

챗봇 수준의 AI를 실제 로봇이나 자율주행차, 가정용 서비스 로봇에 적용하려는 개발자와 연구자에게 현재 어떤 기술 조각들이 있고 어디가 비어 있는지 지도를 제공한다. 특정 모델이나 코드를 새로 내놓은 것은 아니지만, 앞으로 어떤 문제를 풀어야 진짜 쓸만한 로봇 AI가 나올지 방향을 잡는 데 참고할 수 있다.

이 논문의 용어

임바디드 지능(Embodied Intelligence) · 로봇처럼 물리적인 몸을 가지고 실제 환경에서 보고 움직이며 배우는 인공지능
지식베이스(Knowledge Base) · 사실이나 개념을 구조화해 저장해두고 AI가 검색해 쓸 수 있게 만든 데이터베이스
Chain-of-Thought / Tree-of-Thought · AI가 답을 한 번에 내지 않고 단계별로(체인) 또는 여러 갈래로(트리) 생각을 풀어내며 추론하는 방식
ReAct 프레임워크 · AI가 추론과 실제 행동(도구 사용, 외부 환경과의 상호작용)을 번갈아 하도록 만드는 방법
폐쇄루프 지식통합 · AI가 새로운 정보를 접할 때마다 자신의 지식을 실시간으로 갱신하도록 하는 구조

논문 원문 초록 (영문)

The convergence of large language models (LLMs), structured knowledge bases (KBs), and reasoning ability (RA) presents a promising trajectory toward general embodied intelligence (GEI). This paper reviews the evolution of LLM-centered intelligent systems, emphasising their integration with knowledge representation, logical reasoning, and physical embodiment. We analyse LLM architectures, pre-training methods, and inference mechanisms, along with their interaction with external knowledge sources and structured reasoning frameworks. Furthermore, we examine embodied intelligence (EI) paradigms wherein agents learn and act in physical environments. To synthesise these dimensions, we present a conceptual framework that illustrates the synergy among LLMs, KBs, RA, and embodiment, serving as a guiding model for perception, reasoning, and action rather than an implemented engineering architecture. To advance toward GEI, we identify five key challenges: efficient LLM deployment, closed-loop knowledge integration, hybrid symbolic-neural reasoning, perception-action grounding, and continual learning. This survey provides a comprehensive roadmap for developing adaptive, multimodal agents capable of operating in complex, dynamic settings.

저자 · Fujiang Yuan, Xia Huang, Lusheng Wang, Jun Ding, Zhen Tian, Yuxin Wang, Shaojie Gu, Yuki Funabora, Yanhong Peng, Zebing

arXiv에서 원문 보기

METAL LAB 최신 기사

그림 출처: Fujiang Yuan et al., arXiv:2608.19794, arxiv-nonexclusive

Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents

무엇을 했나

왜 중요한가

이 논문의 용어

최신 논문

METAL LAB 최신 기사