每天早上一封邮件，把昨天的 AI 梳理好订阅邮件

METAL LAB

Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents

arXiv:2608.197942026-08-21

一篇综述论文探讨如何让只会说话的AI变成能真正行动的具身智能体

这篇综述梳理了如何把大语言模型、结构化知识库、推理能力和具身智能(能在真实环境中感知和行动的机器人等载体)结合起来,朝着通用具身智能的方向发展。作者用多张对比表总结了各领域现有研究的成果与不足,并提出一个概念性框架来说明这四者如何协同工作。作者明确说明这只是一个指导性的思路模型,而不是已经落地的工程系统。

他们做了什么

  1. 论文的出发点是:GPT-4、PaLM、Gemini、Claude等大语言模型虽然语言能力很强,但本质上是'无身体的',缺乏对物理世界的真实理解
  2. 论文系统回顾了大语言模型的架构与训练方法、大语言模型与知识库的连接方式、推理框架(例如让模型逐步展开思考的思维链Chain-of-Thought,以及让模型并行探索多条思路的思维树Tree-of-Thought)、以及让智能体在真实或模拟环境中学习行动的具身智能研究
  3. 论文提出一个概念框架,展示大语言模型、知识库、推理能力和物理载体如何共同支撑感知、推理与行动,但作者强调这只是指导性模型,而非已实现的工程架构
  4. 论文指出通向通用具身智能还有五大挑战:让大模型能轻量化部署到机器人等设备上、构建能实时更新的闭环知识系统、融合符号规则与神经网络的混合推理、把语言指令对接到具体物理动作的感知-行动对齐、以及不遗忘旧知识的持续学习能力
  5. 与已有的具身智能综述相比,作者称这是首次系统性地将语言模型、知识、推理与物理交互整合进统一架构的工作
Fig. 1: Schematic diagram of a unified multimodal large language model architecture.
Fig. 1: Schematic diagram of a unified multimodal large language model architecture.
TABLE I: Comparative Summary of Related Embodied Intelligence Review Articles
AuthorsYearCore ContributionsLimitations
Yao et al. [151]2025Reviews multimodal embodied intelligence tasks and models; explores LLM/MLM integration with robots; proposes PPA paradigm and evaluates MLMs.Classical AI lacks physical interaction; DRL is data-hungry and less robust; VLN training is costly and generalization is poor; LLMs lack causal reasoning; Sim2Real transfer is weak; evaluation standards are lacking.
Dong et al. [135]2025Analyzes LLM-enabled Human-Robot Symbiotic Manufacturing (HRSM), including interaction, collaboration, and execution.Privacy and data sensitivity risks; hallucination and bias in LLMs; absence of unified evaluation standards; integration difficulties in diverse environments; token length limits document processing.
Tan et al. [70]2025Proposes CEIM to fuse multi-source data and LLM reasoning for semantic integration and decision-making in customized manufacturing.Data fusion issues; low semantic alignment; real-time control accuracy is lacking; high computation cost; insufficient standardization of AI-device interfaces.
Jeong et al. [59]2024Surveys five directions of LLM/VLM in robotics: reward design, control, planning, manipulation, and scene understanding.Limited by embedded compute; weak sensor fusion; prompt bias and poor interpretability; lacks domain-specific knowledge; Sim2Real generalization is weak.
Wang et al. [68]2025Discusses LLMs in robotics tasks and evaluates GPT-4V; explores future applications in agriculture and medicine.Plan robustness is lacking; long prompts require expert input; fixed action sets reduce flexibility; API is closed-source; RGBD-based 3D reasoning is limited.
Our Survey2025Systematically analyzed the synergy among LLMs, knowledge bases, reasoning capabilities, and embodied intelligence; proposed a unified framework and five future directions for GEI.Limited empirical examples; insufficient discussion on multilingual settings and real-world deployment.
Fig. 2: Evolution and Taxonomy of Large Language Models.
Fig. 2: Evolution and Taxonomy of Large Language Models.
TABLE II: Comparison of Research on Large Language Models (LLMs)
Author(s)Core ImprovementAdvantagesApplication Scenarios
Cummins et al. [25]Compiler optimization frameworkEnhances disassembly processesCompiler optimization
Hu et al. [174]Prompting strategies incorporating context to enhance VQA performanceStrengthens contextual understandingKnowledge-based applications
Hu et al. [166]LLM-TIKG: Threat Intelligence Knowledge Graph frameworkEnhances cybersecurity analysisThreat Intelligence Analysis
Qian et al. [80]Infrastructure for large-scale LLM trainingEfficiently handles large-scale training tasksLLM training infrastructure
Sun et al. [121]UniCoder: Intermediate representation to improve downstream NLP task processingEnhances the model’s ability to handle complex tasksModel scaling
Zhang et al. [156]BB-GeoGPT: Framework tailored for Geographic Information ScienceHighly adaptable to domain-specific knowledgeGeographic Information Science (GIS)
Wang et al. [139]VideoAgent: Vision-language integrated system for long-form video interpretationImproves multimodal reasoning capabilitiesMultimodal reasoning
Xia et al. [19]LLMGA: LLM-based generation assistant combining image generation and editing toolsPrecise control over image synthesis,Image generation and editing
Zhou et al. [172]Weak-to-Strong Search: Greedy search for aligning large models with smaller counterpartsCompute-efficient, enhances model generalizationModel alignment and fine-tuning
Land et al. [110]Glitch token analysis and detection methodImproves tokenizer robustnesstokenizer optimization
Fig. 3: Schematic architecture of the Transformer model.
Fig. 3: Schematic architecture of the Transformer model.
TABLE III: Summary of Key Contributions in Embodied Intelligence Research
AuthorYearResearch TopicKey ContributionApplication Domain
Zardini et al. [46]2020Co-Design FrameworkStructured co-design for integrationGeneral systems
Ma et al. [18]2021Soft Robotics ActuatorsIntegrated sensing & actuationSoft robotics
Gupta et al. [4]2021DERL for AdaptationCo-evolution of body and controlLocomotion & control
Roy et al. [96]2021ML in Embodied SystemsEnergy-aware physical learningRobotics & ML
Mengaldo et al. [45]2022Physics ModelingPhysics-based modelingSoft robotics
Iida et al. [43]2022Temporal DynamicsTimescale-based self-organizationAdaptive systems
Long et al. [158]2023Human-in-the-loop LearningInteractive simulation for trainingSurgical robotics
Zeng et al. [39]2023LLMs for RoboticsSurvey of LLM-enhanced roboticsHuman-robot interaction
Fan et al. [54]2024LLMs in IndustryLLM-based autonomous controlIndustrial robots
Feng et al. [69]2025Spatial AI ReasoningSpatial AI via LLMsMultiscale AI systems
Fig. 4: Construction Process of Large Language Models.
Fig. 4: Construction Process of Large Language Models.
TABLE IV: LLM and Knowledge Graph Integration
Author(s)InnovationAdvantagesDrawbacks
Daniel et al. [30]Verifying RDF triples using LLMs by comparing RDF statements to external documentsHigh precision (88%) in RDF verificationRequires human oversight, method may be slower
Feng et al. [155]Knowledge Graph–based thought framework to improve LLM responses with verified KG informationReduces factual errors, enhances drug–cancer associationsLimited to biomedical, requires cross-domain testing
Dai et al. [140]Empirical study on LLMs’ comprehension of KG input formats and prompt organizationImproves LLM performance in KG-based tasksVarying effectiveness across different models, noisy subgraphs
Erickson et al. [75]Enhancing LLM workflows with RAG and KGs to ensure traceability and flexibilityMore accurate and explainable LLM outputsIntegration with external tools can be complex
Xiang et al. [138]LLM-based generative retrieval framework focusing on trustworthy search resultsImproves relevance, responsibility, and trustworthinessWorks better with certain query types, not universally applicable
Sequeda et al. [77]LLM-powered enterprise question answering, supported by KGsProvides formal validation and trusted data for queriesMay not scale well for large or complex queries in diverse enterprises
Fig. 5: Description of the execution process of the intelligent document processing and knowledge question answering system.
Fig. 5: Description of the execution process of the intelligent document processing and knowledge question answering system.
TABLE V: LLM for Domain-Specific Applications
AuthorsInnovationAdvantagesDrawbacks
Wang et al. [41]LLM-KGMQA: Multi-hop question-answering system for the medical field, addressing entity linking and reasoningImproved accuracy (99.80%) and robust performanceRequires significant computational resources for entity linking and multi-hop reasoning
Jiang et al. [72]Knowledge-guided agricultural LLM for domain-specific decision-makingState-of-the-art performance in agriculture, improves accuracy in agricultural queriesData scarcity in the domain, challenges in precision retrieval
Benjira et al. [137]LLM-augmented KG for SDG indicator mapping, integrating open dataHigh precision in mapping open data to SDGsRequires access to high-quality and diverse metadata
Freire et al. [109]LLM-powered knowledge-sharing system for manufacturing, retrieving information from factory documentationEnhances quick information retrieval and resolution of issuesPreference for human experts still prevails, open-source models may have privacy concerns
Xie et al. [122]Theme-based lecture summary system using LLMs and graph-based segmentationEnhances knowledge recall and student satisfactionEffectiveness varies with lecture structure
Sun et al. [88]SF-GPT: A training-free method for enhancing LLM-based KG constructionImproved recall and F1 score, no training requiredMay not work well with all types of datasets or knowledge fusion
Fig. 6: Flowchart of knowledge base construction and use in large language model.
Fig. 6: Flowchart of knowledge base construction and use in large language model.
TABLE VI: Category 1: Research Based on Reasoning Frameworks and Methods
AuthorsYearInnovationAdvantagesLimitations
Wu et al. [51]2025Proposed Long-to-Short (L2S) reasoning and model merging.Increases reasoning efficiency while maintaining performance.Merging efficiency depends on model size, adds computational complexity.
Tung et al. [90]2025Introduced GreenMind, using strategy optimization for reasoning.Outperforms previous works on Vietnamese reasoning tasks.Limited to specific languages, struggles with low-resource languages.
Ma et al. [100]2025Introduced SQL-R1, a reinforcement learning-based model for NL2SQL.Improves accuracy in complex database tasks.Reinforcement learning training may suffer from cold start issues.
Costas [28]2024GNN-RAG combines GNN reasoning with LLMs in a retrieval-augmented generation style.Optimizes graph data processing, improves knowledge graph QA.Performance depends on graph data quality, limited in complex reasoning.
Sui et al. [162]2024Proposed FiDeLiS for factuality-enhanced LLM reasoning.Improves factuality and interpretability.Dependency on reasoning steps increases computational cost.
Luo et al. [116]2025Introduced RALLRec+ for enhancing recommendation system reasoning.Enhances recommendation reasoning, handles dynamic user preferences.Requires continuous data updates, may still be less effective in real-time applications.
Fig. 7: Summary of inference optimization techniques in large language models.
Fig. 7: Summary of inference optimization techniques in large language models.
TABLE VII: Category 2: Application-Specific Reasoning Research
AuthorsYearInnovationAdvantagesLimitations
Yang et al. [57]2025Introduced Reasoning-based Bias Detector (RBD).Reduces bias in evaluations, improves consistency.Needs significant supervision for bias detection.
Zhang et al. [73]2024Designed step-wise multimodal models for document reasoning.Enhances multimodal understanding, improves complex question answering.Relies heavily on high-quality labeled data.
Beary et al. [14]2025Introduced hybrid fine-tuning to enhance medical domain reasoning.Improves reasoning accuracy in radiation oncology.Domain-specific, limited transferability to other medical fields.
Tan et al. [141]2025Proposed Hydra framework for cross-source enhanced reasoning.Excels in multi-hop and multi-entity reasoning tasks.Requires diverse data sources, computationally intensive.
Kunat et al. [81]2025Merged language-specific LLMs with advanced reasoning models.Enhances reasoning in low-resource languages.Can be computationally expensive for merging models.
Lan et al. [95]2024Proposed LLM4QA for graph-based reasoning with SPARQL queries.Improves knowledge graph reasoning efficiency.Relies on structured knowledge graphs, limited in unstructured data.
Li et al. [66]2024Proposed LINKED for filtering and integrating knowledge in reasoning tasks.Improves commonsense reasoning accuracy.Performance may degrade with noisy knowledge sources.
Inoue et al. [160]2025Introduced DrugAgent, a multi-agent LLM system for drug-target prediction.Enhances reliability and transparency in drug-target predictions.Multi-agent approach may face challenges in information integration.
Phuc et al. [127]2025Proposed QUERY2TREE for reasoning over knowledge graphs.Improves logical query accuracy with knowledge graph embeddings.Dependent on knowledge graph quality, may struggle with complex queries.
Fig. 8: The core components and development path of embodied intelligence.
Fig. 8: The core components and development path of embodied intelligence.
TABLE VIII: LLM-driven Embodied Task Execution and Planning
AuthorsYearInnovationAdvantagesLimitations
Fan et al. [55]2025Proposed a three-stage framework for industrial robot task execution using LLMsOutstanding GPT-4 performance in industrial tasksLimited ability in 3D spatial planning
Chen et al. [152]2025Designed RoboGPT with long-term decomposition and feedback adjustmentAdaptive subgoal updatingRequires large datasets and complex modules
Ruaridh et al. [106]2025Introduced ELLMER with RAG for multi-step reasoningStrong sensorimotor adaptationTested mainly in home-like tasks
Li et al. [91]2024Proposed multi-modal LLMs for autonomous mining drivingApplicable to realistic mining scenariosConceptual work, lacks implementation
Song et al. [23]2023Introduced LLM-Planner for few-shot planningStrong performance with limited dataWeak environmental state modeling
Andrew et al. [11]2025Unified GEA model across embodied tasks via MLLMHigh cross-domain generalizationData-hungry and complex training
Vishnu et al. [128]2023Proposed LGX for zero-shot navigationLarge improvement in success rateValidated mostly in simulation
Schumann et al. [102]2024VELMA verbalizes visual context for navigationGreatly improves city-scale VLNRequires explicit visual-language mapping
Wu et al. [173]2023TaPA plans executable actions with scene groundingOutperforms GPT-3.5 in planningObject recognition impacts execution
Danny et al. [31]2023PaLM-E integrates vision, state, and languageVersatile across modalities and tasksCostly training and deployment
Mower et al. [26]2024ROS-LLM enables non-expert robot programming via chatSupports multiple behavior modesRelies on full ROS ecosystem
Fig. 9: The development of embodied intelligence.
Fig. 9: The development of embodied intelligence.
TABLE IX: Multi-Modal Fusion and Perception Enhancement in Embodied Systems
AuthorsYearInnovationAdvantagesLimitations
Sun et al. [153]2025Multi-level LLM framework for coal mine sensor analysisEfficient risk prediction and learningRequires simulated and expert knowledge
Abhay et al. [1]2024EnvGen: LLM-generated adaptive RL training environmentsBoosts training efficiencyDepends on feedback quality
Zheng et al. [118]2023Steve-Eye combines LLM and visual encodersEnables end-to-end multimodal interactionRequires large datasets
Ma et al. [169]2025Perspective on LLMs in autonomous drivingOffers future research outlookLacks technical solutions
Liu et al. [48]2024EAI-SIM simulation platform with ROSControls UAVs and arms in photo-realistic simNeeds powerful simulation hardware
Song et al. [111]2024Guide-LLM aids visually impaired via text mapsCombines commonsense and path planningSmall sample evaluation
Micol et al. [93]2025VITA for adaptive mental well-being coachingPersonalized multi-modal robot coachingLimited experimental scale
Kolby et al. [79]2023DECKARD agent with Dream/Wake explorationGreatly improves RL efficiencyRelies on LLM hypothesized subgoals
Yang et al. [157]2024EMMA distilled from text-world LLM to visual worldSuccessfully transfers to vision tasksNeeds dual-modality synchronization
Fig. 10: Different categories of Embodied Robots: (a) Quadruped Robots, (b) Fixed-base Robots, (c) Tracked Robots, (d) Wheeled Robots, and (e) Humanoid Robots.
Fig. 10: Different categories of Embodied Robots: (a) Quadruped Robots, (b) Fixed-base Robots, (c) Tracked Robots, (d) Wheeled Robots, and (e) Humanoid Robots.
TABLE X: Multi-Agent Collaboration and Organizational Strategies
AuthorsYearInnovationAdvantagesLimitations
Zhang et al. [145]2024ReAd mechanism improves multi-agent LLM efficiencyFewer LLM queries and agent stepsEvaluation mainly in simulation
Liu et al. [65]2024FaGeL uses smart fabric for implicit feedbackLearns user preference non-intrusivelyHigh system complexity
Guo et al. [142]2024Organizational prompting reduces LLM agent conflictBoosts teamwork and efficiencyValidation in real-world pending
Fig. 11: Example diagram of the integration of embodied intelligence and large language model.
Fig. 11: Example diagram of the integration of embodied intelligence and large language model.
TABLE XI: Security and Robustness Challenges in Embodied LLMs
AuthorsYearInnovationAdvantagesLimitations
Liu et al. [5]2025Contextual backdoor attacks in LLM-based agentsDual-modal triggers for covert actionsNo defense strategy yet in real use
Zhang et al. [52]2024Investigated LLM-based robot jailbreak threatsSystematic exposure of jailbreak scenariosNo concrete safety mechanisms
Li et al. [92]2024Universal benchmarking interface for LLM agentsFine-grained error taxonomyNo coverage of real multimodal scenes
Kovalev et al. [7]2022Reviewed LLM-based instruction planners without trainingWorks in zero-shot scenariosLack of real task validations
Szot et al. [12]2023LLaRP: RL-trained LLM policy with visionSuperior multi-task performanceNeeds intensive training and environment
Leon et al. [84]2024Studied virtual agent personality effects in VRExtrovert agents yield better experienceNo embodiment in physical robots
Liu et al. [115]2024EIRAD dataset and BLIP2 for adversarial attack testReveals attack success in LLM agentsEvaluation mainly on textual input
Fig. 12: Embodied Intelligence, Large Language Model, Knowledge Base and Reasoning Ability together constitute the core architecture of the intelligent system.
Fig. 12: Embodied Intelligence, Large Language Model, Knowledge Base and Reasoning Ability together constitute the core architecture of the intelligent system.
TABLE XII: Summary of Studies on LLM-driven Embodied Control
AuthorsYearInnovationAdvantagesLimitations
Ruaridh et al. [107]2025Proposed ELLMER framework combining GPT-4 with RAG for task planning in unpredictable environments.Enables long-horizon task execution via visual/force feedback; adaptable planning.Generalization scope not fully validated.
Fan et al. [56]2024Introduced LLM agents in manufacturing for autonomous design, planning, and control.Achieved 81.88% task completion with GPT-4 in complex scenarios.Challenges remain in 3D spatial task handling and real-time integration.
Liang et al. [62]2023Introduced “Code as Policies” using LLMs to write robot policy code from commands.Enables reactive and trajectory-based control via few-shot prompting.Limited physical-world validation.
Xu et al. [71]2025Reviewed LLM-integrated CPSS frameworks for human-centered Industry 5.0.Connects LLMs, perception, scheduling, and swarm intelligence into unified framework.Lacks empirical task-specific evaluations.
Zhao et al. [134]2024Developed MultiBotGPT system using GPT-3.5 for UAV/UGV task execution.Outperforms BERT in assignment success rate; improves operator experience.Focused on limited command types and single-modal input.
Shen et al. [124]2024Discussed LLMs as foundation for embodied agents in physical environments.Conceptual bridge from virtual to real-world through embodiment.Lacks concrete framework or task-based implementation details.
Jiang et al. [78]2024Proposed KoMA: LLM-driven multi-agent system for autonomous driving.Achieves robust decision-making via planning, shared memory, and reflection.Complexity of real-world deployment and scalability remains a challenge.

为什么重要

对于想把聊天机器人式的AI推进到真实机器人、自动驾驶或家用服务机器人的开发者和研究者来说,这篇综述提供了一张现状地图,标出哪些技术拼图已经存在、哪些环节仍然缺失。它没有提出新模型或代码,但可以作为规划具身智能研究方向的参考。

本文术语

  • 具身智能 · 拥有物理形态、能在真实环境中感知和行动的人工智能系统,例如机器人
  • 知识库 · 把事实或概念结构化存储起来,供AI查询调用的数据库
  • 思维链/思维树 · 让模型按步骤(链)或分支并行(树)展开中间推理过程的方法
  • ReAct框架 · 让语言模型在推理和实际执行动作(如调用外部工具)之间交替进行的方法
  • 闭环知识集成 · AI系统能根据新信息或反馈实时更新自身存储知识的设计

论文原文摘要（英文）

The convergence of large language models (LLMs), structured knowledge bases (KBs), and reasoning ability (RA) presents a promising trajectory toward general embodied intelligence (GEI). This paper reviews the evolution of LLM-centered intelligent systems, emphasising their integration with knowledge representation, logical reasoning, and physical embodiment. We analyse LLM architectures, pre-training methods, and inference mechanisms, along with their interaction with external knowledge sources and structured reasoning frameworks. Furthermore, we examine embodied intelligence (EI) paradigms wherein agents learn and act in physical environments. To synthesise these dimensions, we present a conceptual framework that illustrates the synergy among LLMs, KBs, RA, and embodiment, serving as a guiding model for perception, reasoning, and action rather than an implemented engineering architecture. To advance toward GEI, we identify five key challenges: efficient LLM deployment, closed-loop knowledge integration, hybrid symbolic-neural reasoning, perception-action grounding, and continual learning. This survey provides a comprehensive roadmap for developing adaptive, multimodal agents capable of operating in complex, dynamic settings.

作者 · Fujiang Yuan, Xia Huang, Lusheng Wang, Jun Ding, Zhen Tian, Yuxin Wang, Shaojie Gu, Yuki Funabora, Yanhong Peng, Zebing

在 arXiv 阅读

最新论文

全部论文 →

METAL LAB 最新报道

图片来源： Fujiang Yuan et al., arXiv:2608.19794, arxiv-nonexclusive