AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents

arXiv:2608.088142026-08-08

A city-navigation test built from 360-degree videos of Tokyo's Akihabara shows today's best AI agents scoring only about a fifth of what humans score

360CityArena is a 175-task benchmark built on a photorealistic virtual reconstruction of Tokyo's Akihabara district, stitched together from 602 360-degree video segments, designed to test how well AI agents can understand, navigate, and reason about a real urban environment. Human participants averaged 77.3% accuracy, while the best-performing AI model tested, Gemini 2.5 Flash, reached only 17.1%. AI agents struggled especially with map-based navigation, counting objects, and figuring out spatial relationships between landmarks.

METAL LAB explanatory visual

How 360CityArena is built and tested

Evidence statusMeasured results reported

  1. Build the virtual city602 360-degree video segments stitched into a navigable graph of 193 nodes and 305 edges covering 85 streets in Akihabara
  2. Design 175 tasksThree categories (Environment Understanding, Path Reasoning, Spatial Reasoning) split into seven subtypes with Easy/Medium/Hard difficulty labels
  3. Run AI agentsGPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen2.5-VL, and InternVL3.5 act step-by-step in Unity using visual input, a map, and memory
  4. Measure human baselineFive participants who had visited Akihabara complete the same tasks, averaging 77.3% accuracy
  5. Compare and diagnose failuresBest AI (17.1%) trails humans (77.3%) by far; failures classified into Action, Grounding, Perception, Explore, and Planning categories
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

  1. Existing outdoor benchmarks either lack photorealism (traditional 3D simulators) or lack continuous, dynamic navigability (Google Street View) or use video clips too short for large-scale exploration, so the authors stitched 602 360-degree video segments covering 85 streets (roughly 750m by 650m) into a navigable graph of 193 connected nodes and 305 edges recreating Akihabara.
  2. On top of this environment, they hand-crafted 175 tasks across three categories (Environment Understanding, Path Reasoning, Spatial Reasoning) and seven subcategories, such as self-localization, finding a landmark from a text or image description, navigating with a map, following language instructions, reasoning about relative positions between landmarks, and counting objects, each labeled Easy, Medium, or Hard.
  3. They ran proprietary AI models (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash) and open-source models (Qwen2.5-VL, InternVL3.5) as embodied agents in this environment, and compared their performance against five human participants who had actually visited Akihabara.
  4. Humans reached around 90% accuracy on map navigation, image-based landmark search, language-instruction navigation, and relational spatial reasoning, while all tested AI models fell far short across every task, with performance dropping further as task difficulty increased.
  5. Giving AI agents extra location information (a map marker showing their current position) did not consistently help and sometimes hurt performance, suggesting the models struggle to connect map data with what they actually see.
Figure 1: 360CityArena. We introduce a benchmark for evaluating embodied agents in a photorealistic reconstruction of Akihabara, Tokyo, Japan, built from interconnected 360° video trajectories. The benchmark covers realistic urban streets and evaluates agents on diverse tasks requiring environment understanding, path reasoning, and spatial reasoning.
Figure 1: 360CityArena. We introduce a benchmark for evaluating embodied agents in a photorealistic reconstruction of Akihabara, Tokyo, Japan, built from interconnected 360° video trajectories. The benchmark covers realistic urban streets and evaluates agents on diverse tasks requiring environment understanding, path reasoning, and spatial reasoning.
Figure 2: Examples in each task type in 360CityArena. (i) Environment Understanding tasks include Localization, Landmark Search with Language, and Landmark Search with Image, where the agent infers its location or navigates to a specified landmark. (ii) Path Reasoning tasks evaluate the agent’s ability to plan and execute routes, such as following map-based paths or vision-language navigation. (iii) Spatial Reasoning tasks assess relational understanding and quantitative perception, including identifying spatial relations between landmarks and counting objects in the environment.
Figure 2: Examples in each task type in 360CityArena. (i) Environment Understanding tasks include Localization, Landmark Search with Language, and Landmark Search with Image, where the agent infers its location or navigates to a specified landmark. (ii) Path Reasoning tasks evaluate the agent’s ability to plan and execute routes, such as following map-based paths or vision-language navigation. (iii) Spatial Reasoning tasks assess relational understanding and quantitative perception, including identifying spatial relations between landmarks and counting objects in the environment.
Table 1: Comparison of urban navigation environments – realism, structural complexity, dynamics, interactivity, and exploration capability.
EnvironmentCategoryPhoto- realismStructural complexityDynamicsInteractionDistrict-scale explorationMotion
EmbodiedCity [12]3D simulatorLowLowMedium✓✓Continuous
MetaUrban [44]3D simulatorLowMediumMedium✓✓Continuous
CARLA [9]3D simulatorLowLowLow✓✓Continuous (clip)
Vid2Sim [45]Video-to-simHighHighMedium✓✗Continuous
StreetLearn [30]GSV-basedHighHighLow✗✓Discrete
360CityArena (Ours)360° videoHighHighHigh✗✓Continuous (trajectory)
Figure 3: Example of the visual observations in 360CityArena. The agent moves through the city while scanning its surroundings. At branching points, it must choose a route.
Figure 3: Example of the visual observations in 360CityArena. The agent moves through the city while scanning its surroundings. At branching points, it must choose a route.
Figure 4: Success rate by difficulty level across tasks and models (%). This graph shows model and human performance for each task, divided into three difficulty levels: Easy (E), Medium (M), and Hard (H).
Figure 4: Success rate by difficulty level across tasks and models (%). This graph shows model and human performance for each task, divided into three difficulty levels: Easy (E), Medium (M), and Hard (H).
Table 2: Overall Results (%). Comparison of model and human performance across seven spatial and reasoning tasks, grouped into three major categories. Gemini 2.5 Flash achieves the highest overall performance, while the strongest model varies across individual tasks. All LMMs still fall far short of human performance.
Environment UnderstandingPath ReasoningSpatial Reasoning
LocLandmark (Lang)Landmark (Img)Map NavVLNObj CountRel Reason
GPT-58.016.048.00.08.02.432.0
Claude Sonnet 4.54.04.016.04.04.010.88.0
Gemini 2.5 Flash12.028.036.00.08.024.012.0
Qwen2.5-VL-32B-Instruct4.016.020.00.00.018.84.0
InternVL3.5-8B4.020.020.00.012.02.80.0
InternVL3.5-38B0.016.00.00.012.07.24.0
Human68.064.092.092.088.045.292.0
Figure 5: Failure cause breakdown by task category across models (%). This figure shows the distribution of failure causes for each model across three task categories. Failures are categorized into five types: Action, Grounding, Perception, Explore, and Planning, and each stacked bar reports the percentage breakdown within the corresponding (model, task category) setting.
Figure 5: Failure cause breakdown by task category across models (%). This figure shows the distribution of failure causes for each model across three task categories. Failures are categorized into five types: Action, Grounding, Perception, Explore, and Planning, and each stacked bar reports the percentage breakdown within the corresponding (model, task category) setting.
Figure 6: Example of the agent’s views for Landmark Search with Language, resulting in failure. The agent is instructed to search for “Jonathan”. From t = 8 to t = 38, the gaze repeatedly moved up and down. At t = 39, the gaze briefly shifted to the right, but from t = 41 onward, it returned to the same up–down movement.
Figure 6: Example of the agent’s views for Landmark Search with Language, resulting in failure. The agent is instructed to search for “Jonathan”. From t = 8 to t = 38, the gaze repeatedly moved up and down. At t = 39, the gaze briefly shifted to the right, but from t = 41 onward, it returned to the same up–down movement.
Table 3: Comparison with and without location information. We observe that the performance did not improve consistently across tasks; in some cases, accuracy decreased instead.
Environment UnderstandingPath ReasoningSpatial Reasoning
LocLandmark (Lang)Landmark (Img)Map NavVLNObj CountRel Reason
GPT-58.016.048.00.08.02.432.0
GPT-5 (w/o location)-24.044.04.08.00.048.0
t = 8. Look above.
t = 8. Look above.
t = 40. Look to the right.
t = 40. Look to the right.

Findings

  • Human participants averaged 77.3% accuracy versus 17.1% for the best AI model tested (Gemini 2.5 Flash), a large gap.
  • Image-based landmark search generally outperformed language-based landmark search (e.g., GPT-5: 48.0% vs. 16.0%; Claude: 16.0% vs. 4.0%; Qwen: 20.0% vs. 16.0%), though InternVL models showed no clear gain from images (8B: 20.0% vs. 20.0%; 38B: 0.0% vs. 16.0%).
  • AI performance dropped as task difficulty increased, e.g., GPT-5's accuracy on Environment Understanding fell from 28.0% to 25.0% to 18.5%, and on Path Reasoning from 11.1% to 0.0% to 0.0% across Easy, Medium, and Hard settings.
  • Adding explicit location information did not consistently improve performance: language-based landmark search dropped from 24.0% to 16.0%, relational spatial reasoning dropped from 48.0% to 32.0%, while object counting improved only slightly.
  • Failure analysis showed GPT-5 had few low-level action errors (12%) but dominant exploration-strategy failures (55%), while Gemini 2.5 Flash and InternVL3.5-38B struggled mainly with low-level action control (around 40%), and perception failures spiked across all models on spatial reasoning tasks (reaching 38% for Gemini).
t = 41. Look below.
t = 41. Look below.
Figure 7: Example of the agent’s views for Landmark Search with Image, resulting in success. The agent is instructed to search for “Jonathan” given in image form.
Figure 7: Example of the agent’s views for Landmark Search with Image, resulting in success. The agent is instructed to search for “Jonathan” given in image form.

Where it can be used

  • Testing guide robots or navigation assistants for visually impaired pedestrians under realistic conditions before real-world deployment.
  • Studying how well AI agents connect map information with first-person visual scenes, and comparing how they follow text versus image instructions.
  • Serving as a reference methodology for building photorealistic virtual-city training data for embodied AI systems.
t = 5. Go past.
t = 5. Go past.
t = 16. Spot blue and white vertical stripes.
t = 16. Spot blue and white vertical stripes.

Limits and open work

  • Because the environment is built from pre-recorded 360-degree video paths, agents can only move along captured trajectories rather than freely to arbitrary locations, and cannot physically interact with objects.
  • The benchmark currently covers only one district (Akihabara, Tokyo), so it is untested whether findings generalize to other cities or regions.
  • Transitions across video-segment boundaries can introduce visual discontinuities, though the authors report these were not a major cause of failures in their analysis.
  • The human baseline comes from only five participants familiar with Akihabara, so it represents a local-expert upper bound rather than a general human average.
  • The ability to align map-based location information with real visual cues was not independently evaluated or trained for, and is left as future work.

Why it matters

Building AI assistants that can guide people around real cities or help visually impaired pedestrians requires realistic testing grounds, and 360CityArena offers conditions close to actual streets to expose specific weaknesses. It highlights that today's AI agents particularly lack the ability to align map information with first-person views and to explore purposefully instead of getting stuck.

Terms in this paper

  • LMM (large multimodal model) · a large AI model that can process and reason over both images and text together
  • pose graph · a network connecting recorded camera positions and orientations to form a traversable route map
  • VLN (Vision-Language Navigation) · a task where an agent must follow multi-step natural-language directions to move through an environment
  • MRA (mean relative accuracy) · a scoring method for numeric estimation tasks that averages accuracy across several error-tolerance thresholds
  • embodied agent · an AI system that perceives an environment through sensors like cameras and acts within it

Original abstract (English)

We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos. Existing outdoor benchmarks either lack sufficient photorealism or complexity, resulting in a considerable gap from real-world urban environments. 360CityArena is built on a realistic reconstruction of the Akihabara district in Tokyo, Japan, using 602 360-degree video segments covering 85 streets, and consists of 175 meticulously human-crafted tasks. It encompasses three task categories: Environment Understanding, Path Reasoning, and Spatial Reasoning, covering fundamental abilities required for urban exploration, such as localization, landmark search, path planning, and relational spatial reasoning, thereby enabling comprehensive evaluation in realistic urban scenes. Our evaluation using state-of-the-art LMM-based agents shows that even the strongest model, Gemini 2.5 Flash, performs far below human level (human: 77.3% vs. Gemini 2.5 Flash: 17.1%), revealing substantial challenges that remain in city-scale embodied navigation and reasoning. 360CityArena provides a necessary and challenging testbed for photorealistic urban-district navigation and spatial reasoning.

Authors · Kenta Watanabe

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Kenta Watanabe et al., arXiv:2608.08814, arxiv-nonexclusive