EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving

Accepted at ECCV 2026
Finn Rasmus Schäfer1,  Yuan Gao1,  Dingrui Wang1,  Thomas Stauner2,  Stephan Günnemann3,  Mattia Piccinini1,  Sebastian Schmidt2,3,  and Johannes Betz1
1 Professorship of Autonomous Vehicle Systems, Technical University of Munich, Munich, Germany
2 Bayerische Motoren Werke AG, Munich, Germany
3 Data Analytics and Machine Learning Group, Technical University of Munich, Munich, Germany
Paper arXiv Code Dataset

Vision-centric foundation models fail to ground physical reasoning in observation — classical visual odometry beats them, and ego-motion logic is routed almost entirely through language, not vision.

Technical University of Munich Autonomous Vehicle Systems Lab

Leaderboard

Rank Model Type Input Pars. (%) Semantic Temporal Physics Consistency
Acc. BAcc. ▼ F1 BAcc. F1 PCR PCov

Sorted by Semantic BAcc. (descending). Click any column header to re-sort. PCR = Physics Consistency Rate (weighted ego-motion consistency). PCov = Physics Consistency Coverage. Parsability indicates the fraction of responses automatically parsed into valid answers.

Submit Your Model

EgoDyn-Bench accepts new model submissions on a rolling basis. The benchmark scores 14,000 QA pairs over 1,000 balanced 3-second driving clips and reports the metrics shown above. To add your model:

  1. Run the evaluation pipeline on the benchmark clips using the provided script. The pipeline writes one prediction per clip_id × question_id in a deterministic JSON format and re-parses model outputs with the published cascade.
  2. Generate the results manifest: a single JSON containing your model name, type (Closed / Open / Baseline), input modality (Vision / Vision + Trajectory), and the per-clip predictions. Aggregated metrics are computed by the grader, not submitted.
  3. Submit via one of:

Evaluation code, dataset, and reproduction scripts are publicly available — see the Code and Dataset buttons at the top of this page. We re-run the deterministic parser and grader on every submission to ensure consistent scoring across entries.

Abstract

While Vision-Language Models (VLMs) have advanced high-level reasoning in autonomous driving, their ability to ground this reasoning in the underlying physics of ego-motion remains poorly understood. We introduce EgoDyn-Bench, a diagnostic benchmark for evaluating the semantic ego-motion understanding of vision-centric foundation models. By mapping continuous vehicle kinematics to discrete motion concepts via a deterministic oracle, we decouple a model's internal physical logic from its visual perception. Our large-scale empirical audit spanning 20 models, including closed-source MLLMs, open-source VLMs across multiple scales, and specialized VLAs, identifies a significant Perception Bottleneck: while models exhibit logical physical concepts, they consistently fail to accurately align them with visual observations, frequently underperforming classical non-learned geometric baselines. This failure persists across model scales and domain-specific training, indicating a structural deficit in how current architectures couple visual perception with physical reasoning.

We demonstrate that providing explicit trajectory encodings substantially restores physical consistency across all evaluated models, revealing a functional disentanglement between vision and language: ego-motion logic is derived almost exclusively from the language modality, while visual observations contribute negligible additional signal. This structural finding provides a standardized diagnostic framework and a practical pathway toward physically aligned embodied AI.

Keywords: Ego-motion · Physical Reasoning · Foundation Models

Key Findings

Four headline results from the audit that together reveal a structural Perception Bottleneck in vision-centric foundation models.

63.8%
Classical Visual Odometry (BAcc)

beats the best VLM (Gemini 3 Pro, 47.0%) on the 6-question subset that geometric baselines can answer.

+20.7pp
Text-only trajectory gain

Qwen3-VL-8B jumps from 38.9% (vision only) to 59.6% BAcc when frames are replaced by trajectory text.

+2.6pp
Adding video back to text

Re-introducing visual frames to the text-only baseline recovers only a marginal gain — vision contributes negligible temporal signal beyond static scene context.

20 → 97
WPCR with one static frame

Physical consistency leaps from 20% (no vision) to 97% with a single static image; additional frames add nothing.

Taken together, these results indicate a functional disentanglement between vision and language: ego-motion logic is derived almost exclusively from the language modality, while visual input contributes little temporal signal beyond static scene context.

Dataset Composition

EgoDyn-Bench combines real-world data from nuScenes with simulated scenarios from CARLA to cover the full spectrum of driving dynamics. CARLA clips are generated with five behaviour profiles (Balanced, Comfort, Default, Sporty, Safety) to ensure diverse maneuver coverage.

Real-world driving datasets like nuScenes predominantly contain routine driving with limited dynamic maneuvers. To ensure our benchmark covers the full spectrum of ego-motion behaviors, we augment with CARLA-simulated scenarios featuring aggressive acceleration, emergency braking, and sharp turns. To minimize the visual domain gap, we apply NVIDIA Cosmos Transfer 2 for photorealistic style transfer. All synthetic clips undergo human-in-the-loop quality verification to ensure visual fidelity while preserving physical consistency of the underlying trajectories.

Data source composition and CARLA behaviour breakdown

Left: Equal split between real-world (nuScenes) and simulated (CARLA) clips. Right: Distribution of driving behaviour profiles across the CARLA subset.

Before & After: Cosmos Transfer 2

Original CARLA rendering (top) vs. Cosmos-transferred output (bottom). The domain transfer produces photorealistic imagery while preserving the exact ego-motion trajectory.

Safety-Conservative behaviour
Sporty behaviour
Answer balance comparison: nuScenes only vs. augmented benchmark

Per-question answer balance (normalised entropy). Augmenting with CARLA clips significantly improves label balance across all question types, reducing bias toward dominant answer classes present in the real-world data.

Dynamics feature distributions: nuScenes vs CARLA

Kernel density estimates of key dynamics features show that CARLA clips extend the distribution tails, adding aggressive maneuvers and high lateral accelerations underrepresented in nuScenes.

Feature correlation matrix

Feature correlations across the combined dataset. Most features are moderately correlated, confirming that the 14 question types capture complementary aspects of ego-motion.

Benchmark Overview

EgoDyn-Bench evaluates whether vision-language models can understand ego-vehicle dynamics from short driving clips. Each clip is 3 seconds long and paired with physics-grounded questions derived from actual trajectory data.

1,000

Driving Clips

14,000

QA Pairs

14

Question Types

500

nuScenes (real-world)

500

CARLA + Cosmos Transfer (synthetic)

Question Types

Our benchmark covers 14 question types organized into two categories, testing both direct dynamics understanding and comparative temporal reasoning.

Direct Dynamics

Question Prompt Answers
Speed Trend Is the vehicle accelerating, decelerating, or maintaining steady speed? accelerating decelerating steady
Speed Regime What is the vehicle's speed regime? stopped slow urban highway
Low Mean Speed Is the mean speed below 5 m/s (18 km/h)? yes no
Braking Intensity What is the intensity level of the vehicle's braking? emergency moderate low none
Driving Smoothness How smooth is the driving based on jerk? smooth moderate aggressive
Dominant Motion Axis Is the vehicle's motion primarily longitudinal or lateral? longitudinal lateral none
Turn Direction Is the vehicle turning left, right, or going straight? left right straight
Heading Change Does the vehicle change heading by more than 15 degrees? yes no
High Lateral Accel. Does the vehicle experience high lateral acceleration? yes no
Extreme Maneuver Does the vehicle perform an extreme maneuver (high jerk or hard braking)? yes no
Stop-and-Go Does the vehicle exhibit stop-and-go behavior? yes no
Brake-then-Turn Does the vehicle brake and then turn (sequential maneuver)? yes no

Comparative / Temporal

Question Prompt Answers
Speed Peak Half Does the maximum speed occur in the first or second half? first_half second_half no_peak
Contrastive Sequence Which half of the sequence has more dynamic driving? first_half second_half similar

Comparative questions require temporal reasoning by contrasting the first and second halves of a clip.

Benchmark Design

EgoDyn-Bench generates ground-truth labels automatically from physics quantities extracted from ego-vehicle trajectories. This section outlines the three key design components: the labeling pipeline, threshold calibration, and the evaluation prompt format.

Automated Label Generation

Each question type is backed by a labeling rule that maps continuous dynamics features (speed, acceleration, yaw rate, jerk) to categorical answers. We implement 12 rule families:

Single & Multi-Threshold
Classify a scalar feature into ordered bins (e.g., speed → stopped / slow / urban / highway).
Trend Classification
Detect acceleration / deceleration / steady from the mean of a feature with symmetric dead-zone thresholds.
Yaw-Rate Sign
Determine turn direction from the sign of peak yaw rate, with a configurable dead-zone to filter sensor noise.
Dominant Axis & Lateral Accel.
Compare longitudinal vs. lateral RMS accelerations; detect high lateral forces from v × ω.
Sequential & OR-Event Detection
Identify temporal patterns (brake-then-turn) and compound events (high jerk or hard braking).
Temporal Comparison
Split a clip in half and compare feature aggregates (peak location, first-half vs. second-half dynamics).

Threshold Calibration

Labeling thresholds are calibrated in two stages. First, physics-grounded defaults are set from domain standards (ISO 15622 for ACC control bands, AASHTO for lateral comfort limits, ISO 2631 for ride quality). Second, we run data-driven calibration on the full data pool (N = 41,999 candidate clips) to find percentile-aligned thresholds that yield approximately uniform answer distributions per question type. For example, braking intensity boundaries at the 25th, 50th, and 75th percentiles of minimum longitudinal acceleration ensure roughly equal representation of emergency / moderate / low / none.

As shown in the Robustness Analysis, perturbing all thresholds by α ∈ [0.5, 1.5] does not materially change model rankings (Kendall’s τ > 0.9), confirming that benchmark conclusions are robust to calibration choices.

Evaluation Prompt Format

Models receive a multiple-choice prompt with 10 evenly-spaced frames from the 3-second clip. Each question lists its valid answer options in square brackets and instructs the model to reply with only the chosen option. An example prompt is shown below: 

The 10 images show the forward camera view at evenly
spaced moments across a 3-second driving clip.

Is the vehicle accelerating, decelerating, or maintaining
steady speed? [accelerating / decelerating / steady]

Answer with ONLY the chosen option.

Example prompt for the Speed Trend question (vision-only mode).

Trajectory Embeddings

Beyond vision, our evaluation framework can prepend textual ego-motion context to the prompt via four embedding modes. This lets us ablate whether models benefit from explicit trajectory information in addition to the video frames.

Summary (default)

8 scalar features extracted from the full 3 s window.

Vehicle dynamics: max_speed=8.2m/s (30km/h),
 mean_speed=7.4m/s, min_accel=-1.23m/s²,
 max_yaw_rate=0.042rad/s, max_jerk=2.85m/s³,
 mean_jerk=0.91m/s³, max_lat_accel=0.34m/s²,
 heading_change=0.126rad

Timeseries

Per-channel values at N evenly-spaced instants.

Vehicle dynamics (10 time-steps over 3.0s):
t(s):    0.00, 0.33, 0.67, 1.00, ...
speed(m/s): 7.1, 7.4, 7.8, 8.0, ...
accel(m/s²): 0.82, 0.65, 0.31, -0.05, ...
yaw_rate(rad/s): 0.012, 0.018, 0.025, ...
jerk(m/s³): -0.51, -1.02, -1.08, ...

Coordinates

Zero-centred (x, y) waypoints and heading.

Vehicle trajectory (10 waypoints over 3.0s, metres):
t(s): 0.00, 0.33, 0.67, 1.00, ...
x(m): 0.0, 2.4, 4.9, 7.3, ...
y(m): 0.0, 0.1, 0.3, 0.5, ...
heading(rad): 1.571, 1.578, 1.589, ...

Full

Both timeseries and coordinates combined.

Vehicle dynamics (10 time-steps over 3.0s):
t(s):    0.00, 0.33, 0.67, ...
speed(m/s): 7.1, 7.4, 7.8, ...
accel(m/s²): 0.82, 0.65, 0.31, ...
[...]
Vehicle trajectory (10 waypoints over 3.0s, metres):
t(s): 0.00, 0.33, 0.67, ...
x(m): 0.0, 2.4, 4.9, ...
y(m): 0.0, 0.1, 0.3, ...

The four trajectory embedding modes. Each is prepended to the prompt text between the frame description and the question. The none mode omits trajectory context entirely for a vision-only baseline.

Human-in-the-Loop Clip Viewer

We provide an open-source local web viewer (clip_viewer.py) for interactive quality assurance of the benchmark data. The tool serves a zero-dependency single-page application that displays:

  • Side-by-side video comparison of original CARLA renders and Cosmos-transferred outputs (with automatic H.264 transcoding and caching)
  • All 14 ground-truth QA pairs per clip, including question text, answer options, and the assigned label
  • Extracted dynamics features for manual plausibility checks

Keyboard navigation (arrow keys / j k), clip search, and per-clip feature inspection enable rapid human-in-the-loop verification of label correctness and Cosmos transfer fidelity. The viewer was used to identify and discard clips with visual artifacts before final benchmark assembly.

The clip viewer displays side-by-side video comparisons, synchronized dynamics time-series plots, per-clip feature summaries, and all 14 ground-truth QA pairs for rapid quality assurance.

Explore Examples

Browse random clips from the benchmark. See the input frames, question, ground-truth answer, and how each model responded.

Loading examples…

Interactive explorer coming soon.

Results

Modality Ablation

Where does Qwen3-VL-8B's reasoning actually come from? Vision conditions cluster around 33–39% BAcc; adding trajectory text vaults performance into the 55–60% range — including when video frames are removed.

Metric:

Vision frames contribute negligible temporal signal. Shuffled and temporally ordered frames perform identically (~39% BAcc), and a single static frame is nearly as good. Replacing frames with trajectory text yields a +20.7pp BAcc jump; re-introducing the frames (with the default Summary encoding) recovers no additional gain, confirming that ego-motion reasoning is routed almost exclusively through the language modality.

Overall Performance (Vision only)

Filter:

Semantic F1 (Global) for all evaluated models using vision-only input. Closed-source models generally outperform open-source alternatives, but all models show substantial room for improvement.

Trajectory Feature Ablation

Effect of trajectory features
Per-question trajectory improvement

Left: Adding trajectory features as text context consistently improves all models by +20–28 percentage points. Right: Per-question breakdown shows the largest gains for speed estimation and smoothness assessment, where visual cues alone are most ambiguous.

Per-Question Breakdown (Vision only)

Filter:

Vision only: Performance varies significantly across question types. Models perform best on binary classification tasks but struggle with fine-grained multi-class problems (braking intensity, contrastive sequences).

Per-Question Breakdown (Vision + Trajectory)

Filter:

Vision + trajectory: With trajectory features, all question types see substantial gains. The improvement is most pronounced for speed-related and smoothness questions where visual cues alone are ambiguous.

Trajectory Encoding Ablation

Two distinct model families (Qwen3-VL and InternVL3.5) converge on the same encoding preferences, confirming the bottleneck is architectural rather than data- or fitting-driven.

Metric:

Dense Timeseries kinematics and the Full (Timeseries + Coordinates) encoding consistently outperform the high-level Summary, while raw Coordinates regress sharply across both architectures. The 15.6pp gap between Timeseries and Coordinates indicates models process the dynamics representation rather than retrieving the answer from the text.

Radar Comparison

Filter:

Per-question-type F1 profiles reveal distinct strengths: Gemini 3 Pro excels at lateral dynamics while GPT-5.1 shows more balanced performance. Toggle model groups to compare.

Global vs. Temporal

Filter:

Models above the diagonal excel at temporal reasoning relative to global dynamics. Circle = vision-only; square = vision + trajectory. Trajectory features shift models upward and rightward.

Performance by Data Source

Filter:

Per-model Macro F1 on nuScenes (real-world) vs. CARLA (Cosmos-transferred) clips. Models clustered near the diagonal perform comparably across sources, confirming the domain transfer preserves task difficulty and the benchmark is not biased toward either data source.

Isolating the Domain Gap

Are the results driven by ego-motion complexity or by simulation-to-reality and style-transfer artifacts? Performance stays stable across all three visual domains — the bottleneck is structural, not a domain shift.

Balanced accuracy across nuScenes (real-world), raw CARLA (simulation), and Cosmos-transferred CARLA (style-transferred) for the four geometric baselines and a representative VLM. Visual Odometry and Qwen3-VL-8B vary by only Δmax 1.6 pp and 1.3 pp respectively, supporting the use of synthetic data for assessing real-world ego-motion understanding.

Answer Parsability

Filter:

Fraction of model responses that can be automatically parsed into valid answers. Most models achieve near-perfect parsability, indicating that the multiple-choice format is well-understood. Lower rates for some models reflect verbose or off-format responses rather than question difficulty.

Robustness Analysis

Our benchmark derives ground-truth labels from continuous dynamics features via calibrated thresholds. To validate that results are not artifacts of specific threshold choices, we systematically perturb all thresholds by a factor α ∈ [0.5, 1.5] and re-evaluate the full model set.

Global Threshold Sensitivity

Metric:
Type:
Input:
▮ Closed ▮ Open VLM ▮ Baseline ── Vision    ╌╌ Vision+Traj

Model performance as labeling thresholds are uniformly scaled by α. The dotted vertical line marks the nominal threshold (α = 1.0). Hover over a line to identify the model. Rankings and relative performance remain stable, confirming benchmark conclusions are robust to threshold choice.

Per-Question Sensitivity

Metric:

Per-question type performance as thresholds are scaled by α. Toggle between mean balanced accuracy and oracle label flip rate. Hover a line to identify the question type. The dotted line marks α = 1.0.

Sensitivity Heatmap

Metric:

Per-question sensitivity across threshold perturbation factors. Switch between label flip rate, mean balanced accuracy, and Kendall’s τ ranking stability. Darker red = higher flip rate; darker blue = higher rank stability.

Oracle Consistency

Oracle EMCR and WEMCR under threshold perturbation

Oracle EMCR and WEMCR remain stable across perturbation factors, indicating that the physics-grounded consistency rules are not overly sensitive to the specific threshold calibration.

Limitations & Scope

EgoDyn-Bench is a methodological diagnostic, not a replacement for the explicit state sensors that production autonomous vehicles rely on. Our goal is to expose whether vision-centric foundation models can ground their predictions in the vehicle's own kinematics — a prerequisite for cross-modal consistency and OOD detection, not a substitute for sensor-based estimation. The benchmark operates within a principled, adjustable abstraction: 14 semantic categories spanning lateral, longitudinal, compound, and temporal axes, with thresholds calibrated to standard automotive comfort and safety limits. We do not claim this taxonomy is exhaustive — the released calibrate_thresholds.py lets researchers re-normalize boundaries for new domains. The current scope is passenger-vehicle driving on real-world (nuScenes) and CARLA-augmented sequences; generalization to other embodiments (motorcycles, trucks, off-road platforms) and downstream questions — whether this grounding capability translates into closed-loop driving performance — are deliberately framed as future work.

Citation

@misc{schäfer2026egodynbenchevaluatingegomotionunderstanding,
  title         = {EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving},
  author        = {Finn Rasmus Schäfer and Yuan Gao and Dingrui Wang and Thomas Stauner and Stephan Günnemann and Mattia Piccinini and Sebastian Schmidt and Johannes Betz},
  year          = {2026},
  eprint        = {2604.22851},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2604.22851}
}

Acknowledgments

This research was conducted in collaboration with the BMW Group and was supported by their research funding. Generative AI tools were used for language editing and proofreading during manuscript preparation; all content was reviewed and verified by the authors, who take full responsibility for the final manuscript.