One email each morning — yesterday's AI, sortedGet it in your inbox

METAL LAB

Towards Real-Time and Adaptable LiDAR Scene Completion

arXiv:2608.164902026-08-16

Filling the blind spots in self-driving car LiDAR scans in just 0.1 seconds

LiDAR sensors on self-driving cars produce sparse point clouds with big gaps where objects are occluded or far away. RapidLiDAR learns, from data, where to place the starting points for filling those gaps, instead of relying on random noise or a fixed jitter amount like prior methods. Tested on SemanticKITTI and KITTI-360, it matches the completion quality of the best existing methods while finishing a full scene in 0.1 seconds, 2.3 times faster than the fastest prior method.

What they did

  1. Proposes RapidLiDAR, which takes a sparse LiDAR point cloud and produces a dense, gap-filled 3D scene in a single forward pass through the network.
  2. Introduces an adaptive initialization module that predicts how far and in what direction each input point should spread, moving points further apart near occluded or sparse areas and only slightly in already dense areas.
  3. Adds a multi-scale reconstruction module that queries 3D voxel features and 2D bird's-eye-view (BEV) feature maps at multiple resolutions to further refine point positions without searching point neighborhoods.
  4. Replaces neighborhood-search operations like farthest point sampling and k-nearest neighbor search with voxel- and BEV-based feature extraction, making the method faster and able to handle different input resolutions.
  5. On SemanticKITTI and KITTI-360, achieves completion quality on par with the state of the art while completing a scene in 0.1 seconds, matching the 10 Hz scan rate of typical automotive LiDAR sensors.
Figure 1: Initialization matters. Top row: each method’s initialization; bottom row: the corresponding refined output. The highlighted box marks a large unobserved region. (a) LiDiff [19] starts from Gaussian noise that carries no information about the scene; (b) LiNeXt [6] perturbs the input with a fixed noise variance, so its points stay near the observed surface and never reach across the gap; (c) Our adaptive module learns data-dependent displacements that populate the region from the surrounding geometry, and this coverage is preserved in both the coarse initialization and the final refined result.
Figure 1: Initialization matters. Top row: each method’s initialization; bottom row: the corresponding refined output. The highlighted box marks a large unobserved region. (a) LiDiff [19] starts from Gaussian noise that carries no information about the scene; (b) LiNeXt [6] perturbs the input with a fixed noise variance, so its points stay near the observed surface and never reach across the gap; (c) Our adaptive module learns data-dependent displacements that populate the region from the surrounding geometry, and this coverage is preserved in both the coarse initialization and the final refined result.
Table 1: Scene Completion on SemanticKITTI and KITTI-360. Quantitative comparison with prior methods. † denotes methods with an additional refinement. Best results in each group are highlighted in bold.
SemanticKITTIKITTI-360
MethodCD ↓JSD 3D ↓JSD BEV ↓CD ↓JSD 3D ↓JSD BEV ↓
LMSCNet0.641–0.4310.979–0.496
LODE1.029–0.4511.565–0.483
MID0.503–0.4700.637–0.476
PVD1.256–0.498–––
LiDiff0.4340.5640.4440.564–0.459
LiDPM0.4460.5320.440–––
ScoreLiDAR0.406–0.4250.472–0.444
LiFlow0.309–0.416–––
LiNeXt0.2140.4940.3360.2170.5080.355
Ours0.2060.4750.3320.2110.4920.338
LiDiff†0.3760.5730.4160.517–0.446
ScoreLiDAR†0.342–0.3990.452–0.437
LiDPM†0.3760.5420.403–––
LiNeXt†0.1490.4810.3310.1490.4990.339
Ours†0.1380.4780.3300.1400.4900.336
Figure 2: Overview of RapidLiDAR. The Multi-Scale Feature Extraction module (center) voxelizes the input point cloud X∈ℝM×3 and extracts multi-scale 3D voxel features (F1,F2,…,Fn) and a dense 2D BEV feature map Bdense. The Adaptive Initialization Module (top) predicts a spatially varying displacement Δ for each point in P~ to obtain the initialized scene Pinit. The Multi-Scale Reconstruction Module (bottom) projects voxel features into BEV feature maps and cross-attends per-point features from Pinit with the multi-scale BEV feature maps using multi-scale deformable attention. A final MLP predicts a residual displacement that aligns each point with the underlying target surfaces, producing the completed scene.
Figure 2: Overview of RapidLiDAR. The Multi-Scale Feature Extraction module (center) voxelizes the input point cloud X∈ℝM×3 and extracts multi-scale 3D voxel features (F1,F2,…,Fn) and a dense 2D BEV feature map Bdense. The Adaptive Initialization Module (top) predicts a spatially varying displacement Δ for each point in P~ to obtain the initialized scene Pinit. The Multi-Scale Reconstruction Module (bottom) projects voxel features into BEV feature maps and cross-attends per-point features from Pinit with the multi-scale BEV feature maps using multi-scale deformable attention. A final MLP predicts a residual displacement that aligns each point with the underlying target surfaces, producing the completed scene.
Table 2: Computational Efficiency Comparison. We report Chamfer distance, number of learnable parameters, and inference time per scan.
MethodCD ↓Param (M) ↓Time (s) ↓
LiDiff0.43432.6730.1
ScoreLiDAR0.40632.677.1
LiNeXt0.2141.990.23
Ours0.20611.80.10
Figure 3: Illustration of Dense BEV Head. Converts sparse 3D volumetric features into a dense BEV map. The channel and depth dimensions are merged and projected to Cout using a 2D convolution. Multi-head self-attention over BEV tokens captures global scene context, followed by residual 2D convolutions for feature refinement.
Figure 3: Illustration of Dense BEV Head. Converts sparse 3D volumetric features into a dense BEV map. The channel and depth dimensions are merged and projected to Cout using a 2D convolution. Multi-head self-attention over BEV tokens captures global scene context, followed by residual 2D convolutions for feature refinement.
Table 3: Architectural Ablation Study. We evaluate the impact of our core modules on the SemanticKITTI validation set. Best results are in bold.
MethodCD ↓JSD 3D ↓JSD BEV ↓
Ours0.2060.4750.332
Ours w/o AIM0.2180.4880.345
Ours w/o MSRM0.2150.4940.342
Figure 4: Qualitative Comparison on SemanticKITTI. Our method produces more complete geometry in large occluded regions compared to prior methods.
Figure 4: Qualitative Comparison on SemanticKITTI. Our method produces more complete geometry in large occluded regions compared to prior methods.
Table 4: Effect of Maximum Displacement Bound. Reconstruction performance for different values of Smax, evaluated on a downsampled validation set of 180,000 output points.
SmaxCD ↓
500.2594
700.2592
1000.2589
Table 5: Voxel Resolution Ablation. Impact of voxel resolution η on reconstruction performance, parameter count, and inference time.
η (m)CD ↓Param (M)Time (s) ↓
0.50.21410.00.07
0.40.21011.60.09
0.30.20611.80.10
0.20.20811.90.14

Why it matters

LiDAR scene completion has to run in real time to be useful for self-driving cars, but existing diffusion-based methods are too slow and fixed-noise methods need manual retuning for every new sensor setup. This work reduces both problems at once, moving scene completion closer to speeds usable in actual vehicles.

Terms in this paper

  • LiDAR scene completion · filling in the parts of a 3D scene that a sensor could not observe, to produce a complete map
  • diffusion model · a generative model that turns random noise into a result through many gradual denoising steps, which makes it slow
  • Bird's-Eye-View (BEV) · a 2D grid representation of a 3D scene as seen from directly above
  • voxel · a small cube-shaped unit used to divide 3D space, like a 3D version of a pixel
  • Chamfer Distance · a metric measuring how close two sets of points are to each other, used to score reconstruction accuracy

Original abstract (English)

LiDAR scene completion is a key component of 3D perception in autonomous driving, where the scene must be completed in real time to be usable in downstream tasks. Existing approaches typically follow an initialize-and-refine paradigm, in which a coarse initialization of the scene is first constructed, then refined into complete 3D geometry. Generative models are slower because they iteratively refine random Gaussian noise into the scene, while non-generative methods perturb the partial scene with a fixed noise scale, which limits coverage of large gaps and occluded regions and requires manual recalibration for each new sensor configuration. We present RapidLiDAR, a LiDAR scene completion method that treats the initialization itself as a learned, data-driven component. We propose an adaptive initialization module that predicts a spatially varying displacement for each partial input point, expanding the partial observations into a coarse scene initialization adapted to the local geometry, without requiring manual noise tuning. To refine this coarse initialization into a complete and coherent scene, we additionally propose a multi-scale reconstruction module that further refines point positions by querying multi-scale 3D voxel and 2D BEV feature maps constructed from the input scan. By replacing point-neighborhood operators such as farthest point sampling and k-nearest neighbor search with voxel- and BEV-based feature extraction, our architecture is faster and can handle different input resolutions by design. Experiments on SemanticKITTI and KITTI-360 show that our method achieves completion performance on par with the state of the art while completing a full scene in 0.1 seconds, which is 2.3 times faster than the fastest prior method. This matches the 10 Hz acquisition rate of typical automotive LiDAR sensors, taking a step toward real-time LiDAR scene completion.

Authors · Azhar Hussian

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Azhar Hussian et al., arXiv:2608.16490, cc-by-nc-sa-4.0