ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

Xinze Li¹ Yiyuan Wang^1,2 Pengxu Chen³ Wentao Fan^1,4 Weifeng Su^1,4 Weisi Lin⁵ Wentao Cheng^1,*

¹Beijing Normal-Hong Kong Baptist University ²Hong Kong Baptist University ³Jilin University

⁴Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science

⁵Nanyang Technological University

* Corresponding author: Wentao Cheng. Email: wentaocheng@bnbu.edu.cn

arXiv Code BibTeX

TL;DR: A reliability-calibrated learning-rate rule for streaming 3D reconstruction.

ReCal3R constant memory, no retraining

Abstract

Streaming 3D reconstruction relies on a compact recurrent scene state to process long image streams in linear time and bounded memory. However, repeated updates can gradually corrupt this state, causing reliable historical information to be overwritten by noisy or ambiguous observations. We introduce ReCal3R, a reliability calibrated learning rate method for recurrent 3D reconstruction. Instead of directly applying a candidate learning rate, our method estimates state-token reliability from the maintained scene state and uses it to calibrate a candidate learning rate derived from token alignment, state reconstruction residual, and recent update pressure. The resulting token-wise learning rate interpolates between a conservative base rate and the candidate rate, suppressing aggressive updates on unreliable tokens while preserving adaptation to informative frames. Applied to CUT3R as a training-free calibration rule, ReCal3R reaches strong performance on long sequences in pose, depth, and reconstruction quality, including a 3.7× reduction in ATE, with comparable runtime and memory.

Sequences

Streaming reconstructions produced frame by frame, with no state reset or reinitialization. Each clip is the model's raw output over the full sequence — pick a scene to play it.

Method

ReCal3R overview and qualitative comparison against CUT3R — ReCal3R estimates state token reliability and a candidate learning rate from the recurrent forward pass, and uses reliability calibration to obtain the final state learning rate. This prevents unreliable state tokens from receiving aggressive updates and leads to cleaner reconstructions than CUT3R over long image streams.

Accuracy and Efficiency

Stability shows up most clearly when streams get long. On 1,000-frame ScanNet, ReCal3R lowers ATE from CUT3R's 0.786 to 0.211 — a 3.7× reduction — while running at 19.9 FPS against CUT3R's 23.5 and using 6.1 GB of memory against 6.0. In short, the accuracy gain comes from calibrating which tokens get written, not from spending more compute: runtime and memory stay at CUT3R's level.

ATE, FPS, and GPU memory comparison on 1,000-frame ScanNet sequences

Visualization

Click an RGB thumbnail below to load its corresponding 3D reconstruction. Drag to orbit, scroll to zoom, and inspect the scene geometry interactively in the browser.

BibTeX

@misc{li2026recal3rreliabilitycalibratedlearningrates,
  title={ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction},
  author={Xinze Li and Yiyuan Wang and Pengxu Chen and Wentao Fan and Weifeng Su and Weisi Lin and Wentao Cheng},
  year={2026},
  eprint={2607.05356},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2607.05356},
}

Contact

For questions about the paper, please contact Wentao Cheng: wentaocheng@bnbu.edu.cn.