Gaussian Splatting on the Move

Blur and Rolling Shutter Compensation for Natural Camera Motion

ECCV 2024

Otto Seiskari1, Jerry Ylilammi1, Valtteri Kaatrasalo1, Pekka Rantalankila1,
Matias Turkulainen2, Juho Kannala1,3,4, Arno Solin1,3
1Spectacular AI, 2ETH Zürich, 3Aalto University, 4University of Oulu
Paper Video Code Data

T L D R :

Crisp Gaussian Splatting reconstructions from blurry and wobbly smartphone captures.

Motion blur and rolling shutter compensation for 3DGS using VIO IMU data, pose refinement, and a differentiable image formation model. Results demonstrated on synthetic and casually captured smartphone data.

Motion blur

Rolling shutter

First row: demonstrates simulated motion blur and rolling shutter effects as a post-processing step for a 3DGS model. Effects are exaggerated for visual demonstration purposes.

Second row: shows 3DGS renders from training data affected by motion blur and rolling shutter effects. We compare against the baseline Splatfacto and our model.

Abstract

High-quality scene reconstruction and novel view synthesis based on Gaussian Splatting (3DGS) typically require steady, high-quality photographs, often impractical to capture with handheld cameras. We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data suffering from motion blur and rolling shutter distortion. Our approach is based on detailed modelling of the physical image formation process and utilizes velocities estimated using visual-inertial odometry (VIO). Camera poses are considered non-static during the exposure time of a single image frame and camera poses are further optimized in the reconstruction process. We formulate a differentiable rendering pipeline that leverages screen space approximation to efficiently incorporate rolling-shutter and motion blur effects into the 3DGS framework. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods, thereby advancing 3DGS in naturalistic settings.

Real data

Casual iPhone 15 captures with moderate motion blur. Comparing Splatfacto to our method.

Google Pixel 5 and Samsung Galaxy S20, Android phones with a typical long rolling shutter readout time.
Comparing Splatfacto to our method with rolling shutter and motion blur compensation.

Table: Numerical results and ablation study

Splatfacto \MB \RS \P.opt. \V.opt. \VIO CVR Ours
Motion blur - - ✓ ✓ ✓ ✓ ✓ ✓
Rolling shut. - ✓ - ✓ ✓ ✓ CVR ✓
Pose opt. - ✓ ✓ - ✓ ✓ ✓ ✓
Velocity opt. - ✓ ✓ ✓ - ✓ ✓ ✓
VIO vel. init. ✓ ✓ ✓ - ✓
iphone-lego1 28.05 28.12 29.20 28.59 28.71 29.03 23.26 29.20
iphone-lego2 27.85 27.88 27.95 27.39 28.15 28.55 26.45 27.95
iphone-lego3 23.75 23.71 24.50 24.10 24.22 23.78 22.45 24.50
iphone-pots1 28.32 28.58 29.10 28.91 28.93 29.18 24.44 29.10
iphone-pots2 27.25 27.39 28.00 27.68 26.66 27.81 23.64 28.00
pixel5-lamp 28.22 30.91 28.38 28.77 29.75 29.70 31.95 30.46
pixel5-plant 26.57 27.41 27.45 26.81 27.37 27.69 28.20 27.90
pixel5-table 28.93 30.82 29.25 30.33 31.16 31.47 32.42 31.86
s20-bike 27.35 27.74 27.57 27.58 27.58 27.72 26.62 28.93
s20-bikerack 25.98 29.39 27.92 26.23 26.09 28.92 27.77 29.74
s20-sign 23.71 25.93 24.47 25.44 25.54 26.28 24.19 26.84
average 26.91 27.99 27.62 27.44 27.65 28.19 26.49 28.59

Synthetic data

Our results on a re-rendered and extended version of the Deblur-NeRF dataset (Zenodo link)

Motion blur

Rolling shutter

Pose noise

Table: Comparison to previous work. This uses the BAD-NeRF re-render of the synthetic data.

Cozyroom Factory Pool Tanabata Trolley
PSNR SSIM LPIPS PSNR SSIM LPIPS PSNR SSIM LPIPS PSNR SSIM LPIPS PSNR SSIM LPIPS
Splatfacto 24.93 .802 .225 21.28 .598 .440 27.88 .763 .281 18.52 .533 .433 19.47 .564 .387
MPR+Splatf. 29.26 .894 .093 23.38 .737 .246 30.96 .867 .176 22.77 .773 .227 26.49 .854 .185
Deblur-NeRF 29.88 .890 .075 26.06 .802 .211 30.94 .840 .169 22.56 .764 .229 25.78 .812 .180
BAD-NeRF 30.97 .901 .055 31.65 .904 .123 31.72 .858 .115 23.82 .831 .138 28.25 .873 .091
Ours 31.80 .945 .032 30.54 .946 .078 32.08 .890 .075 24.79 .912 .079 30.16 .933 .044

BibTeX

@misc{seiskari2024gaussian,
      title={Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion}, 
      author={Otto Seiskari and Jerry Ylilammi and Valtteri Kaatrasalo and Pekka Rantalankila and Matias Turkulainen and Juho Kannala and Arno Solin},
      year={2024},
      eprint={2403.13327},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}