OAK-D NeRF

Gaussian Splatting & NeRFs

This page has instructions for post-processing data recorded through the Spectacular AI SDK on supported devices, exporting to Nerfstudio, training NeRFs and 3DGS, and visualizing the process. The Spectacular AI mapping tool (sai-cli process) is powered by the Spectacular AI Mapping API.

Installation

These instructions assume you want to train NeRFs or 3DGS using Nerfstudio. For other uses, Nerfstudio and CUDA are not required, but you simply need Python, pip and FFmpeg.

install Nerfstudio (Requirement: a good NVidia GPU + CUDA).
Install FFmpeg. Linux apt install ffmpeg (or similar, if using another package manager). Windows: see here. FFmpeg must be in your PATH so that ffmpeg works on the command line.
In Nerfstudio’s Conda environment, install the Spectacular AI Python library with all recommended dependencies:
```
pip install spectacularAI[full]
```

Recording data

Choose your device below to see more detailed instructions for creating Spectacular AI recordings (folders or zip files):

iPhone (with or without LiDAR)

Download Spectacular Rec from App Store.
See our instruction video on YouTube on how to create recording files and transfer them to your computer.

Android (with or without ToF)

Download Spectacular Rec from Play Store.
Use like the iPhone version (tutorial here here

Note: the Android pipeline in the first app and sai-cli version is currently less stable than the iOS version and, the Gaussian Splatting accuracy is not as good as NeRFs. This will be improved in the near future.

OAK-D

Plug in the OAK-D to your laptop (or directly the computer with the heavy GPU)
Run sai-cli record oak --no_feature_tracker --resolution=800p.

If the above settings cause issues, try running sai-cli record oak instead. Coming soon: 🌈 colors.

RealSense D455/D435i

See the Recording data section under the RealSense wrapper instructions

Azure Kinect DK

See the https://spectacularai.github.io/docs/sdk/wrappers/k4a.html for more information

Orbbec

See the Recording data section under the Orbbec wrapper instructions

With OAK-D or RealSense devices, you can currently expect to be able to map “table-sized” scenes quite fast and accurately. Move slow while mapping and shoot from different angles to increase quality.

Nerfstudio export and training

First run our conversion script and then Nerstudio training as

sai-cli process INPUT_PATH --preview3d --key_frame_distance=0.05 /example/output/path/my-nerf
ns-train nerfacto --data /example/output/path/my-nerf

Where

INPUT_PATH is the dataset folder recorded using Spectacular Rec or our other recording tools (the value of recordingFolder if using the SDK directly)
/example/output/path/my-nerf (placeholder) is the output folder of this script and the input to Nerfstudio
--key_frame_distance should be set based on the recorded scene size: 0.05 (5cm) is good for small scans and 0.15 for room-sized scans.
--preview3d (optional flag) shows you a 3D preview of the point cloud and estimated trajectory (not the final ones).

If the processing gets slow, you can also try adding a --fast flag to sai-cli process to trade off quality for speed. Without the --fast flag, the processing should take around 10 minutes tops.

Gaussian Splatting

Update Nerfstudio and train as

ns-train gaussian-splatting --data /example/output/path/my-nerf

To use the resulting “splats” in other tools, first export as PLY

ns-export gaussian-splat \
    --load-config outputs/my-nerf/gaussian-splatting/DATE/config.yaml
    --output-dir exports/splats

Then copy the the file exports/point_cloud.ply. Examples:

Edit in Super Splat (splat colors may look wrong here)
Export to .splat or stand-alone HTML using SpectacularAI/point-cloud-tools
View or embed .splat to a web page using gsplat.js

Export mapping API data to other tools

As an alternative to Nerfstudio, the pose and image data computed by the Spectacular AI Mapping API can also be exported to:

Nvidia Instant NGP: see replay_to_instant_ngp.py
https://github.com/wanmeihuali/taichi_3d_gaussian_splatting (use sai-cli process --format=taichi ...)
Other tools that use a similar COLMAP-like folder structure than Nerfstudio (sai-cli process default output format)

The export process can also be customized by modifying the source code of sai-cli process which can also be used as a standalone Python script.

License note

Spectacular AI SDK is free to use for non-commercial purposes. Contact us for commercial licensing (e.g., running this in your own cloud service). Nerfstudio and FFMpeg are used under their own licenses.