Recording

The core SDK includes a built-in data recorder which leverages FFmpeg for video encoding. The data created using the recorder can used for

  • Troubleshooting (together with Spectacular AI support)

  • Post-processing and exporting for NeRF & 3DGS training

  • Other research and development purposes

The easiest way to use the recorder is through the pre-built recording tools that are included with the SDK packages, e.g., sai-record-realsense or sai-cli record oak.

It can also be enabled in any software integrating the SDK by setting the recordingFolder in the device-specific Configuration class (or the VIO builder). Refer to the wrapper pages for more details and examples.

In all cases, remember to first install FFmpeg.

Data format

The recording format created by the SDK and Spectacular Rec applications is documented here. It is based on encoded videos and JSONL, making it a convenient and effective choice for high-resolution, high-frequency multi-camera, multi-sensor data. In particular, we can efficiently and economically store

  • Multi-camera RGB or monochrome data

  • Depth data (via PNGs or FFV1)

  • IMU, synchronized with the camera frames

  • Other sensors, e.g., barometer or GPS

A Spectacular AI SDK recording is a folder (possibly compressed to a zip file) that contains the files:

  • data.jsonl for sensor data and frame metadata

  • data.mkv (or mp4) and possibly data2.*, data3.* in multi-camera cases

  • calibration.json the current calibration file

  • vio_config.yaml the parameters used for the recording

  • possibly other auxiliary files

Recording settings

It is possible to control FFmpeg encoding parameters using the --ffmpeg_codec command line option (for recording tools) or the ffmpegVideoCodec parameter in vio_config.yaml. For example, the following option disables all encoding and writes raw video, which minimizes the CPU consumption overhead caused by enabling recording (on the other hand, this requires a lot of disk space and decent write speed). The contents of this argument are inserted right after -c:v flag, i.e. with option below will be same as -c:v rawvideo if you use ffmpeg executable via command line:

ffmpegVideoCodec: rawvideo

The parameter can also be used to enable platform-specific codecs. When using hardware encoding, it’s recommended to use transport stream (TS) container. Otherwise the encoder will likely conflict with the in-memory write buffer and fail. The following examples enable hardware-accelerated video recording on specific platforms:

Raspberry Pi 4:

ffmpegVideoCodec: "h264_v4l2m2m -b:v 16M"
ffmpegVideoContainer: "ts"

Nvidia Jetson Xavier platforms (not Orin Nano). May require a custom FFmpeg build

ffmpegVideoCodec: "h264_nvenc -b:v 16M"
ffmpegVideoContainer: "ts"

Any hardware encoder supported by FFmpeg installed on your system should work. For easier debugging, you can enable additional FFmpeg logging with:

ffmpegDebug: True

The SDK uses an in-memory buffer with FFmpeg to avoid small disk I/O stalls impacting performance. You can incease this from the default 20 MB per camera if you encounter any of the warnings “ffmpegWriteBuffer is full”, “ffmpegWriteBuffer is too small”, “ffmpegWriteBuffer exceeded”, notice long I/O stalls, or use high frame rates or high resolutions. For example:

ffmpegWriteBufferCapacityMegabytes: 50

You can disable this buffer and write straight to disk with following setting, this is not recommended when live tracking is enabled, because I/O blocking will also block tracking. This may help if you encounter issues with hardware encoding and setting ffmpegVideoContainer to "ts" did not help.

ffmpegUseWriteBuffer: False