Recording

The core SDK includes a built-in data recorder that leverages FFmpeg for video encoding. The data created using the recorder can be used for

  • Troubleshooting (together with Spectacular AI support)

  • Post-processing and exporting for NeRF & 3DGS training

  • Other research and development purposes

The easiest way to use the recorder is through the pre-built recording tools that are included with the SDK packages, e.g., sai-record-realsense or sai-cli record oak.

It can be enabled in any software integrating the SDK by setting the recordingFolder in the device-specific Configuration class (or the VIO builder), Refer to the wrapper pages for more details and examples.

In all cases, remember to first install FFmpeg.

Data format

The recording format created by the SDK and Spectacular Rec applications is documented in this page. It is based on encoded videos and JSONL, making it a convenient and effective choice for high-resolution, high-frequency multi-camera, multi-sensor data. In particular, the SDK can efficiently and economically store:

  • Multi-camera RGB or monochrome data

  • Depth data (via PNGs or FFV1)

  • IMU, synchronized with the camera frames

  • Other sensors, e.g., barometer or GPS

A Spectacular AI SDK recording is a folder (possibly compressed into a zip file) that contains the following files:

  • data.jsonl for sensor data and frame metadata

  • data.mkv (or mp4), and possibly data2.*, data3.* in multi-camera cases

  • calibration.json - the current calibration file

  • vio_config.yaml - the parameters used for the recording

  • possibly other auxiliary files

Recording settings

FFmpeg encoding parameters can be controlled using the --ffmpeg_codec command line option (for recording tools) or the ffmpegVideoCodec parameter in vio_config.yaml. For example, the following option disables all encoding and writes raw video, which minimizes the CPU consumption overhead caused by enabling recording. (On the other hand, this requires a lot of disk space and a decent write speed.) The contents of this argument are inserted right after the -c:v flag; for example, the option below will be the same as -c:v rawvideo if you use the ffmpeg executable via the command line:

ffmpegVideoCodec: rawvideo

The parameter can also be used to enable platform-specific codecs. When using hardware encoding, it is recommended to use a transport stream (TS) container. Otherwise, the encoder will likely conflict with the in-memory write buffer and fail. The following examples enable hardware-accelerated video recording on specific platforms:

Raspberry Pi 4:

ffmpegVideoCodec: "h264_v4l2m2m -b:v 16M"
ffmpegVideoContainer: "ts"

Nvidia Jetson Xavier platforms (not Orin Nano). May require a custom FFmpeg build

ffmpegVideoCodec: "h264_nvenc -b:v 16M"
ffmpegVideoContainer: "ts"

Any hardware encoder supported by FFmpeg installed on your system should work. For easier debugging, you can enable additional FFmpeg logging with:

ffmpegDebug: True

The SDK uses an in-memory buffer with FFmpeg to avoid small disk I/O stalls impacting performance. You can increase this from the default 20 MB per camera if you encounter any of the warnings “ffmpegWriteBuffer is full”, “ffmpegWriteBuffer is too small”, “ffmpegWriteBuffer exceeded”, notice long I/O stalls, or use high frame rates or high resolutions. For example:

ffmpegWriteBufferCapacityMegabytes: 50

You can disable this buffer and write straight to disk with the following setting. This is not recommended when live tracking is enabled, because I/O blocking will also block tracking. This may help if you encounter issues with hardware encoding and setting ffmpegVideoContainer to "ts" does not help.

ffmpegUseWriteBuffer: False