Recording
The core SDK includes a built-in data recorder which leverages FFmpeg for video encoding. The data created using the recorder can used for
Troubleshooting (together with Spectacular AI support)
Post-processing and exporting for NeRF & 3DGS training
Other research and development purposes
The easiest way to use the recorder is through the pre-built recording tools that are included with the SDK packages, e.g., sai-record-realsense
or sai-cli record oak
.
It can also be enabled in any software integrating the SDK by setting the recordingFolder
in the device-specific Configuration
class (or the VIO builder). Refer to the wrapper pages for more details and examples.
In all cases, remember to first install FFmpeg.
Data format
The recording format created by the SDK and Spectacular Rec applications is documented here. It is based on encoded videos and JSONL, making it a convenient and effective choice for high-resolution, high-frequency multi-camera, multi-sensor data. In particular, we can efficiently and economically store
Multi-camera RGB or monochrome data
Depth data (via PNGs or FFV1)
IMU, synchronized with the camera frames
Other sensors, e.g., barometer or GPS
A Spectacular AI SDK recording is a folder (possibly compressed to a zip file) that contains the files:
data.jsonl
for sensor data and frame metadatadata.mkv
(ormp4
) and possiblydata2.*
,data3.*
in multi-camera casescalibration.json
the current calibration filevio_config.yaml
the parameters used for the recordingpossibly other auxiliary files
Recording settings
It is possible to control FFmpeg encoding parameters using the --ffmpeg_codec
command line option (for recording tools) or the ffmpegVideoCodec
parameter in vio_config.yaml
. For example, the following option disables all encoding and writes raw video, which minimizes the CPU consumption overhead caused by enabling recording (on the other hand, this requires a lot of disk space and decent write speed). The contents of this argument are inserted right after -c:v
flag, i.e. with option below will be same as -c:v rawvideo
if you use ffmpeg
executable via command line:
ffmpegVideoCodec: rawvideo
The parameter can also be used to enable platform-specific codecs. When using hardware encoding, it’s recommended to use transport stream (TS) container. Otherwise the encoder will likely conflict with the in-memory write buffer and fail. The following examples enable hardware-accelerated video recording on specific platforms:
Raspberry Pi 4:
ffmpegVideoCodec: "h264_v4l2m2m -b:v 16M"
ffmpegVideoContainer: "ts"
Nvidia Jetson Xavier platforms (not Orin Nano). May require a custom FFmpeg build
ffmpegVideoCodec: "h264_nvenc -b:v 16M"
ffmpegVideoContainer: "ts"
Any hardware encoder supported by FFmpeg installed on your system should work. For easier debugging, you can enable additional FFmpeg logging with:
ffmpegDebug: True
The SDK uses an in-memory buffer with FFmpeg to avoid small disk I/O stalls impacting performance. You can incease this from the default 20 MB per camera if you encounter any of the warnings “ffmpegWriteBuffer is full”, “ffmpegWriteBuffer is too small”, “ffmpegWriteBuffer exceeded”, notice long I/O stalls, or use high frame rates or high resolutions. For example:
ffmpegWriteBufferCapacityMegabytes: 50
You can disable this buffer and write straight to disk with following setting, this is not recommended when live tracking is enabled, because I/O blocking will also block tracking. This may help if you encounter issues with hardware encoding and setting ffmpegVideoContainer
to "ts"
did not help.
ffmpegUseWriteBuffer: False