Spectacular AI SDK documentation

The Spectacular AI SDK fuses data from cameras, IMUs, and various optional sensors to track a device’s position, orientation, and velocity in real time. Its main use cases are drone (UAV) and ground-vehicle navigation in GPS-denied environments and inside-out tracking for AR/VR/XR headsets.

In addition, the SDK can be applied to a wide range of other robotics, virtual reality, and computer vision research scenarios. It offers real-time 3D mapping capabilities (SLAM) and supports rapid prototyping with popular off-the-shelf devices (e.g., OAK-D). The core SDK is hardware-agnostic and can be integrated into any embedded platform with sufficient computing power, and properly configured sensors.

SDK architecture

The main parts of the SDK are documented on their own pages:

  • Core SDK: the device-independent APIs shared between all devices supported by the SDK.

  • Wrappers: for the off-the-shelf devices that are supported out-of-the-box (OAK-D, RealSense, Kinect, ORBBEC).

The SDK also includes a set of tools implemented on top of the APIs, such as the sai-cli process command-line tool for NeRFs and Gaussian Splatting.

Operating modes

VIO. Visual-Inertial Odometry (VIO) is relative positioning of the device based on camera and IMU data, and it accumulates drift error that is proportional to the traveled distance. VIO is the key enabler for all other operating modes. If no other components are in use, the SDK operates in VIO-only mode.

VISLAM mode corrects VIO drift with loop closures: the SDK builds a real-time map of the environment and uses it to recognize previously visited locations. This SLAM map is available through the Mapping API. VISLAM mode is ideal for inside-out tracking of AR/VR/XR HMDs and other small-scale indoor use cases.

GNSS-VIO. In this mode, the SDK occasionally receives absolute position references from an external system, such as GPS. When external inputs stop, positioning continues with VIO in global coordinates (see GnssVioOutput), but accumulates drift until the next external input. GNSS-VIO is suitable for ground vehicle positioning that is robust to natural GPS outages (as demonstrated in this video). This mode can also be used for short-range UAV navigation in GPS-denied environments (as demonstrated in this video).

VIO+VPS. Visual Positioning System (VPS) uses a fixed visual map stored onboard the device as a global reference and is used to eliminate long-term drift. The SDK currently supports maps based on satellite imagery or aerial orthophotos. The VIO+VPS system is ideal for long-range aerial navigation in GPS-denied environments (as demonstrated in this video). In addition to Spectacular AI’s own VPS, the SDK supports integration with 3rd-party VPS systems.

AprilTags. For indoor and robotics use cases, the SDK can also utilize AprilTag fiducial markers for a global map. See this document for more information.

System requirements

Operating system. Windows or Linux. For Linux, Ubuntu 18+ is recommended, but other distributions (e.g., Yocto) can be supported as well. If the binaries do not work out-of-the-box, contact us for commercial support.

Architecture. Binaries for x86-64 are available on the SDK releases. For Linux, 64-bit ARM binaries are also available with a commercial license. Contact us for more information.

CPU. The SDK is designed to work in real-time on Cortex-A57 type or better embedded processors. This includes NVIDIA Jetson Nano and Raspberry Pi 4 (A72). Slower processors, such as Cortex-A53, work under certain conditions or with reduced accuracy and functionality. At least two A5x or better cores are required.

RAM. 6-DoF pose tracking works with less than 50MB of RAM. However, certain optional features or configuration changes related to the Mapping API can increase the RAM consumption.

Software dependencies. The Core SDK itself has zero non-trivial external software dependencies; however, Wrappers, SDK examples, and tools built on the SDK have additional dependencies, which are not all designed to work on embedded platforms (especially the Python-based 3D visualizations). See the relevant documentation sub-pages for more information.

Development dependencies

For development purposes, we recommend a relatively recent Linux or Windows laptop computer.

Windows

The following setup is recommended to run the SDK examples:

For C++ development, you will also need:

Linux

You will need at least Git, Python, and Pip.

sudo apt update
sudo apt install python-is-python3 python3-pip git

For C++ development, you also need a relatively recent compiler (GCC or Clang) and CMake, e.g.,

sudo apt install cmake clang build-essential

FFmpeg

To use the recording and replay APIs, or tools that depend on them, you will need to install FFmpeg. On Linux:

apt install ffmpeg

On Windows, see this guide. FFmpeg must be in your PATH so that the ffmpeg command works on the command line.