Spectacular AI SDK documentation

The Spectacular AI SDK fuses data from cameras, IMUs, and various optional sensors to track a device’s position, orientation, and velocity in real time. Its main use cases are drone (UAV) and ground-vehicle navigation in GPS-denied environments and inside-out tracking for AR/VR/XR headsets.

In addition, the SDK can be applied to a wide range of other robotics, virtual reality, and computer vision research scenarios. It offers real-time 3D mapping capabilities (SLAM) and supports rapid prototyping with popular off-the-shelf devices (e.g., OAK-D). The core SDK is hardware-agnostic and can be integrated into any embedded platform with sufficient computing power and properly configured sensors.

SDK architecture

The main parts of the SDK are documented on their own pages:

  • Core SDK: device-independent APIs shared between all devices supported by the SDK.

  • Wrappers: for off-the-shelf devices that are supported out-of-the-box (OAK-D, RealSense, Kinect, ORBBEC).

The SDK also includes a set of tools implemented on top of the APIs, such as the sai-cli process command line tool for NeRFs and Gaussian Splatting.

Operating modes

VIO. Visual-Inertial Odometry (VIO) is relative positioning of the device based on camera and IMU data, and it accumulates drift error that is proportional to the traveled distance. VIO is the key enabled for all other operation modes. If not other components are in use, the SDK is in VIO-only mode.

VISLAM mode corrects the VIO drift with loop closures: the SDK builds a map of the environment in real-time and uses it to recognize previously visited locations. This SLAM map is available through the Mapping API. VISLAM mode is ideal for inside-out tracking of AR/VR/XR HMDs and other small-scale indoor use cases.

GNSS-VIO. In this mode, the SDK is occasionally given absolute position references from an external system such as GPS. When the external inputs stop, the positioning continues with VIO in global coordinates (see GnssVioOutput), but accumulates drift until the next external input. GNSS-VIO is suitable for ground vehicle positioning that is robust to natural GPS outages (as demonstrated here). The SDK can also be used as well as short-range UAV navigation in GPS-denied environments (as demonstrated here).

VIO+VPS. Visual Positioning System (VPS) uses a fixed visual map stored onboard the device as global reference, and is used to eliminate long-term drift. The SDK currently supports maps based on satellite imagery or aerial ortho-photos. The VIO+VPS system is ideal for long-range aerial navigation in GPS-denied environments (as demonstrated here). In addition to Spectacular AI’s own VPS, the SDK supports integration with 3rd-party VPS systems.

AprilTags. For indoor and robotics use cases, the SDK can also utilize AprilTag fiducial markers for a global map. See here for more information.

System requirements

Operating system. Windows or Linux. For Linux, Ubuntu 18+ is recommended, but other distribution (e.g., Yocto) can be supported as well. If the binaries do not work out-of-the-box, contact us for commercial support.

Architecture Binaries for x86-64 are available on the SDK releases. For Linux 64-bit ARM binaries are also available with a commercial license. Contact us for more info.

CPU. The SDK is designed to work in real-time on Cortex-A57 type or better embedded processors. This includes NVidia Jetson Nano and Raspberry Pi 4 (A72). Slower processors, such as Cortex-A53 work under certain conditions or with reduced accuracy and functionality. At least two A5x or better cores are required.

RAM. 6-DoF pose tracking works with less than 50MB of RAM. However, certain optional features or configuration changes related to the Mapping API increase can the RAM consumption.

Software dependencies. The Core SDK itself has zero non-trivial external software dependencies. However, Wrappers, SDK examples and tools built on the SDK have additional dependencies, which are not all designed to work on embedded platforms (especially the Python-based 3D visualizations). See the relevant documentation sub-pages for more info.

Development dependencies

For development purposes, we recommend a relatively recent Linux or Windows laptop computer.

Windows

The following setup is recommended to run the SDK examples

For C++ development, you additionally need

Linux

You need at least Git, and Python and Pip

sudo apt update
sudo apt install python-is-python3 python3-pip git

For C++ development, you also need a relatively recent compiler (GCC or Clang) and CMake, e.g.,

sudo apt install cmake clang build-essential

FFmpeg

To use the recoding, replay API or tools depending on these APIs, you need to install FFmpeg. Linux

apt install ffmpeg

On Windows, see here. FFmpeg must be in your PATH so that ffmpeg works on the command line.