Spectacular AI C++ plugin for ORBBEC cameras

Supported devices

The SDK has been verified to work with these devices:

Other devices should work with minimal changes. Please see the unsupported devices section.

Quick start

First, download and unpack the relevant package on the SDK releases page.

  1. If you have not used the Orbbec device before, you will need to setup udev rules, run the following script (or follow the instructions in OrbbecSDK):

  2. Attach your Orbbec device to a USB3 port

  3. Run the JSONL example (smoke test):

    cd bin
    # Now you should see rapidly flowing JSONL text
    # press Ctrl+C to exit
  4. Run the Python example (in the bin directory):

    pip install matplotlib # install dependencies
    python vio_visu.py

Recording data

Orbbec data can be recorded using the sai-record-orbbec tool included in the SDK package (in the bin directory). First connect an Orbbec device over USB3, then run (add .exe on Windows):

./sai-record-orbbec # Record with visualization, requires pip install spectacularAI[full]
./sai-record-orbbec --recording_only --no_preview # More lightweight
./sai-record-orbbec --output /path/to/output/folder # Set recording directory
./sai-record-orbbec -h # See all options

The source code of recording tool is also available at https://github.com/SpectacularAI/sdk/tree/main/cpp/orbbec which allows customizing the recording tool, if necessary.

Unsupported devices

The SDK should work on most Orbbec devices, but has only been tested with devices listed here. Other devices require setting certain parameters manually. Namely, the IMU-to-camera matrix needs to be set if the device model was not recognized by the SDK. For example:

// Create Spectacular AI orbbec plugin configuration (depends on device type).
ob::Pipeline obPipeline;
spectacularAI::orbbecPlugin::Configuration config(obPipeline);
config.imuToCameraRgb = {{
   { 1, 0, 0, 0 },
   { 0, 1, 0, 0 },
   { 0, 0, 1, 0 },
   { 0, 0, 0, 1 }

Installation as a library

The following commands should be run in the folder where you have unpacked the SDK release package.

  1. First install the development dependencies.

  2. Download the OrbbecSDK

  3. Select where you want the library installed, e.g.,:

    export MY_INSTALL_PREFIX=~/.local
    # or: export MY_INSTALL_PREFIX=`pwd`/installed
  4. Run make PREFIX=$MY_INSTALL_PREFIX install (or sudo make install)

  5. Build the vio_jsonl example using CMake:

    make PREFIX=$MY_INSTALL_PREFIX ORBBEC_DIR=<path/to/OrbbecSDK> example
  6. Run the vio_jsonl binary as a smoke test.


The following instructions show how to compile and run the Spectacular AI ORBBEC SDK examples.

  1. (Optional) install the SDK as a library (see above).

  2. Clone the SDK examples repository:

    git clone --recursive https://github.com/SpectacularAI/sdk-examples.git

Run the following commands in the sdk-examples/cpp/orbbec directory.

Build the examples using CMake:

mkdir target
cd target
cmake -DspectacularAI_orbbecPlugin_DIR=<path/to/spectacularAI_orbbecPlugin/lib/cmake/spectacularAI/> -DOrbbecSDK_DIR=<path/to/OrbbecSDK> ..

The -DspectacularAI_orbbecPlugin_DIR option is not needed if you have used sudo make install for the SDK.

API Reference

The classes specific to the Spectacular AI ORBBEC Wrapper are documented below. For the main output classes, refer to the C++ documentation in Core SDK.

Defined in #include <spectacularAI/orbbec/configuration.hpp>

namespace spectacularAI
namespace orbbecPlugin
struct Configuration

Plugin and Spectacular AI SDK configuration variables.

Public Functions


Default configuration, works with Femto Mega

Configuration(ob::Pipeline &obPipeline)

Device specific Configuration. Tested: Femto Mega, Astra2

Public Members

bool useSlam = true

Enable SLAM, required for ICP

bool fastVio = false

Enable more lightweight VIO parameters

bool alignedDepth = false

Align depth images to color images using OrbbecSDK.

Note that this can affect performance if the depth image has larger field of view than the color camera.

std::string recordingFolder = ""

If not empty, the session will be recorded to the given folder

bool recordingOnly = false

Enable recording only mode (more lightweight)

bool postProcessFinalMap = false

If true, final mapping API output is refined using post processing.

std::string aprilTagPath = ""

Path to .json file with AprilTag information. AprilTag detection is enabled when not empty. For the file format see: https://github.com/SpectacularAI/docs/blob/main/pdf/april_tag_instructions.pdf Note: sets useSlam=true

std::pair<int, int> rgbResolution = std::make_pair(1280, 720)

RGB video resolution

std::pair<int, int> depthResolution = std::make_pair(512, 512)

Depth video resolution

int cameraFps = 30

Camera FPS

OBAccelSampleRate accFrequencyHz = OBAccelSampleRate::OB_SAMPLE_RATE_1_KHZ

Accelerometer frequency

OBGyroSampleRate gyroFrequencyHz = OBGyroSampleRate::OB_SAMPLE_RATE_1_KHZ

Gyroscope frequency

OBAccelFullScaleRange accRange = OBAccelFullScaleRange::OB_ACCEL_FS_4g

Accelerometer range

OBGyroFullScaleRange gyroRange = OBGyroFullScaleRange::OB_GYRO_FS_1000dps

Gyroscope range

bool ignoreOutputs = false

If false, user should read vio outputs from the Session. If true, the SDK discards vio outputs. Useful when VIO outputs are not needed: e.g. when using mapping API or recording.

Matrix4d imuToCameraRgb = {}

Option to override IMU to RGB camera extrinsic matrix. Required if not using Femto Mega/Bolt or Astra 2.

std::map<std::string, std::string> internalParameters

Internal SDK parameter overrides (key-value pairs). Not safe for unsanitized user input.

Defined in #include <spectacularAI/orbbec/pipeline.hpp>

namespace spectacularAI
namespace orbbecPlugin
class Pipeline

The Pipeline object stores information about the device and also sets the default configuration, e.g., enables and configures necessary Orbbec streams and sensors. The actual VIO reading is started with startSession.

Public Functions

Pipeline(ob::Pipeline &obPipeline)

Create a pipeline with the default configuration.

Pipeline(ob::Pipeline &obPipeline, const Configuration &config)

Create a pipeline with custom configuration

Pipeline(ob::Pipeline &obPipeline, const Configuration &config, std::function<void(mapping::MapperOutputPtr)> onMapperOutput)

Create a pipeline with custom configuration with Mapping API enabled

std::unique_ptr<Session> startSession()

Start a new VIO session on the background.


Private Members

std::unique_ptr<impl> pimpl

Defined in #include <spectacularAI/orbbec/session.hpp>

namespace spectacularAI
namespace orbbecPlugin
struct Session

Public Functions

virtual bool hasOutput() const = 0

Check if new output is available.

virtual std::shared_ptr<const VioOutput> getOutput() = 0

Get output from the queue, if available. If not returns {}.

virtual std::shared_ptr<const VioOutput> waitForOutput() = 0

Wait until new output is available and then return it.

virtual void addTrigger(double t, int tag) = 0

Add an external trigger input. Causes additional output corresponding to a certain timestamp to be generated.

  • t – timestamp, monotonic float seconds

  • tag – additonal tag to indentify this particular trigger event. The default outputs corresponding to input camera frames have a tag 0.

virtual void ignoreOutputs(bool ignore) = 0

DEPRECATED! Use Configuration.ignoreVioOutputs instead.

virtual ~Session()