ORBBEC

Spectacular AI C++ plugin for ORBBEC cameras

Supported devices

The SDK has been verified to work with these devices:

Femto Mega

Femto Bolt

Astra 2

Other devices should work with minimal changes. Please see the unsupported devices section.

Quick start

First, download and unpack the relevant package on the SDK releases page.

If you have not used the Orbbec device before, you will need to setup udev rules, run the following script (or follow the instructions in OrbbecSDK):
```
./bin/3rdparty/OrbbecSDK/install_udev_rules.sh
```
Attach your Orbbec device to a USB3 port

Run the JSONL example (smoke test):

cd bin
./vio_jsonl
# Now you should see rapidly flowing JSONL text
# press Ctrl+C to exit

Run the Python example (in the bin directory):

pip install matplotlib # install dependencies
python vio_visu.py

If you have not used the Orbbec device before, you will need to setup timestamp registration. Please follow the official instructions in OrbbecSDK.
Attach your Orbbec device to a USB3 port

Run the JSONL example (smoke test):

cd bin
./vio_jsonl.exe
# Now you should see rapidly flowing JSONL text
# press Ctrl+C to exit

Run the Python example (in the bin directory):

pip install matplotlib # install dependencies
python vio_visu.py

Recording data

Orbbec data can be recorded using the sai-record-orbbec tool included in the SDK package (in the bin directory). First connect an Orbbec device over USB3, then run (add .exe on Windows):

./sai-record-orbbec # Record with visualization, requires pip install spectacularAI[full]
./sai-record-orbbec --recording_only --no_preview # More lightweight
./sai-record-orbbec --output /path/to/output/folder # Set recording directory
./sai-record-orbbec -h # See all options

The source code of recording tool is also available at https://github.com/SpectacularAI/sdk/tree/main/cpp/orbbec which allows customizing the recording tool, if necessary.

Unsupported devices

The SDK should work on most Orbbec devices, but has only been tested with devices listed here. Other devices require setting certain parameters manually. Namely, the IMU-to-camera matrix needs to be set if the device model was not recognized by the SDK. For example:

// Create Spectacular AI orbbec plugin configuration (depends on device type).
ob::Pipeline obPipeline;
spectacularAI::orbbecPlugin::Configuration config(obPipeline);
config.imuToCameraRgb = {{
   { 1, 0, 0, 0 },
   { 0, 1, 0, 0 },
   { 0, 0, 1, 0 },
   { 0, 0, 0, 1 }
}};

Installation as a library

The following commands should be run in the folder where you have unpacked the SDK release package.

First install the development dependencies.
Download the OrbbecSDK

Select where you want the library installed, e.g.,:

export MY_INSTALL_PREFIX=~/.local
# or: export MY_INSTALL_PREFIX=`pwd`/installed

Run make PREFIX=$MY_INSTALL_PREFIX install (or sudo make install)

Build the vio_jsonl example using CMake:

make PREFIX=$MY_INSTALL_PREFIX ORBBEC_DIR=<path/to/OrbbecSDK> example

Run the vio_jsonl binary as a smoke test.

First install the development dependencies.
Download the OrbbecSDK
Build vio_jsonl example using CMake. The process is a bit involved so the commands are collected in ./examples/build_windows.sh. In PowerShell:
```
cd examples
sh ./build_windows.sh ..\lib\cmake\spectacularAI <path\to\OrbbecSDK>
```
Finally running vio_jsonl.exe should give you the pose of the Orbbec device in real time (smoke test):
```
cd target/Release
./vio_jsonl.exe
```

Examples

The following instructions show how to compile and run the Spectacular AI ORBBEC SDK examples.

(Optional) install the SDK as a library (see above).
Clone the SDK examples repository:
git clone --recursive https://github.com/SpectacularAI/sdk-examples.git

Run the following commands in the sdk-examples/cpp/orbbec directory.

Build the examples using CMake:

mkdir target
cd target
cmake -DspectacularAI_orbbecPlugin_DIR=<path/to/spectacularAI_orbbecPlugin/lib/cmake/spectacularAI/> -DOrbbecSDK_DIR=<path/to/OrbbecSDK> ..
make

The -DspectacularAI_orbbecPlugin_DIR option is not needed if you have used sudo make install for the SDK.

Build the vio_jsonl example using CMake. The process is a bit involved so the commands are collected in ./build_windows.sh. In PowerShell:

sh ./build_windows.sh <path\to\spectacularAI_orbbecPlugin\lib\cmake\spectacularAI> <path\to\OrbbecSDK>

API Reference

The classes specific to the Spectacular AI ORBBEC Wrapper are documented below. For the main output classes, refer to the C++ documentation in Core SDK.

Defined in #include <spectacularAI/orbbec/configuration.hpp>

namespace spectacularAI

namespace orbbecPlugin

struct Configuration

Plugin and Spectacular AI SDK configuration variables.

Public Functions

Configuration(): Default configuration, works with Femto Mega

Configuration(ob::Pipeline &obPipeline): Device specific Configuration. Tested: Femto Mega, Femto Bolt, Astra2

Public Members

bool useSlam = true: Enable SLAM, required for ICP

bool fastVio = false: Enable more lightweight VIO parameters

bool alignedDepth = false

Align depth images to color images using OrbbecSDK.

Note that this can affect performance if the depth image has larger field of view than the color camera.

std::string recordingFolder = "": If not empty, the session will be recorded to the given folder

bool recordingOnly = false: Enable recording only mode (more lightweight)

bool postProcessFinalMap = false: If true, final mapping API output is refined using post processing.

std::string aprilTagPath = "": Path to .json file with AprilTag information. AprilTag detection is enabled when not empty. For the file format see: https://github.com/SpectacularAI/docs/blob/main/pdf/april_tag_instructions.pdf Note: sets useSlam=true

std::pair<int, int> rgbResolution = std::make_pair(1280, 720): RGB video resolution

std::pair<int, int> depthResolution = std::make_pair(512, 512): Depth video resolution

int cameraFps = 30: Camera FPS

OBAccelSampleRate accFrequencyHz = OBAccelSampleRate::OB_SAMPLE_RATE_1_KHZ: Accelerometer frequency

OBGyroSampleRate gyroFrequencyHz = OBGyroSampleRate::OB_SAMPLE_RATE_1_KHZ: Gyroscope frequency

OBAccelFullScaleRange accRange = OBAccelFullScaleRange::OB_ACCEL_FS_4g: Accelerometer range

OBGyroFullScaleRange gyroRange = OBGyroFullScaleRange::OB_GYRO_FS_1000dps: Gyroscope range

bool ignoreOutputs = false: If false, user should read vio outputs from the Session. If true, the SDK discards vio outputs. Useful when VIO outputs are not needed: e.g. when using mapping API or recording.

Matrix4d imuToCameraRgb = {}: Option to override IMU to RGB camera extrinsic matrix. Required if not using Femto Mega/Bolt or Astra 2.

std::map<std::string, std::string> internalParameters: Internal SDK parameter overrides (key-value pairs). Not safe for unsanitized user input.

Defined in #include <spectacularAI/orbbec/pipeline.hpp>

namespace spectacularAI

namespace orbbecPlugin

class Pipeline

The Pipeline object stores information about the device and also sets the default configuration, e.g., enables and configures necessary Orbbec streams and sensors. The actual VIO reading is started with startSession.

Public Functions

Pipeline(ob::Pipeline &obPipeline): Create a pipeline with the default configuration.

Pipeline(ob::Pipeline &obPipeline, const Configuration &config): Create a pipeline with custom configuration

Pipeline(ob::Pipeline &obPipeline, const Configuration &config, std::function<void(mapping::MapperOutputPtr)> onMapperOutput): Create a pipeline with custom configuration with Mapping API enabled

std::unique_ptr<Session> startSession(): Start a new VIO session on the background.

~Pipeline()

Private Members

std::unique_ptr<impl> pimpl

Defined in #include <spectacularAI/orbbec/session.hpp>

namespace spectacularAI

namespace orbbecPlugin

struct Session

Public Functions

virtual bool hasOutput() const = 0: Check if new output is available.

virtual std::shared_ptr<const VioOutput> getOutput() = 0: Get output from the queue, if available. If not returns {}.

virtual std::shared_ptr<const VioOutput> waitForOutput() = 0: Wait until new output is available and then return it.

virtual void addTrigger(double t, int tag) = 0

Add an external trigger input. Causes additional output corresponding to a certain timestamp to be generated.

Parameters:

t – timestamp, monotonic float seconds
tag – additonal tag to indentify this particular trigger event. The default outputs corresponding to input camera frames have a tag 0.

virtual void ignoreOutputs(bool ignore) = 0: DEPRECATED! Use Configuration.ignoreVioOutputs instead.

virtual ~Session()