RealSense

Spectacular AI C++ plugin for Intel RealSense D455 and D435i cameras (note: D435 without the “i” does not work).

Quick start

First, download and unpack the spectacularAI_realsensePlugin_cpp_non-commercial_* archive from the SDK releases page.

LinuxWindows

1. If you have not used the RealSense device before, you will need to setup udev rules, run:

   ./bin/3rdparty/librealsense/setup_udev_rules.sh
   
   # or run from the librealsense GitHub repository
   ./scripts/setup_udev_rules.sh
   

2. Attach your RealSense D4XX or D3XX device to a USB3 port, using a USB3 cable

3. In the extracted directory for your platform, run the pre-compiled example binary:

   cd bin
   ./vio_jsonl
   # Now you should see rapidly flowing JSONL text
   # press Ctrl+C to exit
   

Recording data

RealSense data can be recorded using the sai-record-realsense tool included in the SDK package (in the bin directory). First connect a RealSense device over USB3, then run (add .exe on Windows):

./sai-record-realsense # Record with visualization, requires pip install spectacularAI[full]
./sai-record-realsense --recording_only --no_preview # More lightweight
./sai-record-realsense --output /path/to/output/folder # Set recording directory
./sai-record-realsense -h # See all options

The source code of recording tool is also available at https://github.com/SpectacularAI/sdk/blob/main/cpp/realsense/record.cpp which allows customizing the recording tool, if necessary.

Using as a library

The following commands should be run in the folder where you have unpacked the SDK release package.

LinuxWindows

1. Install the development dependencies.

2. Select where you want the library installed, e.g.,:

   export MY_INSTALL_PREFIX=~/.local
   

3. Run make PREFIX=$MY_INSTALL_PREFIX install (or sudo make install)

4. Install librealsense: https://github.com/IntelRealSense/librealsense

5. Build the vio_jsonl example using CMake:

   make MY_INSTALL_PREFIX=~/.local example
   

Examples

The following instructions show how to compile and run the Spectacular AI RealSense SDK examples.

1. (Optional) install the SDK as a library (see above).

2. Clone the SDK examples repository:

   git clone --recursive https://github.com/SpectacularAI/sdk-examples.git
   

Run the following commands in the sdk-examples/cpp/realsense directory.

LinuxWindows

Build the examples using CMake:

mkdir target
cd target
cmake -DBUILD_MAPPER=OFF -DspectacularAI_realsensePlugin_DIR=<path/to/spectacularAI_realsensePlugin/lib/cmake/spectacularAI/> ..
make

The -DspectacularAI_realsensePlugin_DIR option is not needed if you have used sudo make install for the SDK.

API Reference

The classes specific to the RealSense plugin are documented below. For the main output classes, refer to the C++ documentation in Core SDK.

Configuration

Defined in #include <spectacularAI/realsense/configuration.hpp>

namespace spectacularAI

namespace rsPlugin

struct Configuration

Plugin and Spectacular AI VIO SDK configuration variables.

Public Members

bool useStereo = true

bool useRgb = true

bool fastVio = false

bool alignedDepth = false

bool useSlam = true

bool useIcp = false

std::string inputResolution = "400p"

std::string mapSavePath = ""

std::string mapLoadPath = ""

std::string aprilTagPath = ""

std::string recordingFolder = ""

bool recordingOnly = false

bool postProcessFinalMap = false

std::map<std::string, std::string> internalParameters

Internal SDK parameter overrides (key-value pairs). Not safe for unsanitized user input.

Pipeline

Defined in #include <spectacularAI/realsense/pipeline.hpp>

namespace rs2

namespace spectacularAI

namespace rsPlugin

class Pipeline

The Pipeline object stores information about the device and also sets the default configuration, e.g., enables and configures necessary RealSense streams and sensors. The actual VIO reading is started with startSession. It is possible to change the RealSense configuration between calling configureX and startSession, but the VIO tracking is not guaranteed to be compatible with such customized configurations.

Public Functions

Pipeline()

Create a pipeline with the default configuration.

Pipeline(const Configuration &config)

Create a pipeline with custom configuration

Pipeline(const Configuration &config, std::function<void(mapping::MapperOutputPtr)> onMapperOutput)

Create a pipeline with custom configuration with Mapping API enabled

void configureDevice(rs2::device &device)

Configure the RealSense device for VIO

void configureStreams(rs2::config &config)

Set up the RealSense config for VIO

void setMapperCallback(const std::function<void(spectacularAI::mapping::MapperOutputPtr)> &onMapperOutput)

DEPRECATED: set the callback in Pipeline constructor instead.

std::unique_ptr<Session> startSession(rs2::config &config)

Start a new VIO session on the background. Internally creates an rs2::pipeline and starts it.

std::unique_ptr<Session> startSession(rs2::config &config, const std::function<void(const rs2::frame &frame)> &callback)

Start a new VIO session on the background. Also invoke the given callback on each rs2::frame. Performing heavy computations directly in the callback is not recommended.

~Pipeline()

Private Members

std::unique_ptr<impl> pimpl

Session

Defined in #include <spectacularAI/realsense/session.hpp>

namespace spectacularAI

namespace rsPlugin

struct Session

VIO session.

Public Functions

virtual bool hasOutput() const = 0

Check if new output is available

virtual std::shared_ptr<const VioOutput> getOutput() = 0

Get output from the queue, if available. If not, returns {}

virtual std::shared_ptr<const VioOutput> waitForOutput() = 0

Wait until new output is available and then return it. Do not use with useReaderThread=false.

virtual void addTrigger(double t, int tag) = 0

Add an external trigger input. Causes additional output corresponding to a certain timestamp to be generated.

Parameters:

• t – timestamp, monotonic float seconds

• tag – additonal tag to indentify this particular trigger event. The default outputs corresponding to input camera frames have a tag 0.

virtual ~Session()