Azure Kinect

Spectacular AI C++ plugin for Microsoft Azure Kinect DK.

Note: The Azure Kinect has been discontinued, but nearly equivalent device, ORBBEC Femto is now available and uses the same depth sensing technology, which ORBBEC has licensed from Microsoft.

ORBBEC cameras are supported by their Spectacular AI SDK wrapper. See here.

Quick start

First, download and unpack spectacularAI_k4aPlugin_cpp_non-commercial package (only available for SDK version 1.28)

If you have not used the Azure Kinect device before, you will need to setup udev rules, run the following script (or follow the instructions in Azure Kinect SDK):
```
./bin/3rdparty/k4a/setup_udev_rules.sh
```
Attach your Azure Kinect device to a USB3 port

Run the JSONL example (smoke test):

cd bin
./vio_jsonl
# Now you should see rapidly flowing JSONL text
# press Ctrl+C to exit

Run the Python example (in the bin directory):

pip install matplotlib # install dependencies
python vio_visu.py

Attach your Azure Kinect device to a USB3 port

Run the JSONL example (smoke test):

cd bin
./vio_jsonl.exe
# Now you should see rapidly flowing JSONL text
# press Ctrl+C to exit

Run the Python example (in the bin directory):

pip install matplotlib # install dependencies
python vio_visu.py

Recording data

Azure Kinect data can be recorded using the sai-record-k4a tool included in the SDK package (in the bin directory). First connect Azure Kinect over USB3, then run (add .exe on Windows):

./sai-record-k4a # Record with visualization, requires pip install spectacularAI[full]
./sai-record-k4a --recording_only --no_preview # More lightweight
./sai-record-k4a --output /path/to/output/folder # Set recording directory
./sai-record-k4a -h # See all options

The source code of recording tool is also available at https://github.com/SpectacularAI/sdk/tree/main/cpp/k4a which allows customizing the recording tool, if necessary.

Installation as a library

First verify that you have installed Azure Kinect SDK, e.g. check that open k4aviewer.

The following commands should be run in the folder where you have unpacked the SDK release package spectacularAI_k4aPlugin_cpp_non-commercial.

First install the development dependencies.

Select where you want the library installed, e.g.,:

export MY_INSTALL_PREFIX=~/.local
# or: export MY_INSTALL_PREFIX=`pwd`/installed

Run make PREFIX=$MY_INSTALL_PREFIX install (or sudo make install)
Build the vio_jsonl example using CMake:
```
make PREFIX=$MY_INSTALL_PREFIX example
```
Run the vio_jsonl binary as a smoke test.

Install the development dependencies.
Build vio_jsonl example using CMake. The process is a bit involved so the commands are collected in ./examples/build_windows.sh. In PowerShell:
```
cd examples
sh ./build_windows.sh ..\lib\cmake\spectacularAI
```
Finally running vio_jsonl.exe should give you the pose of the Azure Kinect device in real time (smoke test):
```
cd target/Release
./vio_jsonl.exe
```

Examples

The following instructions show how to compile and run the Spectacular AI K4A SDK examples.

(Optional) Install the SDK as a library (see above).
Clone the SDK examples repository:
git clone --recursive https://github.com/SpectacularAI/sdk-examples.git

Run the following commands in the sdk-examples/cpp/k4a directory.

Build the targets using CMake:

mkdir target
cd target
cmake -DspectacularAI_k4aPlugin_DIR=<path/to/spectacularAI_k4aPlugin/install/lib/cmake/spectacularAI/> ..
make

The -DspectacularAI_k4aPlugin_DIR option is not needed is you have used sudo make install for the SDK.

Build the vio_jsonl example using CMake. The process is a bit involved so the commands are collected in ./build_windows.sh. In PowerShell:

sh ./build_windows.sh <path\to\spectacularAI_k4aPlugin\lib\cmake\spectacularAI>

API Reference

The classes specific to Azure Kinect are documented below. For the main output classes, refer to the C++ documentation in Core SDK.

Defined in #include <spectacularAI/k4a/configuration.hpp>

namespace spectacularAI

namespace k4aPlugin

Functions

k4a_device_configuration_t getK4AConfiguration(const std::string &colorResolution = "720p", int depthMode = K4A_DEPTH_MODE_WFOV_2X2BINNED, int frameRate = 30, bool useStereo = true)

std::string getYAMLConfiguration(const Configuration &config)

struct Configuration

Plugin and Spectacular AI VIO SDK configuration variables.

Public Members

bool useStereo = true: Set true to use RGB-D mode, false for mono (RGB camera only).

bool useSlam = true: Enable SLAM, required for ICP

bool fastVio = false: Enable more lightweight VIO parameters

bool alignedDepth = false

Align depth images to color images using Azure Kinect SDK.

Note that this affects ICP performance since the depth image has larger field of view than the color camera.

std::string recordingFolder = ""

bool recordingOnly = false

bool postProcessFinalMap = false: If true, final mapping API output is refined using post processing.

std::string aprilTagPath = "": Path to .json file with AprilTag information. AprilTag detection is enabled when not empty. For the file format see: https://github.com/SpectacularAI/docs/blob/main/pdf/april_tag_instructions.pdf Note: sets useSlam=true

k4a_device_configuration_t k4aConfig = getK4AConfiguration(): Configuration for the k4a device.

bool ignoreOutputs = false: If false, user should read vio outputs from the Session. If true, the SDK discards vio outputs. Useful when VIO outputs are not needed: e.g. when using mapping API or recording.

std::map<std::string, std::string> internalParameters: Internal SDK parameter overrides (key-value pairs). Not safe for unsanitized user input.

Defined in #include <spectacularAI/k4a/pipeline.hpp>

namespace spectacularAI

namespace k4aPlugin

class Pipeline

The Pipeline object stores information about the device and also sets the default configuration, e.g., enables and configures necessary Azure Kinect streams and sensors. The actual VIO reading is started with startSession.

Public Functions

Pipeline(): Create a pipeline with the default configuration.

Pipeline(const Configuration &config): Create a pipeline with custom configuration

Pipeline(const Configuration &config, std::function<void(mapping::MapperOutputPtr)> onMapperOutput): Create a pipeline with custom configuration with Mapping API enabled

std::unique_ptr<Session> startSession(): Start a new VIO session on the background. Internally starts the k4a device and creates threads for camera and imu.

k4a_device_t getDeviceHandle(): Get an handle to the Azure Kinect device, can be used to adjust device settings. Note that adjusting some settings, e.g. device FPS can affect VIO performance.

~Pipeline()

Private Members

std::unique_ptr<impl> pimpl

Defined in #include <spectacularAI/k4a/session.hpp>

namespace spectacularAI

namespace k4aPlugin

struct Session

Public Functions

virtual bool hasOutput() const = 0: Check if new output is available.

virtual std::shared_ptr<const VioOutput> getOutput() = 0: Get output from the queue, if available. If not returns {}.

virtual std::shared_ptr<const VioOutput> waitForOutput() = 0: Wait until new output is available and then return it.

virtual void ignoreOutputs(bool ignore) = 0

Set whether new outputs are added to the queue. Useful when the user is not interested in VIO outputs: e.g. when using mapping API or recording.

Parameters:: ignore – If true, new outputs are not added to the queue. Otherwise, the user has make sure the outputs are pulled from the queue.

virtual void addTrigger(double t, int tag) = 0

Add an external trigger input. Causes additional output corresponding to a certain timestamp to be generated.

Parameters:

t – timestamp, monotonic float seconds
tag – additonal tag to indentify this particular trigger event. The default outputs corresponding to input camera frames have a tag 0.

virtual ~Session()