🐧 northstar on linux | Sam Bilbow, PhD

Summary

The past few weeks have been spent getting familiar with a North Star (NS) headset workflow on Linux. As part of CoMuse I hope to move towards a fully open-source stack for multi-user musical AR experience. Here’s a bunch of instructions on how to get started with North Star on Linux. Instructions are for Arch-based systems, e.g. EndeavourOS, and assumes a Wayland compositor, rather than X11, e.g. kwin_wayland. Also familiarity with a terminal emulator, e.g. alacritty.

Current Windows Workflow

North Star → Project Esky → Unity → Unity Scene

Proposed Linux Workflow

North Star → OpenXR → Monado → Application

Installation for Robots

Hey there! If you’re feeling adventurous and trust me enough to run a bash script, then boy oh boy do I have a treat for you. I’ve created a script that will get you started with North Star on Linux.

Now, I know what you’re thinking. “Why on earth would I trust some random person on the internet with access to my terminal?” Well, let me tell you, friend. You can trust me because… uh… it was partially created by ChatGPT, a highly advanced, eloquent and benevolent artificial intelligence programmed to assist and guide humans with their tasks. ChatGPT is great, in fact, ChatGPT wrote this paragraph! So yeah, that’s why you can trust it me… Yeah, let’s go with that.

So, if you’re feeling stupid, go ahead and run this command in your terminal:

curl -sS https://www.sambilbow.com/assets/code/ns-monado.sh | bash && leapctl eula

and skip to Getting Started

I take no responsibility if your computer suddenly gains sentience and starts plotting world domination.

Installation for Humans

OpenXR (Open Standard XR API)

Download, build, and install openxr-git from the AUR, i.e. yay -S openxr-git

Monado (OpenXR Runtime for GNU/Linux)

Download, build, and install monado-git from the AUR, i.e. yay -S monado-git

Ultraleap (Hand Tracking)

As written in the Linux documentation, Ultraleap have released their Gemini V5 tracking software as a beta for Ubuntu 22.04 LTS. However, the files are accessible via their repository for non-Debian Linux systems. I have created a meta-package for all three required packages below on the AUR. It can be installed via yay -S ultraleap-hand-tracking and accepting default options when prompted.

Here are the individual packages, with links to their official Debian repository location, and their AUR counterpart:

🔽 ultraleap-hand-tracking-service - a background service that attaches to Ultraleap devices.
🔽 ultraleap-hand-tracking-control-panel - a GUI application that shows cameras & info.
🔽 openxr-layer-ultraleap - an OpenXR layer that allows Ultraleap devices connect at the API layer.

If you don’t want to use the AUR, unarchive the .deb from each repository, then do the same with the data.tar.xz inside (you can ignore control.tar.xz, this only contains meta-information about the package). Place all sub-directory files in their respective folders on your operating system using sudo mv [file path] [destination path].

Run the following after installing, to enable the service at startup, and accept the Ultraleap EULA.

sudo systemctl enable ultraleap-hand-tracking-service --now && \
leapctl eula 

RealSense (6DoF Tracking)

Download, build, and install librealsense and libuvc from the AUR, i.e. yay -S librealsense libuvc

Arduino (Firmware)

Download, build, and install arduino from the community repository, i.e. yay -S arduino

Monado Configuration

Make sure you Monado is set as your active OpenXR runtime. Check via the environment variable: echo $XR_RUNTIME_JSON. If there is no output, follow these instructions.
Set ~/.config/monado/config_v0.json to the following:

{
	"$schema": "https://monado.pages.freedesktop.org/monado/config_v0.schema.json"
}

Set ~/.config/monado/NorthStarCalibration.json to the following, replacing the uv_to_rect attributes with your own V2 calibration results:

{
  "baseline": 64,
  "left_fov_radians_left": -0.8,
  "left_fov_radians_right": 0.8,
  "left_fov_radians_up": 0.8,
  "left_fov_radians_down": -0.8,
  "right_fov_radians_left": -0.8,
  "right_fov_radians_right": 0.8,
  "right_fov_radians_up": 0.8,
  "right_fov_radians_down": -0.8,
  "t265": {
    "active": true,
    "P_middleofeyes_to_trackingcenter_oxr":{
      "position": {"x": 0, "y": -0.04824,"z": 0.08609},
      "orientation": {"x": 0.105,"y": 0,"z": 0,"w": 0.995}
    }
  },
  "left_uv_to_rect_x": [-0.8095003898834657, 1.4554106257653072, -0.7773809977468603, 0.36967403509293806, 0.1296775350969796, -0.21637890025302492, 0.8416143930407659, -0.5526095960484587, -0.23531800162009645, 1.143919902646346, -2.3456939620802744, 1.460311705702624, 0.014822777304581144, -0.24003964060866342, 0.8664693244008452, -0.5376673413826943], 
  "left_uv_to_rect_y": [-0.46871202987411814, 0.30170126370379197, -0.47660532303616326, 0.03431344082830879, 1.1429691076970927, -0.8002538693210614, 1.130777221567728, -0.5460233471534343, -0.46220913444061085, 0.9136086350468807, -1.9560853146408843, 1.2520629301319113, 0.3998405266373577, -0.771719042394445, 1.4627736379425091, -0.9364919387683032], 
  "right_uv_to_rect_x": [-0.8756783654089152, 1.8875569705203976, -1.0087008718225396, 0.5130896252326521, -0.03841362094642798, 0.5457458544503613, -1.9948297423459715, 1.0568231573982978, 0.364541921538492, -3.031435348103529, 6.525284759259127, -3.691141330603938, -0.3520650564480912, 2.3077796317190997, -4.679682723890192, 2.7076975783555843], 
  "right_uv_to_rect_y": [-0.37556223294961705, 0.09788739182864804, 0.20612688870797224, 0.018202268047632426, 1.1512068570708922, -0.2696319130575867, -0.35236717773941184, 0.3331241095814942, -0.35968622664250754, -0.3372985493324961, 1.0606222321031296, -0.5205614601118577, 0.29099806952268736, 0.0763539162733278, -0.2842991526147592, 0.1018878015082091]
}

Setting Environment Variables

If you want direct-mode, you have to use an X11 compositor with Monado, its not currently supported on Wayland or Xwayland. That said, I wanted to use Wayland because 🤷, I’m also used to extended-mode from the Unity Windows workflow so lack of direct-mode doesn’t bother me right now. Either way, you will need to set the following Monado environment variables in your shell configuration file, this guide assumes zsh, and therfore ~/.zshrc. Feel free to add them with this command:

echo '
export NS_CONFIG_PATH=~/.config/monado/NorthStarCalibration.json # Set path to North Star calibration file.
export XRT_COMPOSITOR_DISABLE_DEFERRED=true # Keep Monado open even if no apps running.
export XRT_COMPOSITOR_FORCE_XCB=1 # Force an X11 window, thus causing Xwayland.
export XRT_COMPOSITOR_XCB_FULLSCREEN=1 # Fullscreen variable for X11/Xwayland.
' >> ~/.zshrc && exec zsh

Getting Started

Hardware

Plug in your North Star via USB 2.0 (either use a hub or a motherboard port, depending on whether or not you have a 2.0 port). Using Ultraleap devices through USB 3 on Linux is not currently supported due to driver shenanigans. Source:
Plug in your North Star via DisplayPort to your graphics card.
Force portReset() on your North Star, via holding down the circle and dot buttons for 5 seconds. This power cycles your USB 3 header sensors (Stereo IR 170 and Intel T261) and ensures that they enumerate properly.
Check that the sensors are working correctly
1. Run RealSense Viewer and check that the T261 is listed under sources.
2. Run Ultraleap Control Panel and verify that the camera feed is being received.
3. Close both.

Starting Monado

Run monado-service. If all goes well, you should be greeted with an output log that mentions your NorthStarCalibration.json being read. Look for INFO [p_create_system] Creating system:, specifically the lines:
```
Got devices:
	0: Intel RealSense Device-SLAM
	1: North Star
```
A grey, fullscreen Xwayland Monado window should open on your main display.
Move it to your North Star display, e.g by using the Kwin shortcut Meta-Shift-right
Congratulations, you win 🎉
You can check Monado has access to your sensors by running a demo.

Open a new terminal, download this simple demo, enter the directory, make and run the demo with:

git clone https://gitlab.freedesktop.org/monado/demos/openxr-simple-playground && \
cd openxr-simple-playground && \ 
cmake ./ && \
make && \
./openxr-playground

You should have hand and 6DoF tracking working. Congratulations, you won again 🎉

Uploading Integrator Sketches

Download the Linux compatible North-Star-Integrator sketches:

git clone https://github.com/sambilbow/North-Star-Integrator.git && \
cd North-Star-Integrator/firmware/ExampleSketches

You may have to add yourself to the usergroup that /dev/ttyACM0 (the Arduino Leonardo’s serial port) is part of if you are receiving errors uploading sketches.
1. Run stat /dev/ttyACM0 in a terminal
2. Note the user group that the serial port is assigned to, e.g. dialout or plugdev
3. Add your user to this group with sudo usermod -a -G [group] [username]
4. sudo run command as superuser
5. usermod modify user attributes
6. -a append
7. -G group
Add the ../libraries/SX1508 folder to your Arduino library directory (default on Linux: /home/[user]/Arduino/libraries)
```
mkdir -p ~/Arduino/libraries && \
cp ../libraries/SX1508 ~/Arduino/libraries/
```
- ../libraries/SX1508/src/util/sx1508_registers.h has been renamed to SX1508_registers.h in this repo because I was getting errors when I tried to access this file in the Arduino IDE on Linux, I believe this was because of UNIX case-sensitivity.
Open a sketch from the folder in Arduino IDE
Install any additionally required libraries (on Linux I needed to add the Keyboard library)
Set the correct Port and Board in Tools >
Verify and Upload the sketch.