Tracking Metrics for Monado

When working with tracking systems it can happen that you wake up one day, and the tracking feels better than the day before without any particular reason to attribute the improvement to. Questions like “maybe the USB connection was a bit loose yesterday?” or even “could it be because today is more sunny?” when using camera-based tracking are some I came to formulate in these scenarios. It could even be the case that you just woke up in a better mood and tolerate tracking issues a bit more.

After realizing this, I figured it was essential to have a good way of objectively determining the performance of a system on a given sequence of datasets. More so considering the next steps with the tracking in Monado include going deeper into the SLAM/VIO systems’ pipelines.

A couple of tools for measuring the accuracy and performance of SLAM systems exist, SLAMBench¹ being a very good example, but even then we needed ways of automatizing the process of generating data for evaluation of datasets in Monado. This post tries to summarize what is now possible regarding SLAM/VIO evaluation in Monado.

Generating the data

The first merge request in this line of work was this one. It does a couple of things like allowing the SLAM systems to report their internal pipeline times back to Monado, and integrates into Monado the possibility to use reference trajectories (ground truth) if available from the SLAM sources (e.g., EuRoC dataset player). It enables the possibility to display in real-time both the tracking error² and the timing information of the systems as seen in the curves to the left of the image below.

However, the most important thing that that MR introduces is the ability to save CSV files with the timing information and the estimated, predicted and filtered trajectories from the tracker in CSV files for offline analysis like this:

For the predicted poses saved in tracking.csv you have:

# timestamp [ns], p_RS_R_x [m], p_RS_R_y [m], p_RS_R_z [m], q_RS_w [], q_RS_x [], q_RS_y [], q_RS_z []
6997681930473,0.0000000000,0.0000000000,0.0000000000,0.5557456613,0.0009835553,-0.8313517570,0.0000000000
6997731930601,-0.0000058096,-0.0000130947,0.0000100406,0.5557390451,0.0009709296,-0.8313562274,0.0000112125
6997781930473,-0.0000083494,-0.0000609047,0.0000496814,0.5557395816,0.0009258533,-0.8313559294,0.0000084368
6997831930601,0.0000331269,-0.0001313922,0.0001094043,0.5557386875,0.0009122558,-0.8313565254,0.0000182058
6997881930473,-0.0000239215,0.0001417128,-0.0000660935,0.5551946759,0.0003172281,-0.8317196965,0.0010597851
6997931930601,0.0000156540,0.0003277477,0.0001416763,0.5555265546,0.0004448513,-0.8314981461,0.0009185750
6998031930601,0.0000845920,0.0003950795,-0.0000916694,0.5554929376,-0.0000688768,-0.8315203190,0.0012723863
...

While for the timing information saved in timing.csv you get something like:

# frame_ts,tracker_received,opticalflow_received,vio_received,tracker_consumer_received,received_by_monado
6997681930473,6997694542317,6997695262125,6997699258602,6997699631993,6997700087150
6997731930601,6997743955455,6997745236261,6997749067885,6997749198851,6997749938120
6997781930473,6997791383303,6997792716579,6997796591373,6997796698030,6997799916475
6997831930601,6997838933853,6997839698854,6997842708168,6997842804635,6997850212650
6997881930473,6997888078063,6997888855419,6997891708941,6997894958478,6997899706131
6997931930601,6997936733173,6997937138050,6997940231291,6997942182912,6997949933197
6997981930473,6997986854881,6997987260573,6997989946485,6997992114463,6998000756394
6998031930601,6998036722866,6998037490713,6998040232329,6998042449818,6998050688152
6998081930473,6998086709312,6998087110536,6998089822441,6998091790862,6998101053875

You also get prediction.csv and filtering.csv which correspond, respectively, to the predicted and filtered trajectories the tracker returns to the XR app.

Generating it in bulk

The next merge request introduces the monado-cli slambatch command. This command allows Monado to run multiple datasets at once to generate all of these CSV files for each and is designed to run without UI so that it’s CI friendly. You can run a command like:

monado-cli slambatch \
  $euroc/MH_04_difficult $bsltdeps/basalt/data/monado/euroc.toml MH04 \
  $euroc/V1_02_medium $bsltdeps/basalt/data/monado/euroc.toml V102 \
  $tumvi/dataset-room3_512_16 $bsltdeps/basalt/data/monado/tumvi.toml R3

And then you obtain the respective CSV files:

.
├── MH04
│   ├── filtering.csv
│   ├── prediction.csv
│   ├── timing.csv
│   └── tracking.csv
├── R3
│   ├── filtering.csv
│   ├── prediction.csv
│   ├── timing.csv
│   └── tracking.csv
└── V102
    ├── filtering.csv
    ├── prediction.csv
    ├── timing.csv
    └── tracking.csv

Analyzing the data

Finally, a set of python scripts were written in the xrtslam-metrics repo that are designed to analyze all of this data programmatically.

For analyzing the processing times, you can specify the start and end columns (i.e., pipeline stages) in timing.csv that you are interested in analyzing like this:

$ ./timing.py runs/BNF/EV201/timing.csv tracker_pushed vio_produced -p
TimingStats(mean=10.13590801754386, std=1.895429240022987, min=4.155935, q1=9.018804, q2=10.062223, q3=11.3165765, max=19.492825) from 'tracker_pushed' to 'vio_produced'

And a plot like this is produced:

Some systems are not stable enough to complete all dataset before crashing and so completion.py can tell you what’s the percentage it gets to complete.

$ ./completion.py runs/K/TR1/tracking.csv targets/TR1/cam0.csv
|:---------------------|:--------|
| Groundtruth poses    | 2821    |
| Estimated poses      | 268     |
| Tracking duration    | 56.75s  |
| Groundtruth duration | 141.00s |
| Tracking completion  | 40.25%  |

Then you have the accuracy statistics, both in terms of absolute (ATE) and relative (RTE) trajectory errors, which are standard in the field and are the ones you will encounter in the systems papers. For this xrtslam-metrics uses the awesome EVO library. It’s worth noticing that each paper implements this statistics just a bit different, my decisions here tries to align with what makes more sense for XR.

$ ./tracking.py ate targets/EV201/gt.csv runs/BNF/EV201/tracking.csv
APE w.r.t. translation part (m)
(with SE(3) Umeyama alignment)

       max	0.073318
      mean	0.035746
    median	0.034026
       min	0.006925
      rmse	0.038935
       sse	3.395612
       std	0.015432

$ ./tracking.py rte targets/EV201/gt.csv runs/BNF/EV201/tracking.csv -p
RPE w.r.t. translation part (m)
for delta = 6 (frames) using consecutive pairs
(with SE(3) Umeyama alignment)

       max	0.015671
      mean	0.002855
    median	0.002477
       min	0.000116
      rmse	0.003472
       sse	0.004496
       std	0.001976

These also allow you to produce figures with the -p flag to compare the trajectory to the ground truth in 3D.

Furthermore, you can use the entirety of tools offered by EVO on the tracking.csv files from the CLI to compare specific trajectories. You can see what’s possible in the EVO home page.

And last but not least, batch.py allows you to generate markdown tables out of all runs! Cells are mean ± stdev of the entire run. See below the tables an explanation of what each row and column means.

Processing times [ms]

Completion [%]

Absolute precision [m]

Relative precision [m] in spans of 6 frames

Tables explanation

Datasets:

• C*: Custom datasets, each with two difficulties.
  • C6: RealSense D455 640x480@30fps
  • C8: D455 848x480@60fps
  • CO: Odyssey+ (640x480@30fps)
• E*: Euroc datasets
• T*: TUM-VI datasets

Systems:

• BND: Basalt (new) after update for ICCV21, using double precision
• BNF: Same as above but with single precision
• BO: Basalt (old) before the update
• K: Kimera
• ON: ORB-SLAM3 (new) after recent v1.0 update
• OO: ORB-SLAM3 (old) before update

Final note

As a final note, I want to link to my next blogpost that discusses why ORB-SLAM3 metrics reported in the tables above are so bad and an important issue I found thanks to all the tools developed here.

Footnotes