Tooling for ML research quality management

Built tooling and evaluation pipelines for hand tracking UX at Ultraleap - translating user insights into actionable metrics for ML teams.

Background

In 2019, Ultraleap acquired Leap Motion’s hand tracking technology. The technology was based on infrared lights - however, the OEM market was hesitant to add dedicated IR lights to their XR headset.

As a response, Ultraleap started developing a visible light hand tracking solution in 2024. With Apple, Google and Meta in the market, the quality bar was high - how do you measure the progress in hand tracking models, both internally and against the competitors?

Role

Lead designer
Product manager

Tools

Unity
Figma
ShapesXR
PowerBI

Team

1 - 3 Unity engineers

Stakeholders

ML research team leads
C-level
ML researchers
QA team
Engineering director

Key challenges

Quantifying model quality

How can we determine that e.g. a pinch gesture is consistently better across users and environments between model iterations?

What should we measure?

What kind of user interactions should the machine learning research target? Which qualities of our hands matter?

Analysing the trade-offs

Some experiments may create accuracy in one area - but lower the quality elsewhere. How can you observe the holistic quality?

Artefacts

Quality assessment applications

Designed and managed quality assessment applications for hand tracking performance, used by the ML research & QA teams.

Quality dashboard for hand tracking models

Based on the data created by quality assessment applications, I designed and built a dashboard in Power BI to track model improvements.

Technology demos for testing interactions in context

Designed multiple technology demos, which were used to test the new model iterations in context of the common use cases.

Storyboards of the future

To supplement the tooling and demos, I also created user stories for the future use hand tracking technology. This aligned the team on the vision, environments and user behaviour of the future.

Learn more

More information available upon request.