Gestural interactions for smart glasses

Guided the product and machine learning team on the future of wearables at Ultraleap for an R&D project.

Background

Smart glasses are coming, and hands are an effortless and natural input for interacting with the digital content on-the-go. However, traditional camera technology used for hand tracking consumes a lot of battery - could low-power event sensors be used for interactions instead?

At the beginning of 2024, Ultraleap started a research project to bring an event sensor hand tracking solution to the market. I was the lead designer for the project, exploring the future of interactions.

Role

Leading design
Interaction design
Interaction research
User research
Market research
Project management

Tools

Figma
Unity
Miro

Team

Senior Designer
UX Engineer / prototyper
Product manager
AR research lead

Stakeholders

VP of XR
SVP of Product
ML research team
Product manager

Key challenges

Technical limitations of the near & long-term future

Balancing what’s possible in the near term versus the long term within technical limitations.

Research meets design

Combining design and research in parallel revealed technical limitations as the ML work progressed. Managing stakeholder expectations required careful communication.

Use cases of the devices not yet in the market

Smart glasses are still a nascent device category, with very few users globally. What will the use cases in a few years of time look like, and how will hand tracking fit in?

Artefacts

Smart glasses OS demo

OS demo showing the progress of the research in an applicable use case. Used for both internal research and external showcases. (The design was a collaboration with the other UX designer of the team)

ML quality testing & data capture application

Designed a quality testing application used on a daily basis by the ML research team. The application was also used to capture training data.

Market research

Market research of smart and AR glasses - covering features, technical specifications, interactions, UX, and use cases.

Gathering feedback and reactions at AWE US

Bringing the research out to the public and analyzing the reactions to the prototypes at the AWE US conference in LA.

Learn more

More information available upon request.