Back to projects

Project case study

  • Own project
  • In-browser computer vision
  • User's camera
  • Public repository

Real-Time Hand Gesture Recognition

Recognition of 11 hand gestures from the camera, in the browser, with explainable rules and a CPU fallback.

Web demo that recognises 11 hand gestures from the camera, in the browser, with MediaPipe, explainable rules and a GPU to CPU fallback. No video leaves the machine.

  • Explainable rules
  • Per-video-frame pipeline
  • CPU fallback
  • Screen-reader accessible
My roleWeb version development: frame pipeline, gesture rules, error states and tests
Technologies used
  • MediaPipe Tasks Vision
  • Canvas API
  • Next.js
  • React
  • TypeScript
  • Vitest
  • Playwright
Open demoView repository

Public repository; audited reference on the main branch, commit abacb02.

Home page listing the recognised gestures and the finger pattern of each one.
RECONHECIMENTO-DE-MAO.VERCEL.APP / HOME PAGE

Summary

Summary

The demo reads up to two hands from the camera with the MediaPipe Hand Landmarker and classifies 11 gestures with rules over the hand's 21 landmarks. A 5-frame majority vote steadies the reading, and the interface announces the gesture to screen readers.

Problem

Gesture interfaces often need a local install or send video to a server, and opaque classifiers do not explain why a gesture was recognised.

My role

The web version was written on top of the Python version's logic, with fixtures shared between the two.

Recognised gestures
11
Public repository / web/src/lib/gestures/analyze.ts / commit abacb02
Web unit tests
25
Public repository / declaration count in web/, CI on GitHub Actions / commit abacb02
End-to-end tests with axe
12
Public repository / declaration count in web/e2e / commit abacb02

Data and assumptions

Frames come from the user's camera and are processed in the browser. Tests use about 15 synthetic poses shared between TypeScript and Python; recognition with a real hand was checked manually.

Simultaneous hands
2
Smoothing window
5 frames
Interface update
about 10 Hz
Model
7.8 MB, from a CDN
Data structure

Modeled entities

  • Hand reading (landmarks, handedness)
  • Stabilised gesture per hand

Frame pipeline

From the camera to the announced gesture, in a single Next.js page.

  1. Capture

    getUserMedia and requestVideoFrameCallback, with requestAnimationFrame as a fallback, dropping repeated frames.

  2. Inference

    Hand Landmarker detectForVideo on the main thread, on the GPU with a CPU fallback.

  3. State

    Per-hand tracker with a 5-frame majority vote.

  4. Render

    Landmark overlay on Canvas 2D for every inferred frame; React interface at about 10 Hz.

Pipeline signals

What the demo measures and shows during the session.

DefinitionUnitCalculation ruleUseLimitation
Inference timeDuration of each model call.msperformance.now() difference around detectForVideo.Show the model cost in the session panel.It depends on the hardware and the delegate in use.
Stabilised gestureMost frequent gesture in the hand's last 5 frames.labelPer-hand majority vote.Avoid frame-by-frame gesture flicker.It delays the response by a few frames.

Inference time

Duration of each model call.

Unit
ms
Calculation rule
performance.now() difference around detectForVideo.
Use
Show the model cost in the session panel.
Limitation
It depends on the hardware and the delegate in use.

Stabilised gesture

Most frequent gesture in the hand's last 5 frames.

Unit
label
Calculation rule
Per-hand majority vote.
Use
Avoid frame-by-frame gesture flicker.
Limitation
It delays the response by a few frames.

Technical decisions

Rules over landmarks instead of a trained classifier

Reason
Each gesture has a readable explanation, testable with synthetic poses.
Consequence / trade-off
The rules assume the palm faces the camera.

requestVideoFrameCallback

Reason
It only processes new video frames.
Consequence / trade-off
It needs a requestAnimationFrame fallback in browsers without support.

GPU with a CPU fallback

Reason
The demo keeps running when the GPU delegate fails.
Consequence / trade-off
On the CPU, inference is slower.
Additional technical decisions

Inference on the main thread

Reason
It keeps the pipeline simple.
Consequence / trade-off
Inference is not decoupled from the frame rate; a Worker or a Hz cap would decouple them.

Evidence

Application and code verified without sign-in on the main branch, commit abacb02.

Public application

Published demo

Home page, demo, session panel and about page.

Open evidence

Code reference

Gesture rules

Finger pattern, pinch and OK for the 11 gestures.

Open evidence

Code referenceweb/src/lib/gestures/analyze.ts

Code reference

Majority smoothing

5-frame vote per hand.

Open evidence

Code referenceweb/src/lib/gestures/smoothing.ts

Code reference

Camera pipeline

Per-frame scheduling, inference, CPU fallback and accessible announcement.

Open evidence

Code referenceweb/src/components/demo/CameraDemo.tsx

Implementation references

Code reference

Error states

Insecure context, denied permission, busy camera and model failure.

Code referenceweb/src/lib/camera.ts

Open evidence

Public repository

Repository at the audited commit

Public snapshot used for this audit.

Open evidence

Verifiable measures

measured
26
Python version tests
Declaration count, run in CI.
measured
15
Shared synthetic poses
fixtures/gestures/hand-poses.json, used by both versions.
configuration
1.5 s
Minimum announcement interval
Screen-reader announcement limit.

Counts from the repository at the audited commit. There is no accuracy measure with a real hand.

Public repository, commit abacb02.

What the project demonstrates

  • In-browser computer vision, with no server.
  • Frame pipeline that drops repeated frames.
  • Graceful GPU to CPU degradation.
  • Touchless interaction accessible to screen readers.

Limitations and next steps

Limitations

  • Requires a camera and HTTPS; the model comes from a CDN.
  • There is no accuracy measure with a real hand.
  • Inference runs on the main thread, coupled to the frame rate.

Next steps

  • Move inference to a Worker or cap it in Hz.
  • Measure accuracy with a labelled set of real hands.

Video and media

Project demonstration video.

Captures of the published demo.

Home page listing the recognised gestures and the finger pattern of each one.
Home page listing the recognised gestures and the finger pattern of each one.
Demo screen with the camera off, before permission.
Demo screen with the camera off, before permission.
Session panel with no session recorded yet.
Session panel with no session recorded yet.

Optional technical details

How to run
  1. cd web && npm ci && npm run dev
  2. Desktop version (Python 3.12): pip install -r requirements.txt and python -m hand_gestures

Next project

Public Revenue Intelligence Platform

A demonstration web analytics product for exploring synthetic revenue, targets, time-based changes, and revenue composition through an executive interface.

View full case