Patioplanner Gesture Detection: Sensor to Image Diffing

When you’re kneeling in dirt, hands covered in sand and cement dust, the last thing you want is to grab your phone to scroll through your paving plan. That was the problem I wanted to solve: a hands-free way to interact with Patioplanner while actually laying pavers.

The idea sounds simple. Wave your hand, the app reacts. In practice, getting there was anything but simple.

Attempt 1: the proximity sensor

The first idea felt obvious. Modern phones have a proximity sensor right there at the top. It detects when you hold the phone to your ear during a call. Why not use it to detect a hand wave?

The Web API technically exists. The ProximitySensor interface is part of the Generic Sensor API specification. On paper, you create a sensor instance, listen for readings, done.

In reality, the API is either not implemented or not available on the platforms that matter. Safari doesn’t support it. Firefox doesn’t support it. Chrome has it behind a flag, and even then most devices don’t expose the sensor to the browser. The proximity sensor is locked down to native phone-call functionality on most hardware.

I spent an afternoon writing the integration, testing on multiple devices, and getting exactly zero readings. The sensor either throws a NotAllowedError or simply never fires an event. Dead end.

Attempt 2: camera brightness detection

Next idea: use the camera. If someone waves their hand over the phone, the camera image gets darker. When the hand moves away, it gets brighter again. Detect that dark-to-bright transition, and you have a gesture.

The implementation was straightforward. Grab a video stream via getUserMedia, draw frames to a hidden canvas, calculate the average pixel brightness per frame. When the average drops below a threshold and then rises above it within a short time window, that’s a gesture.

function getAverageBrightness(imageData: ImageData): number {
  const data = imageData.data;
  let sum = 0;
  for (let i = 0; i < data.length; i += 4) {
    sum += data[i] * 0.299 + data[i + 1] * 0.587 + data[i + 2] * 0.114;
  }
  return sum / (data.length / 4);
}

It worked. Sometimes. The problem was reliability. Outdoors, clouds passing overhead trigger false positives. Indoors, someone walking past a lamp triggers it. Even the user’s own shadow while leaning over the phone would register as a gesture. And the threshold that worked in my living room was completely wrong in the garden at noon.

I tried adaptive thresholds, rolling averages, cooldown timers. Each fix helped one scenario and broke another. The underlying issue is that average brightness is a single number. It throws away all spatial information. There’s no way to distinguish “hand moved over screen” from “cloud moved over sun” because both look the same in the data.

Too flaky. Next.

Attempt 3: image difference detection

The thing that worked was to stop reducing the frame to a single value and compare whole frames against each other instead.

Take a reference frame, then keep comparing new frames against it. If enough pixels have changed by enough, something moved in front of the camera. Ambient lighting changes (clouds, slowly shifting shadows) affect the whole image uniformly and gradually. A hand waving over the camera changes a large part of the frame suddenly.

function calculateImageDifference(
  current: ImageData,
  reference: ImageData,
  threshold: number
): number {
  const len = current.data.length;
  let changedPixels = 0;
  for (let i = 0; i < len; i += 4) {
    const diff =
      Math.abs(current.data[i] - reference.data[i]) +
      Math.abs(current.data[i + 1] - reference.data[i + 1]) +
      Math.abs(current.data[i + 2] - reference.data[i + 2]);
    if (diff > threshold) {
      changedPixels++;
    }
  }
  return changedPixels / (len / 4);
}

The reference frame updates slowly over time, so gradual lighting changes get absorbed. But a hand appearing in frame causes a spike in the difference ratio that’s easy to detect.

This held up well indoors and outdoors. Because it looks at relative change rather than absolute brightness, it copes with varying light, and it stayed reliable even on a construction site with dust in the air and shifting shadows.

The final implementation adds a few things. The canvas is small, around 64x48 pixels, so there’s little data to process and the comparison is cheap enough not to drain the battery. The reference frame blends slowly toward the current frame, so gradual changes in the environment don’t pile up as false positives. And a gesture only registers when the difference ratio crosses a high threshold on the way up and then drops below a lower one, which stops a single sustained event from firing several gestures.

What I took away

The proximity sensor would have been ideal if browsers actually supported it. Brightness detection was simple but limited by being a one-dimensional signal. Image diffing looks heavier on paper but turned out to be both more reliable and more flexible.