Hand Tracking

This module tracks the position of up to two hands on the camera image. So you can move your hand up-down or sideways to generate MIDI events.

To be more exact, it currently tracks the Metacarpophalangeal (MCP) joint of the middle finger, which is the joint at the base of the middle finger. This is the most center-position that I can track using iOS Vision.

Additionally, the app computes a few parameters from the detected hand points, like how much your fingers are spread apart, or the angle and tilt of your hand, which can also be used to generate MIDI.

Most values can be measured using the normal 2D camera of the device. For the distance value, the device uses the TrueDepth camera if it is available. If not, it tries to determine the distance from the size of the hand, which is not that accurate as when using the depth info.

If you have a device with a TrueDepth camera, you can have 3 dimensions of control with both your hands simultaneously.

Ports:

This is the position of your hand on the x-axis of the camera image (sideways).

Range: 0.0 to 1.0

This is the position of your hand on the y-axis of the camera image (up-down, or forward-backward when placing the phone on a table).

Range: 0.0 to 1.0

If you have a device with a TrueDepth sensor, this uses the depth measured at the middle of the hand. If there is no TrueDepth sensor in your device, this uses the distance between your wrist and the base of the middle finger to compute the size of your hand on the camera frame. Note that this only works as long as your wrist is visible on the camera.

Range: 0.0 to 1.0

This is the "openness" of your hand. It measures the distance between each finger tip and the thumb tip, so putting those fingers together decreases this value, while spreading the fingers increases it.

Range: 0.0 to 1.0

This is the angle between the middle finger and the x-axis of the camera, so waving towards the left gives you values down to -1.0, while waving to the right increases this value up to 1.0.

Range: -1.0 to 1.0

Tilt the hand sideways to change this value.

Range: 0.0 to 1.0