MusiKraken

Face Tracking

Face Tracking accesses the camera image to detect head rotations and face features. This means you can use your face to generate MIDI events. So you can use your head, mouth, eyes and other parts of your face to generate MIDI Control Change events (Modulation, Volume, Expression and so on) or generate notes.

On iOS, there are two modules for this: The ARKit Face Tracking module and a Mouth Tracking module.

The reason why there are two different Face Tracking modules is that the Mouth Tracker also runs on devices that do not support ARKit. But all newer devices should support ARKit face tracking now, but if not, use the Mouth Tracker.

On Android, the Face Tracking module now can track most of the parameters that the ARKit Face Tracker can (with a few exceptions).

Ports:

The Face Tracking module has value output ports for all kinds of facial parameters that are being tracked. This means you can make music using the face rotation, mouth, eyes, eyebrows, cheek, jaw, tongue and nose position.

xRotation:

This is the rotation along the x-axis of your face. Can be changed by moving your head to look down or up.

Range: -1.0 to 1.0

yRotation:

This is the rotation along the y-axis of your face. Can be changed by moving your head to look left or right.

Range: -1.0 to 1.0

zRotation:

This is the rotation along the z-axis of your face. Can be changed by turning your head sideways.

Range: -1.0 to 1.0

Mouth Open:

This value is controlled by how much you open your mouth.

Range: 0.0 to 1.0

Left Eye:

This value is controlled by how much you close your left eye. Note that open is 0 and closed is 1 (the opposite of the mouth open value).

Range: 0.0 to 1.0

Right Eye:

This value is controlled by how much you close your right eye. Note that open is 0 and closed is 1 (the opposite of the mouth open value).

Range: 0.0 to 1.0

Tongue:

This value is controlled by how much you stick out your tongue.

Range: 0.0 to 1.0

Left Eyebrow:

This value is controlled by moving your left eyebrow up and down.

Range: -1.0 to 1.0

Right Eyebrow:

This value is controlled by moving your right eyebrow up and down. Most users will probably only need left or right, not both, because they usually send the same value. Except if you can move them independently...

Range: -1.0 to 1.0

Left Look: X

This value is controlled by looking left or right. This tracks the pupil of your left eye.

Range: -1.0 to 1.0

Left Look Y:

This value is controlled by looking up or down with your left eye.

Range: -1.0 to 1.0

Right Look: X

This value is controlled by looking left or right. This tracks the pupil of your right eye.

Range: -1.0 to 1.0

Right Look Y:

This value is controlled by looking up or down with your right eye.

Range: -1.0 to 1.0

Cheek:

This value is controlled by puffing your cheeks.

Range: 0.0 to 1.0

Jaw Sideways:

This value is controlled by moving your jaw sideways.

Range: -1.0 to 1.0

Smile Left:

This value is controlled by smiling (or frowning, for negative values). This measures the smile on the left side of your mouth.

Range: -1.0 to 1.0

Smile Right:

This value is controlled by smiling (or frowning, for negative values). This measures the smile on the right side of your mouth.

Range: -1.0 to 1.0

noseX:

This value is controlled by the x-position of your nose on the camera image.

Range: 0.0 to 1.0

noseY:

This value is controlled by the y-position of your nose on the camera image.

Range: 0.0 to 1.0

distance:

This value is controlled by the distance of your face to the camera (= either face size relative to camera image, or TrueDepth distance value of your nose).

Range: 0.0 to 1.0