npm install @mediapipe/tasks-vision

The pose plugin uses MediaPipe's Pose Landmarker and exposes landmark and mask data in a ShaderPad-friendly GLSL format.

import pose from 'shaderpad/plugins/pose';

const shader = new ShaderPad(fragmentShaderSrc, {
	plugins: [pose({ textureName: 'u_webcam', options: { maxPoses: 2 } })],
});

The plugin reads from the textureName texture. Initialize and update that exact ShaderPad texture name, or the detector will have no source to read from.

Options

Events

Subscribe with shader.on(name, callback).

shader.on('pose:result', result => {
	console.log(`${result.landmarks.length} poses detected`);
});

Uniforms

Most shaders should use the helper functions below instead of sampling u_poseLandmarksTex or u_poseMask directly.

Helper Functions

If history is enabled, every helper below also has a a trailing int framesAgo argument. 0 means the current analyzed frame, 1 means the previous stored frame, and so on.

nPosesAt

int nPosesAt(int framesAgo)

Returns the number of poses stored for the specified frame.

poseLandmark

vec4 poseLandmark(int poseIndex, int landmarkIndex)
vec4 poseLandmark(int poseIndex, int landmarkIndex, int framesAgo)

Returns vec4(x, y, z, visibility).

• x, y: normalized landmark position in ShaderPad UV space
• z: MediaPipe landmark depth value
• w: landmark visibility / confidence

vec2 leftHip = vec2(poseLandmark(0, POSE_LANDMARK_LEFT_HIP));
vec2 rightHip = vec2(poseLandmark(0, POSE_LANDMARK_RIGHT_HIP));

poseAt

vec2 poseAt(vec2 pos)
vec2 poseAt(vec2 pos, int framesAgo)

Returns vec2(confidence, poseIndex).

• x: segmentation confidence for the sampled pixel
• y: the matching poseIndex, or -1.0 when no pose matched

This is the fastest way to ask “which pose owns this pixel?” without manually decoding u_poseMask.

vec2 hit = poseAt(v_uv);
if (hit.y >= 0.0) {
  float confidence = hit.x;
  int poseIndex = int(hit.y);
  vec2 torso = vec2(poseLandmark(poseIndex, POSE_LANDMARK_TORSO_CENTER));
  color.rgb = mix(color.rgb, vec3(torso, confidence), confidence);
}

inPose

float inPose(vec2 pos)
float inPose(vec2 pos, int framesAgo)

Returns the confidence component from poseAt(). This is 0.0 when no pose matched, otherwise the pose-mask confidence for that pixel.

Landmark Layout

The plugin exposes MediaPipe’s standard 33 pose landmarks plus six derived landmarks:

• POSE_LANDMARK_BODY_CENTER
• POSE_LANDMARK_LEFT_HAND_CENTER
• POSE_LANDMARK_RIGHT_HAND_CENTER
• POSE_LANDMARK_LEFT_FOOT_CENTER
• POSE_LANDMARK_RIGHT_FOOT_CENTER
• POSE_LANDMARK_TORSO_CENTER

Some landmark indices are given constants for convenience:

• POSE_LANDMARK_LEFT_EYE
• POSE_LANDMARK_RIGHT_EYE
• POSE_LANDMARK_LEFT_SHOULDER
• POSE_LANDMARK_RIGHT_SHOULDER
• POSE_LANDMARK_LEFT_ELBOW
• POSE_LANDMARK_RIGHT_ELBOW
• POSE_LANDMARK_LEFT_HIP
• POSE_LANDMARK_RIGHT_HIP
• POSE_LANDMARK_LEFT_KNEE
• POSE_LANDMARK_RIGHT_KNEE

For the full MediaPipe landmark index map, use the upstream Pose Landmarker model reference.