npm install @mediapipe/tasks-vision

The segmenter plugin uses MediaPipe's Image Segmenter and exposes ShaderPad mask textures and GLSL helper functions.

import segmenter from 'shaderpad/plugins/segmenter';

const shader = new ShaderPad(fragmentShaderSrc, {
	plugins: [
		segmenter({
			textureName: 'u_webcam',
			options: { outputConfidenceMasks: true },
		}),
	],
});

The plugin reads from the textureName texture. Initialize and update that exact ShaderPad texture name, or the detector will have no source to read from.

Options

Events

Subscribe with shader.on(name, callback).

shader.on('segmenter:result', result => {
	console.log(result.categoryMask);
});

Uniforms

Most shaders should use the helper functions below instead of sampling u_segmentMask directly.

Helper Functions

If history is enabled, every helper below also has a a trailing int framesAgo argument. 0 means the current analyzed frame, 1 means the previous stored frame, and so on.

segmentAt

vec2 segmentAt(vec2 pos)
vec2 segmentAt(vec2 pos, int framesAgo)

Returns vec2(confidence, normalizedCategoryIndex).

• x: confidence for the winning category at pos
• y: normalized category index in the range 0.0 to 1.0

When outputConfidenceMasks is false, the confidence component is always 1.0.

If you need the integer category index in GLSL, recover it with:

int categoryIndex = int(floor(segment.y * float(u_numCategories - 1) + 0.5));

For instance:

vec2 segment = segmentAt(v_uv);
float confidence = segment.x;
int categoryIndex = int(floor(segment.y * float(u_numCategories - 1) + 0.5));
float isForeground = float(categoryIndex != 0) * confidence;
color.rgb = mix(color.rgb, vec3(1.0, 0.0, 1.0), isForeground);