How to run @mediapipe/task-vision in a Web Worker

You can apply the same pattern to other MediaPipe models/tasks, including Object Detection, Interactive Segmentation etc.

Mediapipe + Web Workers. Image created by the Ankit Kumar. Mediapipe logo © Google.

Setting up development environment

For the sake of simplicity, we'll set up our environment using Vite and plain JavaScript. You can find all the source code in my GitHub repository, which also includes examples of how to integrate Mediapipe's Pose Landmarker with Web Workers in different frameworks.

Create a new Vite project:

npm create vite@latest

Then follow the prompts to setup the project.

Running MediaPipe in the main thread

Before introducing Web Workers, we'll first run Mediapipe directly on the main thread to confirm that everything is set up correctly and working as expected.

Start by removing the boilerplate files and the following import from main.js:

import * as $mediapipe from "https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/+esm";

This import statement loads all exports from @mediapipe/tasks-vision and places them under the $mediapipe namespace.

Next, we'll initialize MediaPipe's PoseLandmarker:

let poseLandmarker = null;
/**
 * This function does the following:
 * 1. Create MediaPipe PoseLandmarker detector using model path and assign it global poseLandmarker.
 * 2. Set running mode to VIDEO.
 */
async function initializeMediapipePoseLandmarker() {
  const vision = await $mediapipe.FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm",
  );

  poseLandmarker = await $mediapipe.PoseLandmarker.createFromModelPath(
    vision,
    "https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task",
  );

  await poseLandmarker.setOptions({
    runningMode: "VIDEO",
  });
}

Once initialized, we need a function to run pose detection:

/**
 * Detecting the pose.
 */
async function detectPose(imageOrVideoElement) {
  if (!poseLandmarker) {
    throw new Error("PoseLandmarker is not initialized");
  }

  const timestamp = performance.now();
  return new Promise((resolve) => {
    poseLandmarker.detectForVideo(imageOrVideoElement, timestamp, (result) => {
      resolve(result);
    });
  });
}

Finally, you'll need some additional code to request camera access and continuously run pose detection. You can find the complete example here.

This section helps you set up the MediaPipe PoseLandmarker on the main thread. Now, let's move on to the next part, where we'll learn how to configure and run it inside a Web Worker.

Running MediaPipe inside Web Worker

TODO.