Video Processing

Dyte's SDK supports middlewares which are pluggable functions that can be applied to both audio and video streams in a meeting. This lets you tap in the media to either observe or modify. This guide covers the following.

1. Create an video middleware
2. Add or remove video middlewares

Create a video middleware

Core

Javascript

All you need to do is create a function that returns another function which gets called for every frame of the video. This function can be synchronous or asynchronous. Actions that need to be performed just once when the video turns on can be done in the outer function. If a user turns their video off and on again, the outer function is called again as well. Actions that need to be performed on every frame are to be done in the inner function.

The inner function is called with the canvas and the context of the video stream. This is the same canvas that is used to render the video stream on the screen. This means that you can perform any operations on the canvas and it will be reflected in realtime. Here is what a middleware function looks like.

async function GrayScaleFilter() {
  console.log('this will be called only once when you turn on the video');
  // Put your async-await API calls AI/ML model fetch here.

  /*
		Returned function (having canvas, ctx as params) can be an async function as well.
		We support async as well as normal functions.
	*/
  return async (canvas, ctx) => {
    // This will be called 20 - 30 times per second
    ctx.filter = 'grayscale(1)';
    ctx.shadowColor = '#000';
    ctx.shadowBlur = 20;
    ctx.lineWidth = 50;
    ctx.strokeStyle = '#000';
    ctx.strokeRect(0, 0, canvas.width, canvas.height);
  };
}

meeting.self.addVideoMiddleware(GrayScaleFilter);

Add or remove video middlewares

Core

Javascript

Middlewares can be added to a meeting, which is why they are associated with a meeting object. To add a video middleware, you need to call the addVideoMiddleware. This method takes in a video middleware function as a parameter, creating this middleware object was covered in the previous section. To remove a video middleware, you need to call the removeVideoMiddleware method. This method also takes in the video middleware function as a parameter.

meeting.self.addVideoMiddleware(YourMiddleware);

// once done, in a later section remove the middleware
meeting.self.removeVideoMiddleware(YourMiddleware);

If you'd instead like to perform operations on the video stream through your own implementation, you can also programatically access the meeting's video stream. This is accessed from the meeting object.

//Somewhere in your codebase
const meeting = await DyteClient.init(...);

// You can get the video track which everybody will see if your camera is turned on
meeting.self.videoTrack()

// You can also get raw video track
meeting.self.rawVideoTrack();