Real-Time Audio and Video
  • Platform
  • Framework / Engine
  • iOS
  • Android
  • macOS
  • Windows
  • Linux
  • Web
  • WeChat Mini Program

Implement a video call with beautification features

Update Time:2021-09-17 11:33

1 Introduction

1.1 Introduction

The real-time audio and video service provided by ZEGO enables you to build audio and video applications through its flexible and easy-to-use API. Meanwhile, another ZEGO product -- AI Effects, is based on the AI algorithm, which enables you to implement a series of beautification features such as face beautification, face shape retouch, and other features.

These two services can be combined by using the two types of SDK together to create a real-time application with beautification features, which can be widely used in entertainment live streaming, live game streaming, video conference, and other live streaming scenarios.

1.2 Basic concepts

  • The ZegoExpress-Video SDK (Hereafter called the Express SDK): The real-time audio and video SDK provided by ZEGO. This SDK enables you to implement real-time audio and video features, in live streaming, live co-hosting streaming, and other scenarios.

  • The ZegoEffects SDK (Hereafter called the Effects SDK): The AI effects SDK provided by ZEGO provides the AI-based image rendering and algorithm abilities that enable you to implement face beautification, face shape retouch, background segmentation, face detection, and other features.

2 Sample code

ZEGO provides the sample code to implement these two features at one time, for information about how to download and run the sample code, contact the ZEGO team.

3 Prerequisites

  1. Create a project in ZEGO Admin Console, and get the AppID and AppSign of your project. For details, see ZEGO Admin Console - Project management.

  2. Integrate the ZEGO Express SDK into your project. For details, see Integration.

  3. Integrate the ZEGO Effects SDK into your project. For details, see Integration.

  4. Get the unique license file of the ZEGO Effects SDK. For details, see Online privilege authentication.

4 Implementation steps

The overall process of the combination of the two SDKs is as follows:

  1. Initialize the Express SDK and the Effects SDK (in no particular order).
  2. Obtain the raw video data. For details, see Custom video capture or Custom video pre-processing.
  3. Process the raw video data for beautification effects using the Effects SDK, and send the process video data, publish the video streams using the Express SDK.
  4. To adjust the AI effects during the stream publishing and playing operation, you can use the related functions of the Effects SDK to make changes in real time.

4.1 Initialize the Effects SDK

4.1.1 Import the resources and models of the Effects SDK

Before using the AI features of the Effects SDK, you need to import the resources or models required for those features.

// Specify the absolute path of the face recognition model, which is required for features such as face detection, eyes enlarging, and face slimming.
NSString *faceDetectionModelPath = [[NSBundle mainBundle] pathForResource:@"FaceDetectionModel" ofType:@"bundle"];
// Specify the absolute path of the portrait segmentation model, which is required for the portrait segmentation feature.
NSString *segmentationModelPath = [[NSBundle mainBundle] pathForResource:@"SegmentationModel" ofType:@"bundle"];

// Specify the absolute path of the stickers resource.
NSString *pendantBundlePath = [[NSBundle mainBundle] pathForResource:@"PendantResources" ofType:@"bundle"];
// Specify the absolute path of the skin tone enhancement resource.
NSString *whitenBundlePath = [[NSBundle mainBundle] pathForResource:@"FaceWhiteningResources" ofType:@"bundle"];

// Set the path list for resources or models, which must be called before calling the `create` method to create a ZegoEffects object.
[ZegoEffects setResources:@[faceDetectionModelPath, SegmentationModel, pendantBundlePath, whitenBundlePath]];

For all resources and models that the Effects SDK supports, see Import resources and models.

4.1.2 Create an Effects SDK object

To create an Effects SDK object, import the authentication file you obtained in the previous step 3 Prerequisites.

// Pass in the authentication file you obtained.
ZegoEffects *effects = [ZegoEffects create:@"ABCDEFG"];
// Save the  Effects SDK instance. 
self.effects = effects;

4.1.3 Initialize the Effects SDK object

To initialize the Effects SDK object, call the initEnv method, and pass in the width and height of the incoming video data to be processed.

The following is the sample code of processing 1280 × 720 video images:

// Initialize the Effects SDK object, and pass in the width and height of the incoming video data to be processed.
[self.effects initEnv:CGSizeMake(1280, 720)];

4.2 Initialize the Express SDK

To initialize the Express SDK, call the createEngineWithAppID method.

[ZegoExpressEngine createEngineWithAppID:appID appSign:appSign isTestEnv:YES scenario:ZegoScenarioGeneral eventHandler:self];

4.3 Get the video raw data

The Express SDK provides two methods to get the video raw data:

  • Custom video pre-processing: The Express SDK collects video raw data internally, and the SDK sends out the captured video raw data through a callback.
  • Custom video capture: The video raw data is collected by developers themselves and then provided to the Express SDK.

The difference between the two methods is as follows, you can choose based on the actual situation:

Method Description Advantages
Custom video pre-processing The ZEGO Express SDK collects the video raw data. Together with the Express SDK and the Effects SDK, you do not need to manage the device input sources, but simply manipulate the raw data thrown by the Express SDK and pass it back to the Express SDK.
Custom video capture Capture the video raw data by yourself. When multiple manufacturers are taken on, services can be flexibly implemented and performance optimization can be improved.
  • Method 1: Custom video pre-processing

To capture the video raw data using this method, do the following:

a. Select the CVPixelBuffer video frame data type.

b. Call the enableCustomVideoProcessing method to enable the custom video pre-processing.

c. The SDK sends out the captured video raw data through the callback onCapturedUnprocessedCVPixelBuffer.

ZegoCustomVideoProcessConfig *processConfig = [[ZegoCustomVideoProcessConfig alloc] init];
// Select the [CVPixelBuffer] as video frame data type.
processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer;

// Enable the custom video pre-processing.
[[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing:YES config:processConfig channel:ZegoPublishChannelMain];

// Set [self] as the event handler object of the custom video pre-processing callback.
[[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler:self];

For details, see Custom video pre-processing.

  • Method 2: Custom video capture

For details, see Custom video capture.

4.4 Process the video data with the Effects SDK

In the corresponding callback for receiving the video raw data, the Effects SDK can be used to implement AI features.

After implemented the AI features, the Express SDK encodes and publishes the processed data and sends it to the cloud server. At this time, the remote user can play the processed video streams.

  • If you capture the video data using the custom video pre-processing method, to process the captured video data for AI features, do the following:
  1. Process the video data in the onCapturedUnprocessedCVPixelBuffer callback.
  2. After processed, send the video data back to the Express SDK.
// Take using the custom video pre-processing method as an example.
// Obtain the raw video data through a callback.
- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel {
    ...
    // Custom video pre-processing: use the Effects SDK here.
    [self.effects processImageBuffer:buffer];

    // Send the processed buffer to the Express SDK.
    [[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer:output timestamp:timestamp channel:channel];
    ...
}
  • If you capture the video data using the custom video capture method, to process the captured video data for AI features, do the following:
  1. Process the video data in the corresponding callback (for details, see Custom video capture - 4.3 ),
  2. Send the processed data back to the Express SDK.

4.5 Adjust the beautification features using the Effects SDK

To adjust the AI effects during the stream publishing and playing operation, use the Effects SDK to make changes in real time.

// Enable the skin tone enhancement feature.
[self.effects enableWhiten:YES];

// Set the whitening intensity. The value range is [0, 100], and the default value is 50.
ZegoEffectsWhitenParam *param = [[ZegoEffectsWhitenParam alloc] init];
param.intensity = 100;
[self.effects setWhitenParam:param];

For more AI features, see Face beautification, Face shape retouch, Backgroud segmentation, Face detection, Stickers, and Filters.