Mix the video streams

Introduction

Through the stream-mixing service, multiple published media streams can compose a single stream , which allows audiences to play just one stream to improve quality and reduce performance cost.

ZEGOCLOUD supports using stream mixing with three methods, manual stream mix, automatic stream mix, and full-automatic stream mix. The following table shows their differences:

Use cases

You can use stream mixing for various use cases. The following are some typical examples:

• When client devices are not capable of processing the required number of streams simultaneously.
• When it is required to synthesize multiple live videos into one single view. For example, to display the teacher's video and the students' videos in a synthesized view.
• When it is required to synthesize multiple audios into one single stream, especially for the voice chatroom scenarios.

Why stream mixing?

• It reduces the complexity of development. For example, when multiple users co-host an interactive live streaming event, it's much easier to receive and play just one single mixed stream rather than to receive and play multiple streams and do the screen layout for each stream individually.
• It reduces the requirements on the client device's performance and network bandwidth. Having more streams to process means more system resources and network bandwidth are required. With stream mixing, the burden of processing multiple streams is removed from the client side.
• It makes it much easier to forward multiple streams to multiple CDNs. You can achieve this by doing simple output stream configurations for stream mixing tasks.
• When it is required to record multiple streams for future playback, the recording can be done on the CDN to which the mixed stream is forwarded.
• Stream mixing also makes content moderation easier because it can be done by checking the mixed stream rather than checking multiple streams individually.

Prerequisites

Before you begin, make sure you complete the following:

• Create a project in ZEGOCLOUD Admin Console and get the AppID and AppSign of your project.
• Refer to the Quick Start doc to complete the SDK integration and basic function implementation.
• Contact us to activate the Stream Mixing service.
  

Manual stream mixing

With the manual stream mixing, you can customize the stream mixing tasks. which is often used for multiple users to co-host an interactive live streaming event and co-hosting across rooms.

To implement the stream mixing with SDK or ZEGO server API, refer to the Start stream mixing and Stop stream mixing.

The following describes how to implement the manual stream mixing in detail.

Set up a stream mixing task

ZegoMixerTask is a class defined in the SDK for configuring a stream mixing task. The configuration parameters include the layout of input streams, the outputs, and others.

/// Stream mixing task object.
@interface ZegoMixerTask: NSObject

-(instancetype)init NS_UNAVAILABLE;

/// Construct a stream mixing task object using TaskID.
-(instancetype)initWithTaskID:(NSString *)taskID;

/// Task ID of the stream mixing task.
@property (nonatomic, copy, readonly) NSString *taskID;

/// Set up the audio configuration of the mixed stream.
-(void)setAudioConfig:(ZegoMixerAudioConfig *)audioConfig;

/// Set up the video configuration of the mixed stream.
-(void)setVideoConfig:(ZegoMixerVideoConfig *)videoConfig;

/// Set up the list of input streams.
-(void)setInputList:(NSArray<ZegoMixerInput *> *)inputList;

/// Set up the list of mixing outputs.
-(void)setOutputList:(NSArray<ZegoMixerOutput *> *)outputList;

/// Set up the watermark of the mixed stream.
-(void)setWatermark:(ZegoWatermark *)watermark;

/// Set up the background image of the mixed stream.
-(void)setBackgroundImageURL:(NSString *)backgroundImageURL;

@end

1. Create a stream mixing task object

Create a stream mixing task object using the constructor initWithTaskID, and then call the instance methods to set up the input and output parameters.

ZegoMixerTask *task = [[ZegoMixerTask alloc] initWithTaskID:@"task-1"];

// Save the stream mixing task object.
self.mixerTask = task;

2. Optional: Set up the video configuration of the mixed stream

ZegoMixerVideoConfig *videoConfig = [ZegoMixerVideoConfig defaultConfig];
[task setVideoConfig:videoConfig];

3. Optional: Set up the audio configuration of the mixed stream

[task setAudioConfig:[ZegoMixerAudioConfig defaultConfig]];

4. Set up the input streams

For each input stream, create a ZegoMixerInput object and set up the video layout settings by setting up the layout property of the ZegoMixerInput object.

The layout of the input stream takes the upper left corner of the mixed output stream view as the origin of the coordinate system. Set the layout of the input stream according to the origin, that is, pass new Rect(left, top, width, height) into the layout parameter of the input stream. In addition, the layer level of the input stream is determined by the position of the input stream in the input stream list. The further back the order in the list, the higher the layer level.

The following describes the Rect parameter:

Let's suppose the following:

• You want to start a stream mixing task with the resolution of an output stream is 375 × 667.
• The coordinates of the upper-left corner of a stream are (50, 300), and the coordinates of its lower-right corner are (200, 450). Then, the layout parameter is new Rect(50, 300, 150, 150).

Then, the position of this stream in the mixed stream can be illustrated as follows:

You can use the following example code to implement common output layouts: two views side by side, four views tiled vertically, one large view tiled with two small views suspended.

In the following examples, the resolution of the video screen layout is set as 360 × 640.

Layout 1: two views side by side

/** Configure the first input stream, including the streamID (the real ID of the input stream), input type, layout, etc.*/
CGRect firstRect = CGRectMake(0, 0, 180, 640);
ZegoMixerInput *firstInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_1" contentType:ZegoMixerInputContentTypeVideo layout:firstRect];
firstInput.renderMode = ZegoMixRenderModeFill;
firstInput.label.text = @"text watermark";
firstInput.label.left = 0;
firstInput.label.font.type = ZegoFontTypeSourceHanSans;
firstInput.label.top = 0;
firstInput.label.font.color = 123456;
firstInput.label.font.size = 24;
firstInput.label.font.transparency = 50;

/** Configue the second input stream. */
CGRect secondRect = CGRectMake(180, 0, 180, 640);
ZegoMixerInput *secondInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_2" contentType:ZegoMixerInputContentTypeVideo layout:secondRect];
secondInput.renderMode = ZegoMixRenderModeFill;
secondInput.label.text = @"text watermark";
secondInput.label.left = 0;
secondInput.label.font.type = ZegoFontTypeSourceHanSans;
secondInput.label.top = 0;
secondInput.label.font.color = 123456;
secondInput.label.font.size = 24;
secondInput.label.font.transparency = 50;

/** Set up the input steam list for the stream mixing task. */

NSArray<ZegoMixerInput *> *inputArray = @[firstInput, secondInput];

[task setInputList:inputArray];

Layout 2: four views tiled vertically

ZegoMixerInput *firstInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_1" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(0, 0, 180, 320)];
firstInput.renderMode = ZegoMixRenderModeFill;
firstInput.label.text = @"text watermark";
firstInput.label.left = 0;
firstInput.label.font.type = ZegoFontTypeSourceHanSans;
firstInput.label.top = 0;
firstInput.label.font.color = 123456;
firstInput.label.font.size = 24;
firstInput.label.font.transparency = 50;

ZegoMixerInput *secondInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_2" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(180, 0, 180, 320)];
secondInput.renderMode = ZegoMixRenderModeFill;
secondInput.label.text = @"text watermark";
secondInput.label.left = 0;
secondInput.label.font.type = ZegoFontTypeSourceHanSans;
secondInput.label.top = 0;
secondInput.label.font.color = 123456;
secondInput.label.font.size = 24;
secondInput.label.font.transparency = 50;

ZegoMixerInput *thirdInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_3" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(0, 320, 180, 320)];
thirdInput.renderMode = ZegoMixRenderModeFill;
thirdInput.label.text = @"text watermark";
thirdInput.label.left = 0;
thirdInput.label.font.type = ZegoFontTypeSourceHanSans;
thirdInput.label.top = 0;
thirdInput.label.font.color = 123456;
thirdInput.label.font.size = 24;
thirdInput.label.font.transparency = 50;

ZegoMixerInput *forthInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_4" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(180, 320, 180, 320)];
forthInput.renderMode = ZegoMixRenderModeFill;
forthInput.label.text = @"text watermark";
forthInput.label.left = 0;
forthInput.label.font.type = ZegoFontTypeSourceHanSans;
forthInput.label.top = 0;
forthInput.label.font.color = 123456;
forthInput.label.font.size = 24;
forthInput.label.font.transparency = 50;

    /** Set up the input steam list for the stream mixing task. */
NSArray<ZegoMixerInput *> *inputArray = @[firstInput, secondInput, thirdInput, forthInput];
[task setInputList:inputArray];

Layout 3: one large view tiled with two small views suspended

The layer level of the input stream is determined by the position of the input stream in the input stream list. The further back the order in the list, the higher the layer level. As shown in the code below, the layer of input stream 2 and input stream 3 is higher than that of input stream 1, so streams 2 and 3 hover over the screen of input stream 1.

ZegoMixerInput *firstInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_1" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(0, 0, 360, 640)];


ZegoMixerInput *secondInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_2" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(230, 200, 110, 200)];

ZegoMixerInput *thirdInput = [[ZegoMixerInput alloc] initWithStreamID:@"streamID_3" contentType:ZegoMixerInputContentTypeVideo layout:CGRectMake(230, 420, 110, 200)];

/** Set up the input steam list for the stream mixing task. */
NSArray<ZegoMixerInput *> *inputArray = @[firstInput, secondInput, thirdInput];
[task setInputList:inputArray];

5. Set up the mixing outputs

A stream mixing task can have up to 3 mixing outputs.

In the following code example, the stream mixing task generates an output with the stream name output-stream, which will be published to the ZEGO server.

NSArray<ZegoMixerOutput *> *outputArray = @[[[ZegoMixerOutput alloc] initWithTarget:@"output-stream"]];
[task setOutputList:outputArray];

6. Optional: Set up the watermark of the mixed stream

ZegoWatermark *watermark = [[ZegoWatermark alloc] initWithImageURL:@"preset-id://zegowp.png" layout:CGRectMake(0, 0, videoConfig.resolution.width/2, videoConfig.resolution.height/20)];
[task setWatermark:watermark];

7. Optional: Set up the background image of the mixed stream

[task setBackgroundImageURL:@"preset-id://zegobg.png"];

8. Optional: Set up the event callback related sound level of the mixed stream

9. Optional: Set up advanced configurations

// Specify the video encoding format of the output stream a vp8, which requires a particular streaming protocol.
NSDictionary *config = @{@"video_encode": @"vp8"};
[task setAdvancedConfig: config];

// If the video coding format of the output stream is set to vp8, then you need to set the audio codec to LOW3 to make it take effect.
ZegoMixerAudioConfig *audioConfig = [ZegoMixerAudioConfig defaultConfig];
audioConfig.codecID = ZegoAudioCodecIDLow3;
[task setAudioConfig:audioConfig];

Start the stream mixing task

After configuring the ZegoMixerTask object, call the startMixerTask method to start the stream mixing task. If the stream mixing task doesn't start successfully, handle the failure in the callback that returns the execution result.

• Function prototype:

/// Start the stream mixing task.
/// @param task: The stream mixing task object.
/// @param callback: The callback to return the result of starting up the task.
-(void)startMixerTask:(ZegoMixerTask *)task callback:(nullable ZegoMixerStartCallback)callback;

• Sample code:

[self.engine startMixerTask:task callback:^(ZegoMixerStartResult * _Nonnull result) {
    if (result.errorCode == 0) {
        NSLog(@"Start mixing task success.");
    } else {
        NSLog(@"Start mixing task failed.");
    }
}];

Update the configurations of the stream mixing task

When you need to change the stream mixing task, for example, when the number of input streams increases or decrease, or the bitrate of the output stream needs to be adjusted, you can update the stream mixing task object with the new configurations and then call the startMixerTask method again.

The following sample code demonstrates adding an input stream to a stream mixing task that is in progress, with a top-middle-bottom vertical layout.

CGRect firstRect = CGRectMake(0, 0, videoConfig.resolution.width, videoConfig.resolution.height/3);
ZegoMixerInput *firstInput = [[ZegoMixerInput alloc] initWithContentType:ZegoMixerInputContentTypeVideo streamID:@"stream-1" layout:firstRect];

CGRect secondRect = CGRectMake(0, videoConfig.resolution.height/3, videoConfig.resolution.width, videoConfig.resolution.height*(2/3));
ZegoMixerInput *secondInput = [[ZegoMixerInput alloc] initWithContentType:ZegoMixerInputContentTypeVideo streamID:@"stream-2" layout:secondRect];

CGRect thirdRect =CGRectMake(0, videoConfig.resolution.height*(2/3), videoConfig.resolution.width, videoConfig.resolution.height);
ZegoMixerInput *thirdInput = [[ZegoMixerInput alloc] initWithContentType:ZegoMixerInputContentTypeVideo streamID:@"stream-3" layout:thirdRect];

NSArray<ZegoMixerInput *> *inputArray = @[firstInput, secondInput, thirdInput];

// Reconfigure the input stream list. 
[self.mixerTask setInputList:inputArray];

// Call the startMixerTask method again to let the new configurations take effect.
[self.engine startMixerTask:self.mixerTask callback:^(ZegoMixerStartResult * _Nonnull result) {
    if (result.errorCode == 0) {
        NSLog(@"Start mixer task success");
    } else {
        NSLog(@"Start mixer task fail");
    }
}];

Stop the stream mixing task

To stop a stream mixing task, call the stopMixerTask method, with the corresponding task ID passed to the taskID parameter.

• Function prototype:

/// Stop the stream mixing task.
/// @param taskID: The taskID of the steam mixing task to be stopped.
-(void)stopMixerTask:(NSString *)taskID;

• Sample code:

// Pass in the taskID of the steam mixing task to be stopped.
[self.engine stopMixerTask:self.mixerTask.taskID];

Automatic stream mixing

Set up a stream mixing task

ZegoAutoMixerTask is a class defined in the SDK for configuring a stream mixing task. The configuration parameters include the layout of input streams, the outputs, and others.

/// Stream mixing task object.
@interface ZegoAutoMixerTask : NSObject

/// Task ID of the stream mixing task.
@property (nonatomic, copy) NSString *taskID;

/// Room ID of the stream mixing task.
@property (nonatomic, copy) NSString *roomID;

/// Set up the audio configuration of the mixed stream.
@property (nonatomic, strong) ZegoMixerAudioConfig *audioConfig;

/// Set up the list of mixing outputs.
@property (nonatomic, strong) NSArray<ZegoMixerOutput *> *outputList;

/// Set up the event callback related sound level of the mixed stream.
@property (nonatomic, assign) BOOL enableSoundLevel;

@end

1. Create an auto stream mixing task object

Create an auto stream mixing task object and then call the instance methods to set up the input and output parameters.

• The task ID must be unique within a room. We recommend you set the room ID as the task ID.
• The room ID must have already existed; otherwise, the stream mixing will fail.

ZegoAutoMixerTask *task = [[ZegoAutoMixerTask alloc] init];

task.taskID = @"taskID1";
task.roomID = @"roomID1";

2. Optional: Set up the audio configuration of the auto mixed stream

ZegoMixerAudioConfig *audioConfig = [ZegoMixerAudioConfig init];
audioConfig.bitrate = 48;/** Audio bitrate, unit: kbps, use 48 kbps by default. This can't be changed after stream mixing starts. */
audioConfig.channel = ZegoAudioChannelMono;/** Audio channel, use ZegoAudioChannelMono by default. */
audioConfig.codecID = ZegoAudioCodecIDNormal;/** Codec ID, use ZEGO_AUDIO_CODEC_ID_DEFAULT by default.*/
audioConfig.mixMode = ZegoAudioMixModeRaw;/** Audio mix mode, use ZegoAudioMixModeRaw by default.  */
[task setAudioConfig:audioConfig];

3. Set up the auto mixing outputs

To set up the auto mixing outputs, call the ZegoMixerOutput method to set the output list. And you can play the mixed stream according to the output-stream of the outputs.

NSArray<ZegoMixerOutput *> *outputArray = @[[[ZegoMixerOutput alloc] initWithTarget:@"output-stream"]];/** Mixing output target, URL or streamID. */
[task setOutputList:outputArray];
```tOutputList:outputArray];

4. Optional: Set up the event callback related sound level of the auto mixed stream

Start the auto stream mixing task

After configuring the ZegoAutoMixerTask object, call the startAutoMixerTask method to start the stream mixing task. And the SDK sends out the result notifications through the onAutoMixerSoundLevelUpdate callback.

[[ZegoExpressEngine sharedEngine] startAutoMixerTask:task callback:^(int errorCode, NSDictionary * _Nullable extendedData) {
    if (errorCode == 0) {
        /// Auto stream mixing task started successfully. 
    }
}];

Stop the auto stream mixing task

To stop a stream mixing task, call the stopAutoMixerTask method.

//  Pass in the task object of the steam mixing task to be stopped.
[[ZegoExpressEngine sharedEngine] stopAutoMixerTask:self.autoMixerTask callback:^(int errorCode) {
    if(errorCode == 0) {
        ///Auto stream mixing task stopped successfully.
    }
}];

Full-automatic stream mixing

To fully-automatical mix the audio streams in each room, contact the ZEGOCLOUD Technical Support.

FAQ

1. Can I forward the mixed stream to a third-party CDN or even multiple CDNs?

   Yes. To forward the mixed stream to a third-party CDN, specify the CDN URL in the target property of the ZegoMixerOutput object. The URL must be in RTMP format: rtmp://xxxxxxxx.

   To forward the mixed stream to multiple CDNs, create multiple ZegoMixerOutput objects accordingly, and put them into the outputList of the ZegoMixerTask object.

2. When the aspect ratio of the ZegoMixerInput's layout parameter doesn't match the resolution of the input stream itself, how does the SDK handle the video cropping?

   The SDK scales the video proportionally.

   Let's suppose the following:

   • The resolution of an input stream is 720 × 1280, whose aspect ratio is 9:16.
   • The layout parameter of the ZegoMixerInput object is CGRectMake(0, 0, 100, 100), whose aspect ratio is 1:1.

   Then, the SDK displays the middle part of the video and cuts off the upper part and lower part.

3. The hosts co-hosting a live streaming session want their respective viewers to see their own video in the main (larger) video view on the screen after mixing. How to do the stream mixing for them?

   The hosts can start a stream mixing task with their own output stream layout.

   Let's suppose the following:

   • Host A and Host B are co-hosting a live streaming session.

   • Host A is publishing Stream A.

   • Host B is publishing Stream B.

   Then, Host A and Host B can respectively start a stream mixing task with different video layouts:

   • In Host A's stream mixing task, the video size (width and height) of stream A is larger than that of stream B.
   • In Host B's stream mixing task, the video size (width and height) of stream B is larger than that of stream A.

4. For a live streaming session with two hosts, what are the pros and cons of the following two options for starting the stream mixing?

   A. To start the stream mixing immediately after the first host starts publishing a stream.

   B. To start the stream mixing only after both hosts start publishing streams.

   Option	Pros	Cons
   Option A	It's easy to implement. The viewers can play the mixed stream right at the beginning and don't need to do any stream switching in the middle.	It costs more CDN usage because the stream mixing starts earlier.
   Option B	It costs less CDN usage because the stream mixing starts later.	The implementation is more complicated compared to option A. The viewers need to play the first host's stream at the beginning and then switch to the mixed stream later.

5. Does the SDK support round or square video layout for stream mixing?

   The SDK doesn't support a round video layout for stream mixing.

   You can set up square video layouts by configuring the layout parameter of the input streams.