zqz/platform-external-webrtc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Support native scaling of VideoFrameBuffers in LibvpxVp9Encoder.

Browse Source 
This CL is part of Optimized Scaling efforts. In Chromium, the native
frame buffer is getting an optimized CropAndScale() implementation. To
support HW accelerated scaling, returning pre-scaled images and skipping
unnecessary intermediate downscales, WebRTC needs to 1) use CropAndScale
instead of libyuv::XXXXScale and 2) only map buffers it actually intends
to encode.
- To achieve this, WebRTC encoders are updated to map kNative video
  buffers so that in a follow-up CL VideoStreamEncoder can stop mapping
  intermediate buffer sizes.

In this CL LibvpxVp9Encoder is updated to map kNative buffers of pixel
formats it supports and convert ToI420() if the kNative buffer is
something else. A fake native buffer that keeps track of which
resolutions were mapped, MappableNativeBuffer, is added.

Because VP9 is currently an SVC encoder and not a simulcast encoder, it
does not need to invoke CropAndScale.

This CL also fixes MultiplexEncoderAdapter, but because it simply
forwards frames it only cares about the pixel format when
|supports_augmented_data_| is true so this is the only time we map it.
Because this encoder is not used with kNative in practise, we don't care
to make this path optimal.

Bug: webrtc:12469, chromium:1157072
Change-Id: I74edf85b18eccd0d250776bbade7a6444478efce
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/212580
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#33526}
This commit is contained in:
Henrik Boström
2021-03-22 12:24:30 +01:00
committed by Commit Bot
parent 2ff25db72a
commit bd9e4a95eb
10 changed files with 488 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
modules/video_coding/codecs/multiplex/multiplex_encoder_adapter.cc
View File

@ -157,20 +157,38 @@ int MultiplexEncoderAdapter::Encode(
return WEBRTC_VIDEO_CODEC_UNINITIALIZED;
}
// The input image is forwarded as-is, unless it is a native buffer and
// |supports_augmented_data_| is true in which case we need to map it in order
// to access the underlying AugmentedVideoFrameBuffer.
VideoFrame forwarded_image = input_image;
if (supports_augmented_data_ &&
forwarded_image.video_frame_buffer()->type() ==
VideoFrameBuffer::Type::kNative) {
auto info = GetEncoderInfo();
rtc::scoped_refptr<VideoFrameBuffer> mapped_buffer =
forwarded_image.video_frame_buffer()->GetMappedFrameBuffer(
info.preferred_pixel_formats);
if (!mapped_buffer) {
// Unable to map the buffer.
return WEBRTC_VIDEO_CODEC_ERROR;
}
forwarded_image.set_video_frame_buffer(std::move(mapped_buffer));
}
std::vector<VideoFrameType> adjusted_frame_types;
if (key_frame_interval_ > 0 && picture_index_ % key_frame_interval_ == 0) {
adjusted_frame_types.push_back(VideoFrameType::kVideoFrameKey);
} else {
adjusted_frame_types.push_back(VideoFrameType::kVideoFrameDelta);
}
const bool has_alpha = input_image.video_frame_buffer()->type() ==
const bool has_alpha = forwarded_image.video_frame_buffer()->type() ==
VideoFrameBuffer::Type::kI420A;
std::unique_ptr<uint8_t[]> augmenting_data = nullptr;
uint16_t augmenting_data_length = 0;
AugmentedVideoFrameBuffer* augmented_video_frame_buffer = nullptr;
if (supports_augmented_data_) {
augmented_video_frame_buffer = static_cast<AugmentedVideoFrameBuffer*>(
input_image.video_frame_buffer().get());
forwarded_image.video_frame_buffer().get());
augmenting_data_length =
augmented_video_frame_buffer->GetAugmentingDataSize();
augmenting_data =
@ -185,7 +203,7 @@ int MultiplexEncoderAdapter::Encode(
MutexLock lock(&mutex_);
stashed_images_.emplace(
std::piecewise_construct,
std::forward_as_tuple(input_image.timestamp()),
std::forward_as_tuple(forwarded_image.timestamp()),
std::forward_as_tuple(
picture_index_, has_alpha ? kAlphaCodecStreams : 1,
std::move(augmenting_data), augmenting_data_length));
@ -194,7 +212,8 @@ int MultiplexEncoderAdapter::Encode(
++picture_index_;
// Encode YUV
int rv = encoders_[kYUVStream]->Encode(input_image, &adjusted_frame_types);
int rv =
encoders_[kYUVStream]->Encode(forwarded_image, &adjusted_frame_types);
// If we do not receive an alpha frame, we send a single frame for this
// |picture_index_|. The receiver will receive |frame_count| as 1 which
@ -206,23 +225,24 @@ int MultiplexEncoderAdapter::Encode(
rtc::scoped_refptr<VideoFrameBuffer> frame_buffer =
supports_augmented_data_
? augmented_video_frame_buffer->GetVideoFrameBuffer()
: input_image.video_frame_buffer();
: forwarded_image.video_frame_buffer();
const I420ABufferInterface* yuva_buffer = frame_buffer->GetI420A();
rtc::scoped_refptr<I420BufferInterface> alpha_buffer =
WrapI420Buffer(input_image.width(), input_image.height(),
WrapI420Buffer(forwarded_image.width(), forwarded_image.height(),
yuva_buffer->DataA(), yuva_buffer->StrideA(),
multiplex_dummy_planes_.data(), yuva_buffer->StrideU(),
multiplex_dummy_planes_.data(), yuva_buffer->StrideV(),
// To keep reference alive.
[frame_buffer] {});
VideoFrame alpha_image = VideoFrame::Builder()
.set_video_frame_buffer(alpha_buffer)
.set_timestamp_rtp(input_image.timestamp())
.set_timestamp_ms(input_image.render_time_ms())
.set_rotation(input_image.rotation())
.set_id(input_image.id())
.set_packet_infos(input_image.packet_infos())
.build();
VideoFrame alpha_image =
VideoFrame::Builder()
.set_video_frame_buffer(alpha_buffer)
.set_timestamp_rtp(forwarded_image.timestamp())
.set_timestamp_ms(forwarded_image.render_time_ms())
.set_rotation(forwarded_image.rotation())
.set_id(forwarded_image.id())
.set_packet_infos(forwarded_image.packet_infos())
.build();
rv = encoders_[kAXXStream]->Encode(alpha_image, &adjusted_frame_types);
return rv;
}