Add documentation for video/adaptation
Bug: webrtc:12564 Change-Id: I24e807be6e7bbf1cd6d8b7ed0fa25bde6b257f34 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/215078 Commit-Queue: Evan Shrubsole <eshr@google.com> Reviewed-by: Artem Titov <titovartem@webrtc.org> Reviewed-by: Henrik Andreassson <henrika@webrtc.org> Reviewed-by: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#33728}
This commit is contained in:
Evan Shrubsole
committed by
Commit Bot
Commit Bot
parent
24bc419303
commit
08d30a2a38
@ -20,6 +20,7 @@
|
||||
* AudioProcessingModule
|
||||
* [APM](/modules/audio_processing/g3doc/audio_processing_module.md)
|
||||
* Video
|
||||
* [Adaptation](/video/g3doc/adaptation.md)
|
||||
* DataChannel
|
||||
* PeerConnection
|
||||
* Desktop capture
|
||||
|
||||
115
video/g3doc/adaptation.md
Normal file
115
video/g3doc/adaptation.md
Normal file
@ -0,0 +1,115 @@
|
||||
<?% config.freshness.reviewed = '2021-04-13' %?>
|
||||
|
||||
<?% config.freshness.owner = 'eshr' %?>
|
||||
|
||||
# Video Adaptation
|
||||
|
||||
Video adaptation is a mechanism which reduces the bandwidth or CPU consumption
|
||||
by reducing encoded video quality.
|
||||
|
||||
## Overview
|
||||
|
||||
Adaptation occurs when a _Resource_ signals that it is currently underused or
|
||||
overused. When overused, the video quality is decreased and when underused, the
|
||||
video quality is increased. There are currently two dimensions in which the
|
||||
quality can be adapted: frame-rate and resolution. The dimension that is adapted
|
||||
is based on the degradation preference for the video track.
|
||||
|
||||
## Resources
|
||||
|
||||
_Resources_ monitor metrics from the system or the video stream. For example, a
|
||||
resource could monitor system temperature or the bandwidth usage of the video
|
||||
stream. A resource implements the [Resource][resource.h] interface. When a
|
||||
resource detects that it is overused, it calls `SetUsageState(kOveruse)`. When
|
||||
the resource is no longer overused, it can signal this using
|
||||
`SetUsageState(kUnderuse)`.
|
||||
|
||||
There are two resources that are used by default on all video tracks: Quality
|
||||
scaler resource and encode overuse resource.
|
||||
|
||||
### QP Scaler Resource
|
||||
|
||||
The quality scaler resource monitors the quantization parameter (QP) of the
|
||||
encoded video frames for video send stream and ensures that the quality of the
|
||||
stream is acceptable for the current resolution. After each frame is encoded the
|
||||
[QualityScaler][quality_scaler.h] is given the QP of the encoded frame. Overuse
|
||||
or underuse is signalled when the average QP is outside of the
|
||||
[QP thresholds][VideoEncoder::QpThresholds]. If the average QP is above the
|
||||
_high_ threshold, the QP scaler signals _overuse_, and when below the _low_
|
||||
threshold the QP scaler signals _underuse_.
|
||||
|
||||
The thresholds are set by the video encoder in the `scaling_settings` property
|
||||
of the [EncoderInfo][EncoderInfo].
|
||||
|
||||
*Note:* that the QP scaler is only enabled when the degradation preference is
|
||||
`MAINTAIN_FRAMERATE` or `BALANCED`.
|
||||
|
||||
### Encode Usage Resource
|
||||
|
||||
The [encoder usage resource][encode_usage_resource.h] monitors how long it takes
|
||||
to encode a video frame. This works as a good proxy measurement for CPU usage as
|
||||
contention increases when CPU usage is high, increasing the encode times of the
|
||||
video frames.
|
||||
|
||||
The time is tracked from when frame encoding starts to when it is completed. If
|
||||
the average encoder usage exceeds the thresholds set, *overuse* is triggered.
|
||||
|
||||
### Injecting other Resources
|
||||
|
||||
A custom resource can be injected into the call using the
|
||||
[Call::AddAdaptationResource][Call::AddAdaptationResource] method.
|
||||
|
||||
## Adaptation
|
||||
|
||||
When a a *resource* signals the it is over or underused, this signal reaches the
|
||||
`ResourceAdaptationProcessor` who requests an `Adaptation` proposal from the
|
||||
[VideoStreamAdapter][VideoStreamAdapter]. This proposal is based on the
|
||||
degradation preference of the video stream. `ResourceAdaptationProcessor` will
|
||||
determine if the `Adaptation` should be applied based on the current adaptation
|
||||
status and the `Adaptation` proposal.
|
||||
|
||||
### Degradation Preference
|
||||
|
||||
There are 3 degradation preferences, described in the
|
||||
[RtpParameters][RtpParameters] header. These are
|
||||
|
||||
* `MAINTIAIN_FRAMERATE`: Adapt video resolution
|
||||
* `MAINTIAIN_RESOLUTION`: Adapt video frame-rate.
|
||||
* `BALANCED`: Adapt video frame-rate or resolution.
|
||||
|
||||
The degradation preference is set for a video track using the
|
||||
`degradation_preference` property in the [RtpParameters][RtpParameters].
|
||||
|
||||
## VideoSinkWants and video stream adaptation
|
||||
|
||||
Once an adaptation is applied it notifies the video stream. The video stream
|
||||
converts this adaptation to a [VideoSinkWants][VideoSinkWants]. These sink wants
|
||||
indicate to the video stream that some restrictions should be applied to the
|
||||
stream before it is sent to encoding. It has a few properties, but for
|
||||
adaptation the properties that might be set are:
|
||||
|
||||
* `target_pixel_count`: The desired number of pixels for each video frame. The
|
||||
actual pixel count should be close to this but does not have to be exact so
|
||||
that aspect ratio can be maintained.
|
||||
* `max_pixel_count`: The maximum number of pixels in each video frame. This
|
||||
value can not be exceeded if set.
|
||||
* `max_framerate_fps`: The maximum frame-rate for the video source. The source
|
||||
is expected to drop frames that cause this threshold to be exceeded.
|
||||
|
||||
The `VideoSinkWants` can be applied by any video source, or one may use the
|
||||
[AdaptedVideoTraceSource][adapted_video_track_source.h] which is a base class
|
||||
for sources that need video adaptation.
|
||||
|
||||
[RtpParameters]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/rtp_parameters.h?q=%22RTC_EXPORT%20RtpParameters%22
|
||||
[resource.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/adaptation/resource.h
|
||||
[Call::AddAdaptationResource]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/call/call.h?q=Call::AddAdaptationResource
|
||||
[quality_scaler.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/utility/quality_scaler.h
|
||||
[VideoEncoder::QpThresholds]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::QpThresholds
|
||||
[EncoderInfo]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::EncoderInfo
|
||||
[encode_usage_resource.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/adaptation/encode_usage_resource.h
|
||||
[VideoStreamAdapter]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/call/adaptation/video_stream_adapter.h
|
||||
[adaptation_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/call/adaptation/adaptation_constraint.h
|
||||
[bitrate_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/adaptation/bitrate_constraint.h
|
||||
[AddOrUpdateSink]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_source_interface.h?q=AddOrUpdateSink
|
||||
[VideoSinkWants]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_source_interface.h?q=%22RTC_EXPORT%20VideoSinkWants%22
|
||||
[adapted_video_track_source.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/media/base/adapted_video_track_source.h
|
||||
Reference in New Issue
Block a user