This will allow us to optimize the internal buffers of
webrtc::VideoFrame for the resolution(s) that we actually want to
encode.
Bug: webrtc:12469, chromium:1157072
Change-Id: If378b52b5e35aa9a9800c1f7dfe189437ce43253
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/208540
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Harald Alvestrand <hta@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#33342}
Use bandwidth allocation instead of encoder target bitrate in DropDueToSize when incoming resolution increases to avoid downgrades due to target bitrate being limited by the max bitrate at low resolutions.
Bug: none
Change-Id: Ic41b31c1a86911d4e97b61b0cbc41ce0da739bd4
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/205622
Commit-Queue: Åsa Persson <asapersson@webrtc.org>
Reviewed-by: Sergey Silkin <ssilkin@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#33168}
This could potentially lead to unnecessary restarts since it is also
started after the encoder is created. However, it is needed since the
hardware acceleration support can change even though the encoder has
not been recreated.
Bug: b/145730598
Change-Id: Iad1330e7c7bdf769a68c4ecf7abb6abbf3a4fe71
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/203140
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Jakob Ivarsson <jakobi@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#33060}
It turned out that the negotiated rtp header extensions are not fully known in WebRtcVideoChannel::AddSendStream.
The cl also remove the unnecessary factory for creating VideoStreamEncoder.
Bug: webrtc:12000
Change-Id: If994c8deb69f3ce4212896d3ad757dac94c6e09f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/198840
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Per Kjellander <perkj@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32916}
VideoCodecInitializer::VideoEncoderConfigToVideoCodec is modified to always set correct frame rate, width and height on spatial layer 0 so the rest of the code does not need to differentiate between scalable/none scalable codecs.
Bug: webrtc:12000
Change-Id: I5a068b98ca2038621205f55e4024f949ab51587a
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/198540
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Per Kjellander <perkj@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32890}
This test that a new allocation is reported if the input resolution
changes.
Bug: webrtc:12000
Change-Id: Iaf8be1af62bbc8a2ca19b58f0587ceacfcfa5991
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/197807
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Per Kjellander <perkj@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32837}
External reasons here are simulcast configuration and
source resolution change.
Initial frame dropper should be enabled in these cases because the
client can request way too big resolution for available bitrate and
usual quality scaling would take too long.
Bug: none
Change-Id: I02fbbd3c15b53b39672c083c2a1f9a780256c507
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/195004
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Commit-Queue: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32707}
This changes the default behavior to use pacing factor of 1.1x instead
of 2.5x, it also sets libvpx rate controler as trusted, turns on the
encoder pushback mechanism and sets spatial hysteresis to 1.2.
The unused "dynamic rate" settings in libvpx is removed.
The new settings matches what has been used in chromium since 2019.
If needed, the legacy behavior can be enabled using the field trial
WebRTC-VideoRateControl.
Bug: webrtc:10155
Change-Id: I8186b491aa5bef61e8f568e96c980ca68f0c208f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/186661
Commit-Queue: Erik Språng <sprang@webrtc.org>
Reviewed-by: Per Kjellander <perkj@webrtc.org>
Reviewed-by: Christoffer Rodbro <crodbro@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32477}
Adds a field to EncoderInfo called preferred_pixel_formats which a
software encoder populates with the pixel formats it supports. When a
kNative frame is received for encoding, the VideoStreamEncoder will
first try to get a frame that is accessible by the software encoder in
that pixel format from the kNative frame. If this fails it will fallback
to converting the frame using ToI420.
This minimizes the number of conversions made in the case that the
encoder supports the pixel format of the native buffer or where
conversion can be accelerated. For example, in Chromium, the capturer can
emit an NV12 frame, which can be consumed by libvpx which supports NV12.
Testing: Tested in Chrome with media::VideoFrame adapters.
Bug: webrtc:11977
Change-Id: I9becc4100136b0c0128f4fa06dedf9ee4dc62f37
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/187121
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Markus Handell <handellm@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#32353}
VideoBitrateAllocation is instead reported through the EncoderSink.
Enable VideoBitrateAllocation reporting from WebRtcVideoChannel::AddSendStream in preparation for
using the extension RtpVideoLayersAllocationExtension instead of RTCP XR.
Bug: webrtc:12000
Change-Id: I5ea8e4f237a1c4e84a89cbfd97ac4353d4c2984f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/186940
Commit-Queue: Per Kjellander <perkj@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32347}
Original description
Move reporting of target bitrate to just after the encoder has been
updated. Originall submitted as refs/heads/master@{#32275}
Patch 1 contains the original cl
,patch 2 the fix to send rtcp even if BWE does not change.
Bug: webrtc:12000
Change-Id: I16766e08229fe1f6f65f449e0e074bed03338693
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/186948
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Per Kjellander <perkj@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32340}
This needs to be done still for kNative frames, but all other frame types
can be passed in.
I have checked all VideoEncoder implementations in Chromium and confirmed they either convert the frame to their preferred pixel format, or just
forward the frame to a delegate encoder.
Tested:
- video_loopback with NV12 generated frames for VP9, the only
codec supporting NV12, as well as VP8 which only accepts I420 frames.
- internal_tests tryrun
Bug: webrtc:11976,webrtc:11635
Change-Id: If39a815fb0c5636fceb1040c8946c3db2fb350a1
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/185803
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Niels Moller <nisse@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32306}
Currently, key frames are scheduled even when the encoder is not reset
during reconfigeration. This means whenever new parameters like max
bitrate or min bitrate are updated through SetRtpParameters(), the
triggered encoder reconfigeration will always schedule key frames even
they are not necessary. Since parameters' changes like bitrate doesn't
require encoder instance reset.
This causes flood of key frames in our app since we do regularly max
bitrate update according to server control message.
Bug: None
Change-Id: I15d953b24c30e6026c0e97b30f44495d845f293f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/185380
Commit-Queue: Rasmus Brandt <brandtr@webrtc.org>
Reviewed-by: Rasmus Brandt <brandtr@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32245}
RequestEncoderFallback, RequestEncoderSwitch and
SetVideoCodecSwitchingEnabledRequest are now all called on the
worker thread. Before, the work already happened on that thread but
WebRtcVideoChannel adapted internally when needed.
With this CL, there are thread checks to make sure that these calls are
always made the same way, we don't need the async invoker and there
are fewer calls out from the encoder thread in VideoStreamEncoder
(reducing the chance of unintentional blocking).
Bug: webrtc:11908
Change-Id: If8738bc2a708a0fefc6fe850b32655f049f30bdc
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/184603
Commit-Queue: Tommi <tommi@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32151}
This turned out to be a bit complicated, mostly
related to the tests, but here's what's changed:
* No AsyncInvoker (and avoid ClearInternal) in
WebRtcVideoSendStream (WVSS)
* The reason it was there is due to a "design leak" from
VideoSourceSinkController/VideoStreamEncoder where the former uses
locks in all methods and is unaware of a threading model. That design
affected downstream objects, pushed the need for an async hop into
WVSS and added a lock.
A suggestion was made to address this in a follow-up change, here:
https://webrtc-review.googlesource.com/c/src/+/165684
* All methods in VideoSourceSinkController are now called on a known
and checked sequence and this CL removes the lock. This also makes
checking state consistent (i.e. calling a getter twice in a row on the
same sequence, will always return the same value, avoiding race with
other threads).
* Handling of reporting state changes from the encoder queue to the
VSSC, is done by VideoStreamEncoder.
* VideoSendStreamImpl is still instantiated on the incorrect thread [1]
but has two initialization steps [2]. The second one already runs on
the right thread. Addressing that TODO [1] is something we should do
but it has side effects to consider. For the purposes of this CL
the steps relating to the encoder (setting the sink pointer) have
been moved to [2].
[1] https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/video_send_stream.cc;l=94
[2] https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/video_send_stream.cc;drc=f4a9991cce74a37d006438ec0e366313ed33162e;l=115
Bug: webrtc:11222, webrtc:11908
Change-Id: Ie46d46e3a52bbe225951b4bd580ecb8cc9cad873
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/184508
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Commit-Queue: Tommi <tommi@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#32150}
This is a reland of ba8abbb630cdd9d05e22c830d0845e920762850d
This can be relanded as the queuing issues that were causing a
crash in the WebRTC roll in Chromium have been resolved. I have
added the Chromium failing targets to the CQ for this commit and
they have succeeded.
Original change's description:
> [Adaptation] Remove QualityScalerResource when disabled.
>
> Bug: webrtc:11843
> Change-Id: I2d3e40356c266f189db0242f3c7590e6d83e4456
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/181369
> Commit-Queue: Evan Shrubsole <eshr@google.com>
> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
> Cr-Commit-Position: refs/heads/master@{#31924}
Bug: webrtc:11843
Change-Id: I228331293060ef996f1dd7f8e18d52b0818f526b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/182080
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31996}
Resource adaptation needs refactoring for async adaptations. For now
the resource adaptation processor can work on the encoder thread, until
it is refactored to support async adaptation.
Bug: webrtc:11867
Change-Id: I9c46da356db19c0fd52748c999ccb216f2ca923b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/182040
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31991}
It was not used by any class and all future uses can use the
VideoSourceRestrictionsListener.
Bug: webrtc:11834
Change-Id: I5c71b93cc503f458dce0ccdd78b91b5a1debc56d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/181062
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31896}
This way can double adapt right away instead of relying
on the qp scaler checking soon into the future.
Bug: webrtc:11830
Change-Id: I8e878168303cf6a4c3edcf3997dd8ac2413a4479
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/181060
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31895}
RTPFragmentationHeader is already ignored by H264 packetizer
and thus doesn't need to be provided and calculated.
Bug: webrtc:6471
Change-Id: I45bc22827f0dc811457e3ebe477a16293501c2fb
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/179843
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Marco Paniconi <marpan@webrtc.org>
Commit-Queue: Danil Chapovalov <danilchap@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31791}
This is a step needed for multi-stream and new mitigations. It also
cleans up needing to signal adaptation changes in mutiple places
from ResourceAdaptationProcessor.
R=hbos@webrtc.org
Bug: webrtc:11754
Change-Id: Ib185dc9f66fbb4a087eb9e970c68c3f47eafb17f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178874
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31684}
This moves this responsibility from the ResourceAdaptaitonProcessor
to the VideoStreamAdapter. A new interface for listening to adaptation
changes was added, and the ResourceAdaptationProcessor now listens on
the VideoStreamAdapter for those changes.
This means that going forward,
1. We can do adaptations outside of resource limitations, like setting
prior adaptations on a resource like initial frame dropper is designed
to.
2. Adaptations can be on a different queue than the
ResourceAdaptaitonProcessor's queue since updates are pushed through
the listener.
Bug: webrtc:11700
Change-Id: I6de0dec748dba095e702f0b9893c5afa50b51aa9
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176859
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31615}
A more detailed explaination is in the bug, but this changes
the way that adaptation happens when multiple resources are
limited. Only the one that is most limited can trigger an
adaptation up. If multiple resources are most limited both
need to underuse to adapt up.
Some of the changes in this patch to make it all work:
* VideoStreamEncoder unittests that did not reflect this
new behaviour have been changed.
* PeekNextRestrictions returns the adaptation counters as
well as the restrictions.
* Adaptation statstics have changed so that when adapting
up all resources are tagged as triggering the adaptation.
Additionally the statistics for the current adaptation is
now the total number of adaptations per reason, rather then
the number of adaptations due to that reason.
* PreventAdaptUpDueToActiveCounts is removed as most limited
resource is a strong implementation of that.
Bug: webrtc:11553
Change-Id: If1545a201c8e019598edf82657a1befde8b05268
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176128
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31497}
IsAdaptationUpAllowed is moved from Resource to AdaptationConstraint.
OnAdaptationApplied is moved from Resource to AdaptationListener.
In a future CL, Resource will be moved to api/, but
AdaptationConstraint and AdaptationListener will stay in call/.
The processor, encode stream and manager are updated to keep track of
both resources, constraints and listeners. Fakes and tests are updated.
After this CL, the manager's inner classes that prevent adaptation
implement AdaptationConstraint instead of Resource.
Bug: webrtc:11525
Change-Id: Ie9cd5b1ba7d8e161951e131ab8f6bd9d5cf765bf
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176368
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31409}
This CL is in preparation for moving Resource to the api/ folder. It
does not move it, but makes it such that the moving CL can be a pure
move.
In order to do this, we must stop depending on rtc_base/rtc::TaskQueue
in favor of api/webrtc::TaskQueueBase.
There are also other rtc_base/ dependencies that we do not want to
expose to the api/ folder, like critical sections and thread
annotations which are not publically exposed. To get around this, we
make Resource an abstract interface and move all of the base class
functionality into a new non-api/ class: VideoStreamEncoderResource.
The Resource now has Register/UnregisterAdaptationTaskQueue() methods.
By explicitly unregistering, we can ensure validity of the pointer even
if the Resource outlives the PeerConnection. While public interface
methods are only to be called on the adaptation task queue, posting to
the task queue happens off-queue, so a |lock_| is introduced to guard
it.
Bug: webrtc:11525
Change-Id: I50b3a30960cdec9032016c779b47001c01dad32f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176320
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31402}
