Commit Graph

172 Commits

Author SHA1 Message Date
a376518817 Migrate video/ except video/end_to_end_tests and video/adaptation to webrtc::Mutex.
Also migrates test/ partly.

Bug: webrtc:11567
Change-Id: If5b2eae65c5f297f364b6e3c67f94946a09b4a96
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178862
Commit-Queue: Markus Handell <handellm@webrtc.org>
Reviewed-by: Sebastian Jansson <srte@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31672}
2020-07-08 12:21:08 +00:00
a827a30bb7 Revert "Migrate video/ except video/end_to_end_tests and video/adaptation to webrtc::Mutex."
This reverts commit 0eba415fb40cc4e3958546a8ee53c698940df0a1.

Reason for revert: previously unknown lock recursion occurring downstream.

Original change's description:
> Migrate video/ except video/end_to_end_tests and video/adaptation to webrtc::Mutex.
> 
> Also migrates test/ partly.
> 
> Bug: webrtc:11567
> Change-Id: I4203919615c087e5faca3b2fa1d54cba9f171e07
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178813
> Reviewed-by: Erik Språng <sprang@webrtc.org>
> Commit-Queue: Markus Handell <handellm@webrtc.org>
> Cr-Commit-Position: refs/heads/master@{#31653}

TBR=sprang@webrtc.org,handellm@webrtc.org

Change-Id: I13f337e0de5b8f0eb19deb57cb5623444460ec4d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: webrtc:11567
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178842
Reviewed-by: Markus Handell <handellm@webrtc.org>
Commit-Queue: Markus Handell <handellm@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31656}
2020-07-07 20:46:48 +00:00
0eba415fb4 Migrate video/ except video/end_to_end_tests and video/adaptation to webrtc::Mutex.
Also migrates test/ partly.

Bug: webrtc:11567
Change-Id: I4203919615c087e5faca3b2fa1d54cba9f171e07
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/178813
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Markus Handell <handellm@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31653}
2020-07-07 18:01:44 +00:00
ec0af26ff8 [Adaptation] VideoStreamAdapter broadcasts adaptations
This moves this responsibility from the ResourceAdaptaitonProcessor
to the VideoStreamAdapter. A new interface for listening to adaptation
changes was added, and the ResourceAdaptationProcessor now listens on
the VideoStreamAdapter for those changes.

This means that going forward,

1. We can do adaptations outside of resource limitations, like setting
prior adaptations on a resource like initial frame dropper is designed
to.
2. Adaptations can be on a different queue than the
ResourceAdaptaitonProcessor's queue since updates are pushed through
the listener.


Bug: webrtc:11700
Change-Id: I6de0dec748dba095e702f0b9893c5afa50b51aa9
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176859
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31615}
2020-07-02 12:16:25 +00:00
64469037b7 Only allow most limited resource to trigger adapt up
A more detailed explaination is in the bug, but this changes
the way that adaptation happens when multiple resources are
limited. Only the one that is most limited can trigger an
adaptation up. If multiple resources are most limited both
need to underuse to adapt up.

Some of the changes in this patch to make it all work:

* VideoStreamEncoder unittests that did not reflect this
new behaviour have been changed.

* PeekNextRestrictions returns the adaptation counters as
well as the restrictions.

* Adaptation statstics have changed so that when adapting
up all resources are tagged as triggering the adaptation.
Additionally the statistics for the current adaptation is
now the total number of adaptations per reason, rather then
the number of adaptations due to that reason.

* PreventAdaptUpDueToActiveCounts is removed as most limited
resource is a strong implementation of that.

Bug: webrtc:11553
Change-Id: If1545a201c8e019598edf82657a1befde8b05268
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176128
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31497}
2020-06-11 09:59:42 +00:00
09eb6e249d [VP9 SVC] Round spatial layers dimensions to ensure integer scaling factors are used
Bug: webrtc:11652
Change-Id: Id3642d607f62b72a567d521d9874b8588c2ce429
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176517
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31465}
2020-06-08 19:00:30 +00:00
0f0aa9c7a8 [Adaptation] Move IsAdaptationUpAllowed/OnAdaptationApplied out of API.
IsAdaptationUpAllowed is moved from Resource to AdaptationConstraint.
OnAdaptationApplied is moved from Resource to AdaptationListener.

In a future CL, Resource will be moved to api/, but
AdaptationConstraint and AdaptationListener will stay in call/.

The processor, encode stream and manager are updated to keep track of
both resources, constraints and listeners. Fakes and tests are updated.
After this CL, the manager's inner classes that prevent adaptation
implement AdaptationConstraint instead of Resource.

Bug: webrtc:11525
Change-Id: Ie9cd5b1ba7d8e161951e131ab8f6bd9d5cf765bf
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176368
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31409}
2020-06-02 13:02:36 +00:00
16038abb90 FrameForwarder: remove lock recursions.
This change removes lock recursions and adds thread annotations.

Bug: webrtc:11567
Change-Id: I5416cfc8e482bd966eec87c3790abbebc37a84d8
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176224
Reviewed-by: Magnus Flodman <mflodman@webrtc.org>
Commit-Queue: Markus Handell <handellm@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31403}
2020-06-02 09:41:54 +00:00
5cc28b0c6a [Adaptation] Prep Resource for api/ move. Introduce VSE-Resource.
This CL is in preparation for moving Resource to the api/ folder. It
does not move it, but makes it such that the moving CL can be a pure
move.

In order to do this, we must stop depending on rtc_base/rtc::TaskQueue
in favor of api/webrtc::TaskQueueBase.

There are also other rtc_base/ dependencies that we do not want to
expose to the api/ folder, like critical sections and thread
annotations which are not publically exposed. To get around this, we
make Resource an abstract interface and move all of the base class
functionality into a new non-api/ class: VideoStreamEncoderResource.

The Resource now has Register/UnregisterAdaptationTaskQueue() methods.
By explicitly unregistering, we can ensure validity of the pointer even
if the Resource outlives the PeerConnection. While public interface
methods are only to be called on the adaptation task queue, posting to
the task queue happens off-queue, so a |lock_| is introduced to guard
it.

Bug: webrtc:11525
Change-Id: I50b3a30960cdec9032016c779b47001c01dad32f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176320
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31402}
2020-06-02 07:56:38 +00:00
de2049e85a Unflake VideoStreamEncoderTest.StatsTracksCpuAdaptationStatsWhenSwitchingSource_Balanced
R=hbos@webrtc.org

Bug: webrtc:11603
Change-Id: I6644f96cf1ec22ceb268349f0f325d73082a0148
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175917
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31350}
2020-05-26 07:17:02 +00:00
5fd4060ca2 Replace all Verify* methods with matchers for VideoStreamEncoder
This is a follow up to
https://webrtc-review.googlesource.com/c/src/+/175912
to replcae the wants comparisons as well.

R=ilnik@webrtc.org

Bug: None
Change-Id: Ic11c7fe62ceead643932f02a3287d6bed9160be3
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175915
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31346}
2020-05-25 15:04:38 +00:00
5cd7eb84c3 Replace Verify methods with matchers in VideoStreamEncoder tests
This only replaces the methods with the signatures VerifyX(wants,
value). This ensures that failing expectations give the corrent line,
where previously it gave the line in the helper methods.

Bug: None
Change-Id: I89e883ccd15b3ea5ce364d2c16b0c3ac219f139c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175912
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31345}
2020-05-25 12:50:47 +00:00
2671dac29c Fix all known VideoStreamEncoderTest flakes.
Some VideoStreamEncoderTests such as
AdaptsFramerateForLowQuality_MaintainResolutionMode had been observed
to be flaky. In a local run, this test failed 1/12000 times, but on
bots it may have failed more often.

All tests could sometimes fail due to expecting something to be the
case without waiting for that thing to happen, which was made evident
when adding SleepMs() at strategic points in the code and running the
tests locally.

With this CL, the tests should no longer be flaky after having added
waiting for adaptation to be applied or EXPECT_TRUE_WAIT for sink wants
to have the appropriate values.

Bug: webrtc:11586
Change-Id: I60e76c742d9ccc8305feb14a834a4f61a60a62a3
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175654
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31327}
2020-05-19 16:05:00 +00:00
91fdc607d8 In video/ replace mock macros with unified MOCK_METHOD macro
Bug: webrtc:11564
Change-Id: Iec9dded7a3f045e048d0546c268ae206a7d3a7c7
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175128
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Danil Chapovalov <danilchap@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31264}
2020-05-14 18:12:40 +00:00
2e2f67443e Add VideoStreamEncoder tests for DegredationPreference switching.
This test ensures that when changing degradation preference,
a resource that was previously downgraded in a different degradation
preference can not adapt up.

Bug: webrtc:11522, webrtc:11523
Change-Id: Id362530408db4c49b0d0b2516be9a11ccc7c8f37
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175012
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31250}
2020-05-14 09:21:32 +00:00
5bf60e446d VideoStreamEncoderTest: Wait for QP usage handled callback event.
(Misc cleanup associated with
https://webrtc-review.googlesource.com/c/src/+/174441.)

This test was previously assuming what when QP usage is handled it
first posts to the adaptation queue and then back with the result from
the encoder queue.

While the assumption is correct it is not an implementation detail that
the test was trying to assert, nor do we need such a test.

TriggerQualityScalerHighQpAndReturnIfQpSamplesShouldBeCleared() is
updated to wait for an event associated with QP having been handled,
which is all that the test really cares about.

Bug: webrtc:11542, webrtc:11520
Change-Id: I3286c3ab631f09c43abe0fd59f31c3997aedd9f8
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/175004
Commit-Queue: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31243}
2020-05-13 15:23:25 +00:00
381d10963a [Adaptation] Move adaptation logic to a separate task queue.
This CL unblocks future Call-Level Mitigation strategies by moving the
ResourceAdaptationProcessor to a separate task queue. This signifies a
major milestone in the new resource adaptation architecture because
with this CL the threading model is in place and moving the Processor
to the Call and increasing its responsibilities is made possible.

In this CL, we still have one Processor per VideoStreamEncoder and the
VideoStreamEncoder is responsible for the creation and the destruction
of its Processor and that Processor's task queue. But the PostTasks are
in place and the decision-making is executed on a separate queue.

This CL:
- Moves ResourceAdaptationProcessor to an adaptation task queue.
  It continues to be entirely single-threaded, but now operates on a
  separate task queue.
- Makes Resources thread-safe: Interaction with the Processor, i.e.
  OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens
  on the adaptation task queue. State updates are pushed from the
  encoder task queue with PostTasks.
- QualityScalerResource operates on both task queues; the QP usage
  callbacks are invoked asynchronously.
- The VideoStreamEncoderResourceManager operates on the encoder task
  queue with the following exceptions:
  1) Its resources are accessible on any thread (using a mutex). This
     is OK because resources are reference counted and thread safe.
     This aids adding and removing resources to the Processor on the
     adaptation task queue.
  2) |active_counts_| is moved to the adaptation task queue. This makes
     it possible for PreventAdaptUpDueToActiveCounts to run
     IsAdaptationUpAllowed() on the adaptation task queue.
     A side-effect of this is that some stats reporting now happen on
     the adaptation task queue, but that is OK because
     VideoStreamEncoderObserver is thread-safe.

The Manager is updated to take the new threading model into account:
- OnFrameDroppedDueToSize() posts to the adaptation task queue to
  invoke the Processor.
- OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation
  task queue, updates |active_counts_| synchronously but posts to the
  encoder task queue to update video source restrictions (which it
  only uses to calculate target frame rate).
- MaybePerformQualityRampupExperiment() posts to the adaptation task
  queue to maybe reset video source restrictions on the Processor.
  |quality_rampup_done_| is made std::atomic.

Bug: webrtc:11542, webrtc:11520
Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-13 08:21:23 +00:00
c55516dd55 [Adaptation] Make Resources reference counted and add more DCHECKs.
In a future CL, adaptation processing and stream encoder resource
management will happen on different task queues. When this is the case,
asynchronous tasks will be posted in both directions and some resources
will have internal states used on multiple threads.

This CL makes the Resource class reference counted in order to support
posting tasks to a different threads without risk of use-after-free
when a posted task is executed with a delay. This is preferred over
WeakPtr strategies because WeakPtrs are single-threaded and preferred
over raw pointer usage because the reference counted approach enables
more compile-time and run-time assurance. This is also "future proof";
when resources can be injected through public APIs, ownership needs to
be shared between libwebrtc and the application (e.g. Chrome).

To reduce the risk of making mistakes in the future CL, sequence
checkers and task queue DCHECKs are added as well as other DCHECKs to
make sure things have been cleaned up before destruction, e.g:
- Processor gets a sequence checker. It is entirely single-threaded.
- Processor must not have any attached listeners or resources on
  destruction.
- Resources must not have any listeners on destruction.
- The Manager, EncodeUsageResource and QualityScalerResource DCHECKs
  they are running on the encoder queue.
- TODOs are added illustrating where we want to add PostTasks in the
  future CL.

Lastly, upon VideoStreamEncoder::Stop() we delete the
ResourceAdaptationProcessor. Because the Processor is already used in
posted tasks, some if statements are added to ensure the Processor is
not used after destruction.

Bug: webrtc:11542, webrtc:11520
Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 20:35:30 +00:00
91aa73255e [Adaptation] Add OnAdaptationApplied(), remove ResourceListenerResponse.
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing

The ResourceListenerResponse was used to make the QualityScaler
not clear QP samples and instead increase its frequency of checking for
QP under certain circumstances, see enum description:
https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33

Because the QualityScaler depends on whether and how adaptation
happened it should listen to adaptation happening.

This CL moves the logic that was previously in VideoStreamAdapter closer
to the QualityScaler: QualityScalerResource::OnAdaptationApplied().

This would allow the VideoStreamAdapter to operate on a separate task
queue in the future, with no dependencies on any stream-specific
resources that might operate on other task queues.

Bug: webrtc:11172, webrtc:11521
Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31143}
2020-04-29 09:08:46 +00:00
dc4f75f7ee [Adaptation] Make ResourceUsageState nullable, remove kStable.
This CL is part of the Call-Level Adaptation Processing design doc:
https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing

The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}.
The assumption was that if a resource neither wanted to adapt up or
down it would report kStable. But with the addition of
Resource::IsAdaptationUpAllowed() (prior CL) the notion of being
"stable" was already captured outside of ResourceUsageState.
Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did
not: whether we can go up depends on the resulting resolution or frame
rate (restrictions_after). Perhaps we can go up a little, but not a lot.

This CL also adds Resource::ClearUsageState(). After applying an
adaptation, all usage states become invalidated (new measurements are
needed to know if we are still over- or underusing). This was always
the case, but prior to this CL this was not accurately reflected in the
Resource::usage_state() in-between measurements.

Bug: webrtc:11172
Change-Id: I140ff3114025b7732e530564690783e168d2509b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 10:54:53 +00:00
ce0a11d5f9 Unify AdaptationReason and AdaptReason enums.
Moves the unified AdaptationReason to the api/ folder.

Bug: webrtc:11392
Change-Id: I28782e82ef6cc3ca3b061f65b0bbdc3766df1f9c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172583
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#31084}
2020-04-16 13:33:49 +00:00
dff792591f Remove VideoStreamEncoderObserver::AdaptationReason::kNone
Replaces this with 2 methods instead, adding clarity.

ClearAdaptationStats
- Resets the adaptations statistics to 0. This is done,
when the degredation is reset, for example when the preference
is changed to/from BALANCED.

UpdateAdaptationMaskingSettings
- Updates the settings for adaptation statistics reporting.
This way we don't report quality adaptations if quality scaling
is not enabled (same for resolution/fps scaling).

The adaptation counting inside the SendStatisticsProxy is
now done in a struct that counts the totals, and then masks
out these counts based on the adaptation settings. The
MaskedAdaptationSteps uses optionals to hide the values we
shoudn't report, while the AdaptationSteps always hold the real
totals.

All tests have been updated to use the Reset/Clear method as needed.

Now that AdaptationCounters and AdaptSteps use the same structure,
AdaptationCounters was moved to api/video and replaces AdaptSteps.

The AdaptReason enum is also redundant now, and will be removed
in a follow-up CL.

R=hbos@webrtc.org

Bug: webrtc:11392
Change-Id: Iaed6488581325d341a056b5bbf76a01c19d6c282
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/171685
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#31083}
2020-04-16 13:27:50 +00:00
93be66cdaa Calculate video padding for vp9 in the same way as for vp8
Bug: webrtc:11476
Change-Id: I8d7b5aac91868e10061605cc5043226ee916cc09
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172722
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Niels Moller <nisse@webrtc.org>
Commit-Queue: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30982}
2020-04-02 13:49:10 +00:00
fb86274198 NiceMock MockFecController in VideoStreamEncoderUnittests
The MockFecController is spitting out lots of warnings, as it is
being called when we don't care about it, in normal tests. Making
it a NiceMock allows it to receive calls without expectation and
not warn.

Bug: None
Change-Id: I1ea219c4665d86917718692dc013ae3ac47222ad
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/170600
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Elad Alon <eladalon@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30820}
2020-03-18 14:26:17 +00:00
33be9dfe7a Replace AdaptCount with a single counter.
There is still a counter for the active counts for the
scaling, but these will be removed at a later date.

BUG=webrtc:11392

Change-Id: Ie9bcf3f744a0bbac601f0da61197f4bac1e9f879
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169447
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30701}
2020-03-06 08:43:47 +00:00
6038383565 [Overuse] Separate getting adaptation target from applying it.
This CL takes us one step closer to being able to evaluate alternative
possible adaptation targets (e.g. multi-stream adaptation) by exposing
the target separately from applying it.

This is a refactoring of OnResourceUnderuse() and OnResourceOveruse().

Prior to this CL, the target resolution or frame rate was calculated
inside these methods and applied if possible. This CLs makes these two
steps (calculating a usable target + applying it) separate methods.

After this CL, the target is expressed as AdaptationTarget and is
calculated and returned by GetAdaptUpTarget() and GetAdaptDownTarget().
The target is only returned if it can be applied - and CanAdaptUp() +
CanAdaptDown() are merged with these methods.

Applying the target happens at ApplyAdaptationTarget().

Bug: webrtc:11222
Change-Id: I8e488be1d1590c23848db816d49a7738562e176d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169100
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30643}
2020-02-28 09:00:31 +00:00
4f34d78c85 Report available instead of encoding bitrate to VideoEncoderSelector.
The encoding bitrate might be limited depending on the current encoder.

Bug: webrtc:11341
Change-Id: I734fce12734b1e703e7948847cdb1365c08a137b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/169123
Reviewed-by: Sami Kalliomäki <sakal@webrtc.org>
Reviewed-by: Artem Titov <titovartem@webrtc.org>
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Reviewed-by: Rasmus Brandt <brandtr@webrtc.org>
Commit-Queue: Mirta Dvornicic <mirtad@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30619}
2020-02-26 15:56:36 +00:00
aa6fbc156e Support injecting new Resources for overuse
* This replaces the video stream methods for forcing adaptation
with a mock resource that triggers overuse.
* Resources can now be injected to the Module using the AddResource
function.
* Resources now have tests for adding and removing callbacks.
* Quality/EncoderUse% resources are tracked in the Resource list of
the adaptation module.
* The adaptation module ties all resources to a reason to keep stats
working as expected.

BUG=webrtc:11377

Change-Id: I1f5902f7416dc41b4915c0072e6f0da2bb3bb2b7
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168948
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30610}
2020-02-25 16:17:42 +00:00
5ed40cfa2e Do not request encoder switch when the video is suspended.
Bug: None
Change-Id: I0ecd4db4ee53e1eb6682a2a98b684fcdf5c2e93b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168924
Commit-Queue: Mirta Dvornicic <mirtad@webrtc.org>
Reviewed-by: Rasmus Brandt <brandtr@webrtc.org>
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30585}
2020-02-21 17:54:52 +00:00
cad3e0e2fa Replace DataSize and DataRate factories with newer versions
This is search and replace change:
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataSize::Bytes<\(.*\)>()/DataSize::Bytes(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataSize::bytes/DataSize::Bytes/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataRate::BitsPerSec<\(.*\)>()/DataRate::BitsPerSec(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataRate::BytesPerSec<\(.*\)>()/DataRate::BytesPerSec(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataRate::KilobitsPerSec<\(.*\)>()/DataRate::KilobitsPerSec(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataRate::bps/DataRate::BitsPerSec/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/DataRate::kbps/DataRate::KilobitsPerSec/g"
git cl format

Bug: webrtc:9709
Change-Id: I65aaca69474ba038c1fe2dd8dc30d3f8e7b94c29
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168647
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Commit-Queue: Danil Chapovalov <danilchap@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30545}
2020-02-18 16:09:50 +00:00
6799d732d5 Delete DefaultVideoBitrateAllocator.
It was removed from tests in https://webrtc-review.googlesource.com/c/src/+/123540.

If simulcast is not used, SimulcastRateAllocator returns the
same allocation as DefaultVideoBitrateAllocator.

Bug: webrtc:10164
Change-Id: I3d3e1aefe2fcc2bf853cd63c75e008b86eff9241
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168496
Reviewed-by: Erik Språng <sprang@webrtc.org>
Reviewed-by: Niels Moller <nisse@webrtc.org>
Commit-Queue: Mirta Dvornicic <mirtad@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30509}
2020-02-12 21:29:09 +00:00
0c626afcf3 Use newer version of TimeDelta and TimeStamp factories in webrtc
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/TimeDelta::Micros<\(.*\)>()/TimeDelta::Micros(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/TimeDelta::Millis<\(.*\)>()/TimeDelta::Millis(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/TimeDelta::Seconds<\(.*\)>()/TimeDelta::Seconds(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/TimeDelta::us/TimeDelta::Micros/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/TimeDelta::ms/TimeDelta::Millis/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/TimeDelta::seconds/TimeDelta::Seconds/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/Timestamp::Micros<\(.*\)>()/Timestamp::Micros(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/Timestamp::Millis<\(.*\)>()/Timestamp::Millis(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/Timestamp::Seconds<\(.*\)>()/Timestamp::Seconds(\1)/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/Timestamp::us/Timestamp::Micros/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/Timestamp::ms/Timestamp::Millis/g"
find . -type f \( -name "*.h" -o -name "*.cc" \) | xargs sed -i -e "s/Timestamp::seconds/Timestamp::Seconds/g"
git cl format

Bug: None
Change-Id: I87469d2e4a38369654da839ab7c838215a7911e7
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168402
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Commit-Queue: Danil Chapovalov <danilchap@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30491}
2020-02-10 12:21:17 +00:00
9b05803e19 Implement injectable EncoderSelectorInterface and wire it up in the VideoStreamEncoder.
The EncoderSelectorInterface is meant to replace the "WebRTC-NetworkCondition-EncoderSwitch" field trial, so the field trial will be ignored if an EncoderSelectorInterface object has been injected.

Bug: webrtc:11341
Change-Id: I5371fac9c9ad8e38223a81dd1e7bfefb2bb458cb
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168193
Commit-Queue: Philip Eliasson <philipel@webrtc.org>
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Reviewed-by: Rasmus Brandt <brandtr@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30490}
2020-02-10 12:12:47 +00:00
9b881abea9 Enable congestion window pushback to reduce bitrate by only drop video frames.
With current congestion window pushback, when congestion window is filling up, it will reduce bitrate directly and encoder may reduce encode quality, resolution, or framerate to adapt to the allocated bitrate, the behavior is depending on the degradation preference.
This change enable congestion window to only drop frames to reduce bitrate (when needed) instead of reduce general bitrate allocation.

Bug: webrtc:11334
Change-Id: I9cf5c20a0858c4d07d006942abe72aa5e1f7cb38
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168059
Commit-Queue: Ying Wang <yinwa@webrtc.org>
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30483}
2020-02-07 14:14:47 +00:00
7875c99e82 [Overuse] Add EncodeUsageResource and QualityScalerResource.
This refactors the usage of OveruseFrameDetector in
OveruseFrameDetectorResourceAdaptationModule into an inner class of the
module, making the interaction between the detector and the module the
responsibility of this helper class instead.

Similarly, QualityScaler usage is moved into QualityScalerResource.

This takes us one step closer to separate the act of detecting
overuse/underuse of a resource and the logic of what to do when
overuse/underuse happens.

Follow-up CLs should build on this in order to materialize the concept
of having resources, streams and a central decision-maker deciding how
to reconfigure the streams based on resource usage state.

Bug: webrtc:11222
Change-Id: I99a08a42218a871db8f477f31447a6379433ad05
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168057
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30468}
2020-02-06 11:29:02 +00:00
02d51f9fdc Remove unused field trial WebRTC-InitialFramedrop
Bug: webrtc:9176, webrtc:6086
Change-Id: Ie02800963f790f07b4c60ff01a04ecd6b5e1113d
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/167181
Reviewed-by: Kári Helgason <kthelgason@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Commit-Queue: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30361}
2020-01-23 13:42:03 +00:00
cd02ebaea0 Use intersection of app and encoder bitrate limits.
Before this change, if both app and encoder provided bitrate limits,
WebRTC ignored the limits provided by encoder. Now intersection of
these sets is used.

Also changed DCHECKs in GetEncoderBitrateLimits to allow zero values
of min_bitrate_bps and min_start_bitrate_bps.

Bug: none
Change-Id: Ib8be965ea43f51013b0a0f82fd4256a372432dda
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/166600
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Rasmus Brandt <brandtr@webrtc.org>
Commit-Queue: Sergey Silkin <ssilkin@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30338}
2020-01-21 15:16:29 +00:00
07b17df771 Move DegradationPreference logic to the encoder queue.
This moves SetHasInputVideoAndDegradationPreference() to the encoder
queue. OveruseFrameDetectorResourceAdaptationModule is now entirely
single-threaded, including its inner class VideoSourceRestrictor.

VideoStreamEncoder now protects the module with RTC_GUARDED_BY. This
ensures it is safely used, even without a SequenceChecker inside of the
module. The module's |encoder_queue_| is removed.

The one task queue reference that is needed - passing down the current
task queue to StartCheckForOveruse() - is replaced by a TaskQueueBase*
(instead of rtc::TaskQueue*), enabling obtaining the current queue with
TaskQueueBase::Current(). (There is no rtc::TaskQueue::Current().)

Furthermore, the only uses of VideoSourceSinkController that isn't on
the encoder queue are documented, with a TODO saying if these are moved
the VideoSourceSinkController could also be made single-threaded.
However since this requires introducing a delay to
VideoStreamEncoder::SetSource() and VideoStreamEncoder::Stop(),
arguably a more risky change, if this is to be attempted that should be
in a separate CL.

Bug: webrtc:11222
Change-Id: I448ca5125708d5f66b95b0b180d6d24cc356dfa9
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/165783
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30263}
2020-01-15 11:58:04 +00:00
b08882b625 Refactor out VideoStreamEncoder's overuse logic to separate module.
This CL puts the VideoStreamEncoder's current adaptation logic inside
the new class OveruseFrameDetectorResourceAdaptationModule. The
intention is not to change any behavior, only to move code.

Future CLs should step by step decrease the coupling between
OveruseFrameDetectorResourceAdaptationModule, VideoStreamEncoder and
the VideoStreamEncoder's QualityScaler by introducing more abstract
interfaces. This is not done in this CL because it is large enough as
it is, but the long term goal is to make it possible to replace the
existing overuse module with a different implementation.

This CL relies on existing tests exercising the VideoStreamEncoder, but
part of making overuse logic modular should include testing each module
separately as well as continued integration testing of the
VideoStreamEncoder.

Bug: webrtc:11222
Change-Id: I316a174adfd00d60cdd224a23a5f616efd235d13
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161953
Commit-Queue: Henrik Boström <hbos@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Reviewed-by: Evan Shrubsole <eshr@google.com>
Cr-Commit-Position: refs/heads/master@{#30163}
2020-01-07 09:55:54 +00:00
5cad55b240 Signal requested resolution alignment requirements from sinks to sources.
Bug: webrtc:11218
Change-Id: I593b0515ea389bece472234a3c4082ccc5321ea5
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/162400
Commit-Queue: Rasmus Brandt <brandtr@webrtc.org>
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30113}
2019-12-19 10:39:04 +00:00
ef3998ffd1 Add directive to make webrtc metrics optional.
Bug: webrtc:11144
Change-Id: I4e75e6aec033784685de3670e880bb9f2b6ee8d2
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161043
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org>
Commit-Queue: Ying Wang <yinwa@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30040}
2019-12-09 13:55:50 +00:00
33f9d2b383 Migrate WebRTC on FrameGeneratorInterface and remove FrameGenerator class
Bug: webrtc:10138
Change-Id: If85290581a72f81cf60181de7a7134cc9db7716e
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161327
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Commit-Queue: Artem Titov <titovartem@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30033}
2019-12-07 00:54:26 +00:00
648b9d77c7 Implement automatic animation detection in VideoStreamEncoder
If WebRTC-AutomaticAnimationDetectionScreenshare experiment is enabled,
content type is screenshare and degradation preference is BALANCED,
then input resolution is restricted if update_rect of the incoming frames
is the same for considerable amount of time and is big enough.

This entails treating BALANCED degradation preference for screenshare as
MAINTAIN_RESOLUTION in adaptation logic.

Bug: webrtc:11058
Change-Id: I903dddf53fcbd7c8eac6c5b1447225b15fd8fe5f
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161097
Reviewed-by: Erik Språng <sprang@webrtc.org>
Commit-Queue: Ilya Nikolaevskiy <ilnik@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#30002}
2019-12-04 11:24:31 +00:00
5256d8bc4b Refactor FrameGenerator to return VideoFrameBuffer with VideoFrame::UpdateRect
Bug: webrtc:10138
Change-Id: I22079e2630bb1f3bb27472795fe923f9143b3401
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161010
Commit-Queue: Artem Titov <titovartem@webrtc.org>
Reviewed-by: Karl Wiberg <kwiberg@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29973}
2019-12-02 17:11:37 +00:00
287e464705 Change VideoAdapter::OnResolutionFramerateRequest to VideoAdapter::OnSinkWants
This change makes it easier to propagate more information from the sink
 to the video adapter, for example alignment requirements.

Bug: None
Change-Id: I536248d59f871c103a18a48615b6c5e61f61697b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/159281
Reviewed-by: Erik Språng <sprang@webrtc.org>
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Commit-Queue: Rasmus Brandt <brandtr@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29827}
2019-11-19 10:14:31 +00:00
e644a03195 Add field trial for rampup in quality based on available bandwidth.
Bug: none
Change-Id: I32e1ea6fb2f2e20fc631e09b02c8f3a11b6c9fac
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/158888
Reviewed-by: Niels Moller <nisse@webrtc.org>
Reviewed-by: Sergey Silkin <ssilkin@webrtc.org>
Commit-Queue: Åsa Persson <asapersson@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29751}
2019-11-11 10:13:28 +00:00
2b9317ad76 Stop checking VP8BaseHeavyTl3RateAllocation field trial on every frame.
- Centralize field trial string reading to RateControlSettings
- Cache RateControlSettings at all production code use sites

Bug: None
Change-Id: I0dbce9cc97fea0bc780982e7ef270b417a8c15bf
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/158664
Commit-Queue: Rasmus Brandt <brandtr@webrtc.org>
Reviewed-by: Elad Alon <eladalon@webrtc.org>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29680}
2019-11-04 13:50:59 +00:00
41c650bea2 Use bitrate limits provided by encoder.
- Use minimum start bitrate to drop frame and adapt resolution in the
beginning of call.

- Use minimum bitrate to decide whether or not resolution should be
increased based on quality in MAINTAIN_FRAMERATE and BALANCED modes.
In BALANCED mode bitrate limits provided by the corresponding field
trial are prioritized over the limits provided by encoder.

Bug: webrtc:10853
Change-Id: I8257eb64565bcafa6ae9887a1af18e90f8400cac
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/156302
Commit-Queue: Sergey Silkin <ssilkin@webrtc.org>
Reviewed-by: Åsa Persson <asapersson@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29461}
2019-10-14 12:57:24 +00:00
7c079f650d Reland "Fix minor regression caused by a8336d3"
This is a reland of 809198edfff416fce8d75b574a43afab5e67b1cd

A fix was made in https://webrtc-review.googlesource.com/c/src/+/154343
which fixed the regression issues caused by the original patch.

Original change's description:
> Fix minor regression caused by a8336d3
>
> VideoEncoder::SetRates was being called unnessesarily when the fields
> appended to RateControlParameters were changed. Since SetRates only
> cares about RateControlParameters, it should have only been called if
> the RateControlParameters themselves were actually changed.
>
> Bug: webrtc:10126
> Change-Id: Ic47d67e642a3043307fec950e5fba970d9f95167
> Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/152829
> Reviewed-by: Erik Språng <sprang@webrtc.org>
> Commit-Queue: Evan Shrubsole <eshr@google.com>
> Cr-Commit-Position: refs/heads/master@{#29208}

Bug: webrtc:10126
Change-Id: Iecc3ab6a5cd1193a1fa8e824dcf4f0b8165f9bf8
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/154359
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Erik Språng <sprang@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29356}
2019-10-01 11:49:38 +00:00
45b176fc22 Downgrade fps in same step as resolution in initial drop due to size.
Bug: none
Change-Id: If0943ee291a029fa81035c72607873995ba8ab8b
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/154742
Reviewed-by: Sergey Silkin <ssilkin@webrtc.org>
Commit-Queue: Åsa Persson <asapersson@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#29342}
2019-09-30 12:28:26 +00:00