Unified BeginFrame scheduling for Chrome

Posted 2021-01-25 huangguanyuan

Unified BeginFrame scheduling for Chrome

 

http://goo.gl/D1Qxrr

Status: http://crbug.com/401331 and http://crbug.com/416760

Original Author: [email protected]

Second Revision Authors: [email protected] [email protected] ([email protected])

Reviewers: [email protected], [email protected], [email protected]

• Last updated: March 1, 2015

 

Motivation

We‘ve learned from previous study, Improved vsync scheduling for Chrome on Android, about how explicit BeginFrame notification message from browser to renderer can improve frame rate with input handling. We need to adopt this mechanism to other platforms such as aura or mac for improving performance and unified implementation on all platforms.

 

 

Goal

This document will propose new BeginFrame scheduling to achieve below goals.

• Separate BeginFrame ipc handling from OutputSurface to new handler

○     https://codereview.chromium.org/619843002/ (done by @simonhong)

 

• Make BeginFrame start by browser compositor’s scheduler

○     add forward_begin_frames_to_children in SchedulerSettings

○     https://codereview.chromium.org/723713003/ (done by @simonhong)

○     Refactoring is in progress (ProxyBeginFrameSource)

(https://codereview.chromium.org/845393002/ by @simonhong)

 

• Send BeginFrame message from browser to renderer on all platform

○     set true to forward_begin_frames_to_children in SchedulerSettings on browser side cc scheduler

○     Aura based implementation is WIP (https://codereview.chromium.org/775143003/ by @simonhong)

 

•  Remove CompositorVSyncManager

○     We don’t need vsync synchronization with renderer process on unified BeginFrame scheduling

○     (https://codereview.chromium.org/775143003/ by @simonhong)

 

• Feeds system’s VSync to browser compositor‘s scheduler

○     Routes Windowandroid::OnVSync to the scheduler in android

○     Routes Mac’s DisplayLink to the scheduler in MacOSX

○     Aura already sends vsync to browser compositor’s scheduler

 

• Use external BeginFrameSource for Mac & Android for feeding vsync params

○     Set true to use_external_begin_frame_source in SchedulerSettings and implements new external BeginFrameSources

 

• Remove ViewMsg_UpdateVSyncParameters

○     vsync parameter is no longer used in renderer process

 

• Limited use of self-ticking on renderer-side

○     In normal situation, scheduler of renderer process on all platforms will rely on external BeginFrameSource

○     Use un-throttled mode when renderer is in background mode

○     Use un-throttled mode when renderer’s frame-rate should be adjusted

 

• Consider re-usability with DisplaySurface which will become a parent scheduler

 

External dependencies

• STP should be act as a cc::Scheduler’s client

○     https://codereview.chromium.org/638653003/  (Done by @weiliangc)

• Make Android CompositorImpl use scheduled SingleThreadProxy

○     https://codereview.chromium.org/344743002 (@simonhong is working on this)

 

Benefits from unified BeginFrame (@brianderson)

• Coordination with input events. Android coordinates the input for us today, but for other platforms we need to coordinate the input events manually with the unified BeginFrame.
• After unified BeginFrame, the Browser will know about all active renderers before the renderer swap, which will allow us to:

○     Implement latency feedbackWait for multiple active renderers to swap in the browser, rather than only picking up the first renderer to swap

○      betwr aeen renderend browser (i.e. SwapNack). If an active renderer misses the browser’s deadline, the browser can tell the renderer that it was too slow immediately

• We can extend the info inside the BeginFrame once it is on all platforms. For example, it might make sense to include SwapAck/Nack, input events, and future sync points as part of the BeginFrame to reduce the number of messages and thread hops we have per frame.
• Unifying the code base across platforms

 

 

Terms

Sometimes, VSync and BeginFrame are used interchangeably. Below is my understanding about both terms in chromium code base.

 

• VSync: notification from system or OS level when system or OS want to output new frame
• BeginFrame: message that requests to compositor scheduler to trigger new frame.

 

You can easily think with above terms like this - “When browser process gets VSync notification from system, it send BeginFrame message to renderers to trigger their new frame if they needs”

 

 

Overall diagram (originated from tim’s)

 

https://docs.google.com/a/chromium.org/drawings/d/1WEj-6A-8FmJNIMbd9hvkvxAuOOTwQvkSjbKR79YCt-c/edit

 

 

 

 

Proposed implementations

 

Below diagram is more specific version of above diagram. Browser side scheduler ‘s BeginFrame message will be delivered to LTH and then ui::Compositor.

ui::Compositor will have BeginFrame observer list and it will toss BeginFrame to all observers which are DelegatedFrameHost. Whenever renderer side scheduler needs BeginFrame and doesn’t needed, its corresponding DelegatedFramHost will be added as an observer or removed.

 

 

 

 

 

Synchronising vsync params

We need to synchronize system’s vsync params with browser side scheduler when platform doesn’t support frame start callback. In this section, I’ll talk about how to modify current vsync update mechanism. Between browser process and renderer process vsync synchronization will be removed.

 

Current implementation

• Aura: CompositorVSyncManager is used to spread new vsync params to browser and renderers.
• Mac: RWHVMac fetches vsync params from DisplayLinkMac and sends it to renderer.
• Android: vsync params is not used.

 

Proposed implementation

With our next generation BeginFrame scheduling, we don’t need to send vsync params to renderer anymore because browser will request BeginFrame explicitly to renderer.

 

On Aura, only browser scheduler needs to be updated with vsync params.

 

 

Browser-side cc::Scheduler will use SyntheticBeginFrameSource to get next BeginFrame because Windows and Linux doesn’t provide frame start callback. Also, we can remove CompositorVSyncManager which is owned by ui::Compositor because vsync synchronization doesn’t need anymore.[1] [2]  Renderer process will only rely on BeginFrame message from browser process.

 

On MacOSX, we don’t need to update vsync params with browser’s Scheduler anymore. Instead, we can use CVDisplayLinkOutputCallback which is called whenever the display link wants to output a frame by registering vsync callback via CVDisplayLink. CVDisplayLink will invoke registered callback when it needed a new frame. This callback will feed next BeginFrame to browser’s scheduler with externalBFS.

 

On Android[3] [4] [5] , there is no change with regards to vsync update.

We will use this vsync from android platform and deliver it to compositor scheduler by using externalBFS like mac when android starts to use SingleThreadProxy + Scheduler.

 

 

VSync-Aligned input handling

IIUC, When SendBeginFrame() is called at RWHV, BufferedInput[6] [7]  ([email protected] wrote VSync-Aligned Bufferred Input) should be delivered to renderer first before send BeginFrame to renderer. So we can have more chance to handle these inputs in the next frame.

 

 

Implementation plans

1. Enable unified BeginFrame on aura (https://codereview.chromium.org/775143003/)
2. Implements new DisplayLinkMac to receive vsync callback and deliver vsync params to browser side scheduler (which will be act like externalBFS)
3. Unify code base with android platform

 

 

Question?

1. Does we need to modify current android implementation[8] [9] ? (solved - yes we need!)

2. Does we need to consider this feature in single thread mode[10] [11] , too?

 

 

 

BeginFrame API

See cc/scheduler/begin_frame_source.h file.

BeginFrameObserver

BeginFrameObserver {

  OnBeginFrame(BeginFrameArgs)

  BeginFrameArgs LastUsedBeginFrameArgs()

}

 

OnBeginFrame

Core functionality of the API, tells you that input has been processed and that now is a good time to begin rendering.

 

Notes: BeginFrame calls *normally* occur just after a vsync interrupt when input processing has been finished and provide information about the time values of the vsync times. However, these values can be heavily modified or even plain made up (when no vsync signal is available or vsync throttling is turned off).

LastUsedBeginFrameArgs

LastUsedBeginFrameArgs exists because any non-trivial observer needs to store BeginFrameArgs for usage until it receives the next BeginFrameArgs. The observer is also the authoritative source on if the BeginFrameArgs is going to be used in some way.

 

This functionality enables;

• Dropped frames can be detected by checking if the value is updated.
• Throttling, multiplexing and any other frame sources which needs information about the last information sent;

○     Don’t also need store a copy of the args.

○     Correctly take into account dropped frames.

 

BeginFrameSource

BeginFrameSource {

  bool NeedsBeginFrames()

  SetNeedsBeginFrames(bool)

  DidFinishFrame(size_t remaining_frames)

 

  AddObserver(BeginFrameObserver);

  RemoveObserver(BeginFrameObserver);

}

 

NeedsBeginFrames / SetNeedsBeginFrames

Provides a level triggered system for enabling / disabling the generation of OnBeginFrame messages. The signal is level triggered to reduce IPC overhead when the source is not local to a process.

DidFinishFrame (optional)

Provides async back pressure information into the source about how many frames are still in the process of being processed. Is useful if a source wants to keep only a single frame pending or other similar options. These messages might be delayed by IPC overhead, so should be used as informational only.

AddObserver / RemoveObserver

Normal observer interface, probably implemented with base::ObserverList.

 

ProxyBeginFrameSource

---

CompositorVSyncManager is essentially there because it receives information from 2 threads:

1- one "authoritative" frame rate from XRandR, on the UI thread

2- the latest rate + alignment from the GPU, on the compositor thread.

Yes, CompositorVSyncManager should be existed until it is changed to run on SingleThreadProxy.

Before that, ui::Compositor should maintain two options.

If it is running on ThreadProxy(CrOS), CompositorVSyncManager is used as now.

If it is running on SingleThreadProxy(aura and mac with delegated rendering), it will use new BeginFrame scheduling.

[email protected] who may have some thoughts here as it relates to recent Android browser compositor changes.

I think it‘s either UpdateVSyncParameters() *or* BeginFrame(). For the render compositors, we use the latter and don‘t need to tell those compositors about vsync params.

For the browser compositor, we currently do our own scheduling, but there are patches in the works for switching this to SingleThreadProxy+cc-scheduler+BeginFrame-scheduling.

[email protected], Will android use cc-scheduler as a BeginFrame source when the enne‘s patch (makes SingleThreadProxy as a SchedulerClient) is landed? If so, we should also update vsync params with cc-scheduler.

[email protected] might be able to chime in here.  A slight difference between Android and Aura is that Aura‘s gesture detector is more or less a singleton, whereas on Android (for content-targeted input) we have one detector per view.

 

As touch input buffering occurs  *before* gesture detection in the pipeline, there may be some subtle differences in how BeginFrame serves as an input flush signal.

Aura has one gesture detector per aura::Window. (GestureRecognizerAura is a singleton, but it stores a mapping from aura::Windows to GestureProviderAura‘s)

 

What needs clarification here?

I‘m probably not the best person to speak on the ordering of the BeginFrame message and input.

Thanks for the write-up! It would be great if we could minimize the differences between the Android implementation and other platforms as much as possible. [email protected] recently did some major cleanup on the Android browser side logic, so unifying it should be much easier now.

Thanks for reading, Sami!

I added proposed diagram for android, too.

Yes. All platforms are moving to Browser compositors that use SingleThreadProxy.

 

In this case, it‘ll actually be nice since you don‘t need to make a thread hop to get from the Scheduler to the RWHV‘s BeginFrameManager.

[email protected], When browser compositor uses single thread + cc-scheduler, how we can get BeginFrame message? Does SyntheticBeginFrameSource take that role, too on main thread?