Support using `dart:ui` `Scene`s where `Image`s are accepted

This issue has been created since 2022-06-30.

Use case

There have been many cases recently where developers need to snapshot UI as a bitmap for efficiency reasons, or to achieve effects that are unobtainable with the current SceneBuilder APIs. Previously, developers could use toImage() to convert a Scene or Picture asynchronously. And now developers can use toGpuImage() to do the same, but lazily.

With the GC issues seen in #106621, there are more valid usecases for snapshotting a potentially updating Scene while letting the Raster Cache manage the memory (instead of Dart).

A Scene-level solution can also take advantage of damage rect calculations, allowing automatic partial updates which improves memory bandwidth and rendering performance.

Proposal

This API is a WIP, and is subject to change with feedback.

At the core of the API, we have SceneImage. As a subclass of dart:ui Image it can go wherever it goes.

class SceneImage extends Image {
  SceneImage(int width, int height);

  void render(Scene scene); // Mirrors [FlutterView.render] intentionally. May not render the scene immediately.
}

However the key difference is that these images weakly reference an Image in the Raster Cache. The only thing SceneImage stores is the dimensions, and the last scene passed to SceneImage.render. This reduces the size of SceneImage reported to Dart's GC.

As long as the SceneImage is utilized within a frame, the scene is retained in the Raster Cache. Updates to the scene (through calls to SceneImage.render) may use the previously rendered contents to do a partial update.

SceneImage is a bridge to the Raster Cache, without exposing it through dart:ui.

cc @flar @dnfield @jonahwilliams

jonahwilliams wrote this answer on 2022-06-30

With the GC issues seen in #106621, there are more valid usecases for snapshotting a potentially updating Scene while letting the Raster Cache manage the memory (instead of Dart).

The GC issues are almost certainly due to a bug in dart regarding allocation of objects with a large external size. Working around this should not be a consideration for a new API

dnfield wrote this answer on 2022-06-30

Why is letting the raster cache manage memory a benefit?

The raster cache does not know as much about when you will be done with a scene's resources - it uses a relatively aggresive heuristic.

I'm not completely opposed to this, but I'm failing to see why the engine's raster cache would be an upside here - in particular, if the engine were factoredin such a way that the raster cache got dropped, what then?

ds84182 wrote this answer on 2022-06-30

@dnfield

Why is letting the raster cache manage memory a benefit?

The raster cache does not know as much about when you will be done with a scene's resources - it uses a relatively aggresive heuristic.

In the most common case, the app is finished with a scene's resources (e.g. SceneImage) when it stops drawing the scene. If you're looking for a way to persist a GPU resident image of a scene for a long period of time this isn't the API to use.

This is exactly why it's beneficial. In the case of a full screen transition, the rasterized image should only exist as long as it is used.

There are also open questions about GPU context loss. An intermittent GPU crash would discard the contents of a GPU resident image. But for SceneImage you can recover from this because there's an explicit reference to the source Scene.

I'm not completely opposed to this, but I'm failing to see why the engine's raster cache would be an upside here - in particular, if the engine were factoredin such a way that the raster cache got dropped, what then?

Since this is separated from the Raster Cache, the internal implementation details can change over time. The only guarantees given to Flutter is:

  • The scene is rasterized before use, if uncached or dirty
    • The exact timing of scene rasterization is undefined, but it is always rasterized before use
  • The rasterized scene is persisted across consecutive frames, as long as the image is used within those frames
    • The rasterized scene can be dropped and rerasterized between two frames in extreme circumstances (GPU crash/context loss)
  • Scenes with circular dependencies will not render correctly ("undefined behavior", but not a crash)
dnfield wrote this answer on 2022-07-01

In the most common case, the app is finished with a scene's resources (e.g. SceneImage) when it stops drawing the scene. If you're looking for a way to persist a GPU resident image of a scene for a long period of time this isn't the API to use.

The rater cache is really only suitable for use in cases where you're drawing for more than one frame though.

The scene is rasterized before use, if uncached or dirty

The if uncached or dirty party is very expensive and not likely to generalize well without help from the framework and/or application.

I feel like I'm probably misunderstanding something here though. It's not entirely clear to me why having this API would be better than having the now implemented Scene.toGpuImage.

flar wrote this answer on 2022-07-01

In the most common case, the app is finished with a scene's resources (e.g. SceneImage) when it stops drawing the scene. If you're looking for a way to persist a GPU resident image of a scene for a long period of time this isn't the API to use.

The rater cache is really only suitable for use in cases where you're drawing for more than one frame though.

Any cache is really only suitable for those cases. The existing engine RasterCache suffers from not being able to predict that very well, though.

One odd thing is that the RC hints are designed around "I'm going to change" which is obvious when the painting changes, so why do we need a hint for that? What we really should have had originally would be a "I'm going to persist for a while" hint. If we invert the "willChange" hint then we have a useful mechanism here.

Also, the isComplex and willChange hints are buried where only CustomPaint objects will encounter/have access to them.

flar wrote this answer on 2022-07-01

Why is letting the raster cache manage memory a benefit?

I don't necessarily see that as the main driving force for this concept.

The main thing I see is to provide a single way of drawing parts of the scene with modifications and not requiring a particular implementation for that method to work.

The main thing is that the recent transition optimization required that the scene be turned into an image in order to even function. If the toGpuImage mechanism isn't available (such as on web?) then it has to have some other kind of code to back up to in order to render anything. This means that it has to have multiple ways to express what it is trying to accomplish.

What I'd like to see is that it could express its existing desires using some sort of "drawSceneThingy(...params resemble a call to drawImage...)" and the engine impl (web vs. native vs. CanvasKit) could decide if it could rasterize the thingy and perform a drawImage or not. The API would also include hints that doing the operation as a rasterization would be beneficial, so basically the impact to the transition animation would be something like:

  drawChildren(Paint..alpha..otherstuff, transform, pleaseCache: true);
  // OR
  drawScene(getChildScene(), Paint..alpha..otherstuff, transform, pleaseCache: true);

(with the latter, you could even freeze the children by caching their sub-scene when the animation starts)

The implementation of that could be something flexible, like:

  if (pleaseCache && toGpuImage available) {
    if (_image_ == null) {
      _image = toGpuImage(scene);
    }
    if (_image != null) {
      canvas.drawImage(_image, paint);
      return;
    }
  }
  pushTransform;
  pushOpacity;
  pushOtherAttrs;
  paintChildren();

and that same code would work on platforms that can create these images, and on platforms that cannot, and potentially even render something in a low-memory situation.

The idea is that the platform code should be able to express this concept without having to explicitly bake in "and to do that you need to turn this piece of a scene into an image". That aspect of the solution is both micro-managing and assumes that all implementations can create those images. Once the platform code needs to call toGpuImage itself, it is forcing a specific solution.

dnfield wrote this answer on 2022-07-01

The engine's raster cache is very useful when it's useful. However, there are only a relatively small number of cases where it is so universally useful that we feel confident in forcing that usage on all applications. IOW, we are conservative about letting things into the engine's raster cache and aggressive about evicting them.

There are a large number of application specific cases where it would be even further useful, which leads to developers wanting to use it for more things. We get a lot of requests to make it easier or more likely to raster cache things, and always seem to land on "this will work for application X but not application Y." We also have active efforts underway to make the raster cache less necessary with Impeller.

Drawing an Image is relatively cheap, especially if that image is already GPU resident.

Drawing (including rasterizing) a Scene is not necessarily cheap.

As far as multi-plat support, we can evolve the API on web so that it boils down to the draw calls behind it where necessary and make toGpuImage work on the HTML backend. But the HTML backend already has a non-trivial number of gotchas around rendering anyway, so what's one more? :)

dnfield wrote this answer on 2022-07-01

And letting the application use an Image lets it choose to cache a rasterization of a complex drawing for as long as the application needs - as opposed to a more flexible API where the application no longer knows what it's actually holding on to or how expensive that thing is to rasterize.

flar wrote this answer on 2022-07-01

The engine's raster cache is very useful when it's useful. However, there are only a relatively small number of cases where it is so universally useful that we feel confident in forcing that usage on all applications. IOW, we are conservative about letting things into the engine's raster cache and aggressive about evicting them.

I would just like to point out that these reservations you mention here are policy and not a required design feature of the cache. We've made poor choices, but rather than fixing the choices you assume that it will never work.

dnfield wrote this answer on 2022-07-01

I don't think we've made poor choices though. I think the raster cache is very good at a limited number of things, and it would be a mistake to try generalize it too much.

dnfield wrote this answer on 2022-07-01

(This is not to take away from the very good efforts going on to improve its ability to do the things that it is already good at either)

flar wrote this answer on 2022-07-01

And letting the application use an Image lets it choose to cache a rasterization of a complex drawing for as long as the application needs - as opposed to a more flexible API where the application no longer knows what it's actually holding on to or how expensive that thing is to rasterize.

But by the same token, requiring the application to use an Image requires it to only work on platforms where that is possible.

The point I've been trying to make is to express this in a way that if a GpuImage mechanism exists we end up with exactly the same behavior. But when it doesn't exist, we can fall back to other opportunities rather than reject their expression of what they were trying to do as "syntactically impossible".

Remove the specifics from the expression of the operation so that it doesn't tie the code into a single mechanism. The specifics can still be the way that the operation is actually performed, but alternatives are not explicitly ruled out.

flar wrote this answer on 2022-07-01

I don't think we've made poor choices though. I think the raster cache is very good at a limited number of things, and it would be a mistake to try generalize it too much.

Having worked on a number of caching mechanisms in the past... We have made poor choices.

dnfield wrote this answer on 2022-07-01

Could you elaborate?

flar wrote this answer on 2022-07-01

Could you elaborate?

  • Providing a non-sensical "willChange" hint which is useless and doesn't give the important information needed from the framework.
  • Caching on the 3rd frame with no override. It should be the 2nd frame unless a "wontChange" hint is provided and then it should be on the 1st visible frame.
  • Transform anomalies that are pretty nitty-gritty in the details
  • No reuse of cached image memory when cached items change
  • The engine layers weren't designed to make best use of either saveLayers or cached images
dnfield wrote this answer on 2022-07-01

I think we can and should fix those issues - but that doesn't change the fact that the engine sometimes just lacks knowledge like "this is a transition and I'm willing to care only about particular changes during it and discard others"

flar wrote this answer on 2022-07-01

I think we can and should fix those issues - but that doesn't change the fact that the engine sometimes just lacks knowledge like "this is a transition and I'm willing to care only about particular changes during it and discard others"

It doesn't lack the knowledge, it lacks the communication channel. We should create that communication channel rather than acting like the two will never see eye to eye.

The engine exists as the implementation arm of Flutter framework. If it lacks info it's not by design, it's by failing to fix the communication between the two.

dnfield wrote this answer on 2022-07-01

I'm concerned that adding a communication channel (or expanding on ones that exist) would further complect the engine.

I think in an ideal world, users would not know that the engine has a raster cache. We would remove the current hints that exist, and rather than asking for hints we'd provide API(s) that allow users to achieve the kind of raster-related caching that is appropriate for their framework/package/application. The engine would only raster cache when it knows that raster caching is no more expensive than otherwise rasterizing the scene, and make whatever enhancements it needs to be as efficient about that as possible.

flar wrote this answer on 2022-07-01

Caching involves decisions that they are unlikely to make involving whether or not it is available, whether the allocations required are supported by the Gpu. Whether the transforms under which you are caching make sense.

In the easy cases when you start caching everything seems so simple, but there are nitty gritty details that end up making it a mechanism that can bite you if you don't maintain the code. That's kind of why it is behind an implementation wall.

Even with the PR for the page transition there is currently a comment about a "draw nothing" fallback that is missing an implementation. If the mechanism had a fallback then that wouldn't be an issue, but since the implementation is done in a way where "make me an image or my entire approach fails" is accepted, the fallback is needed and must be done manually.

It's not rocket science, but it is fiddly and it has a number of failure modes that have to be programmed in. Putting this behind a more general API that expresses what you want to cache and what you want to do with it means that the implementation can fail gracefully. Exposing the "image-ness" of it requires the developer (in that PR, it is a framework developer, but if this mechanism becomes more exposed this will be "every Flutter developer") to be aware of all of the failure modes and deal with them.

And I don't think this issue is saying "don't do anything like the approach you are trying and go back to the old ways", it's just saying "express it in a nearly identical way that can manage the fallbacks for the programmer".

Caching pieces of the scene to images is an implementation detail, not a friendly API feature.

flar wrote this answer on 2022-07-01

Also, I'm not looking at this API as "this is a good way to introduce framework level caching", I'm actually looking at it as a friendlier way for the framework and Flutter developers to express "I want to draw that part of the scene with these modifications" than our existing "pushThis, pushThat, render children".

Would it be more developer friendly to offer "pushOpacity/Transform/ThisThingy/ThatThingy"?

Or "drawScene(...mods...)"?

Every existing layer except for DisplayList, Platform Views and perhaps Texture (all of which are leaf nodes), boils down to "push<PaintAttribute>()" or "pushTransform (which itself is a Canvas attribute)".

ds84182 wrote this answer on 2022-07-07

I'm trying to understand the benefit of drawScene. Since you can potentially rebuild a new scene every frame, that would mean rebuilding the associated display list (and then display list layer, and associated scene) every frame.

Something I haven't understood about the scene representation is why it requires rebuilding a Scene every frame for trivial changes. Unlike RenderObjects, there's no way to say "change this transform" or "change this paint attribute" without having to destroy and recreate a Scene. This is in direct contrast to Windows' DirectComposition API, Fuchsia's Flatland API, and even the DOM on Web.

The current model for scenes grants the ability to atomically update everything, but it has the obvious downside of having to update everything to change one thing.

In my eyes, exposing drawScene as a general primitive further propagates these "rebuilding issues", which is why the SceneImage API outlined above was intentionally decoupled. Changing the sub-Scene shouldn't require rebuilding the world. Similar to how a Texture or PlatformView can redraw without having to change the Scene from the Dart side.

flar wrote this answer on 2022-07-07

RepaintBoundary widgets and their associated RenderObjects break the scene up into pieces that firewall the changes made. A RepaintBoundary can reuse its representation if none of its descendants were marked as different.

Similarly various animations use a "builder" paradigm so they only rebuild their portion of the scene.

Then the various passes through the trees will attempt to reuse objects from the previous scene. Widgets -> Elements -> RenderObjects -> Layers. You can see more about it in the various talks that go over the Mahogany Staircase as the concept is called. For example https://www.youtube.com/watch?v=dkyY9WCGMi0

ds84182 wrote this answer on 2022-07-08

Yes I know all of that, but when going from Layers (in Dart) to a Scene (in C++) you do it via SceneBuilder, which partially reconstructs the resulting Scene. So when you have an EngineLayer for a transform, you can't simply update the transform without having to push the parent layers and add the sibling layers.

With drawScene, the reference to the scene is directly inlined within the DisplayList that the RenderObject is painting into. So when you want to change the internal scene, you have to write a completely different DisplayList for PictureLayer. Updating PictureLayer marks all of its ancestor layers as dirty as it needs to be readded to the scene to update.

Then you enter the framework's compositing phase. Dirty layers are visited, pushing their respective engine layer into SceneBuilder. For non-dirty layers, they're added via addRetainedLayer which prevents recursing into its children.

The issue is, we should be able to skip framework-level painting and compositing for the "root" Scene when we want to update a sub-Scene drawn via drawScene. This should be possible because we already skip these framework-level phases when repainting due to an engine-side TextureLayer or PlatformViewLayer updating.

And additionally, if we can skip framework-level compositing in this scenario, what's preventing us from skipping framework-level compositing when updating transforms on an existing TransformEngineLayer, or updating opacity on an existing OpacityEngineLayer?

ds84182 wrote this answer on 2022-07-11

Revisiting this again, I think both drawScene and SceneImage will run into issues with Platform Views using Hybrid Composition. In particular a PlatformView can appear inside of a DisplayListLayer that uses either mechanism to nest a Scene.

flar wrote this answer on 2022-07-11

Related to an earlier comment: #101810

flar wrote this answer on 2022-07-11

I had an idea once that I thought was in an issue, but I can't find an issue that mentions it.

Basically, rather than having to add children to the tree by value, we add them by reference. Then when a child (subtree) updates, it just redefines its referenced value in the new scene. We would need a "child by reference" object to insert into the tree and it would only resolve itself when the tree was actually operated on by the engine. A Scene would be a skeleton of non-retained layers with a few "reference layers" sprinkled throughout it and a list of "reference -> layer" mappings.

Not all children would be "referenced", just the ones that we expect might change - such as the children of a RepaintBoundary.

ds84182 wrote this answer on 2022-07-12

That's an interesting idea. Over the weekend I was experimenting with a Scene editing API which boiled down to:

abstract class SceneEditor {
  void updateTransform(TransformEngineLayer layer, Float64List newTransform);
  void updatePicture(PictureEngineLayer layer, Picture newPicture);
  void rebuildChildren(EngineLayer layer, void Function(SceneBuilder) build);
}

The idea was that we should be able to record changes to the Scene and apply them to the Window's previous Scene.

For example, scrolling a list would use updateTransform on the viewport's TransformLayer. When list items are added and removed, rebuildChildren is used to rebuild part of the scene in the scrolling viewport. If an individual item changes, updatePicture is used to update it visually.

ds84182 wrote this answer on 2022-07-12

"Reference layers" also accomplish the same goal, with the advantage of not increasing the API surface by much. But I feel SceneEditor would also benefit the HTML backend for Flutter Web.

More Details About Repo
Owner Name flutter
Repo Name flutter
Full Name flutter/flutter
Language Dart
Created Date 2015-03-06
Updated Date 2022-09-30
Star Count 145381
Watcher Count 3565
Fork Count 23363
Issue Count 11208

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
[Question] Optional brackets parser? 5 2021-05-04 2022-07-18
🐞: Multiple errors in data convertion 2 2020-04-12 2022-05-07
💡: Support for complex number 2 2020-04-14 2022-05-07
uWSGI randomly resets TCP connections 3 2021-05-24 2022-05-15
supporting `protocol: raw` in matchProtocols 1 2022-04-21 2022-07-26
0.1.8 - downgrade possible? No libgraphics.so (by the way: why?) for CentOS#8 2 2021-01-12 2022-09-19
Add-PodeWebPage Group support with spaces 1 2022-07-19 2022-08-04
Training NERF using real-captured data 34 2020-05-30 2022-07-20
Support fox Pixiv Ugoira animated files 0 2021-10-25 2022-02-15
No dependency management found 0 2021-04-30 2022-09-07
installinstallmacos.py gives Not a gzipped file when using other catalogs 4 2020-09-25 2022-09-06
Garbled output in `3.0.1` 0 2021-08-14 2022-09-12
[Deploy] GitKraken Boards 0 2021-09-13 2022-09-15
Google Calendar is not working properly 0 2021-04-30 2022-09-15
[Deploy] Diolinux Plus 0 2021-07-01 2022-09-15
[FEATURE] 请问启动耗时 那部分可以开源吗 包括so ,另外 启动耗时的原理是啥 现在我测试app 冷启动耗时 感觉一点不准,计算出来的时间 起始点是什么呢感谢 0 2022-04-14 2022-09-15
Select dropdown does not pre-select existing value in Settings when string is used 3 2021-10-16 2022-07-21
Can't enumerate devices, method not supported 2 2020-12-21 2022-09-29
understanding-arrow-functions-in-javascript/ 5 2020-08-05 2022-07-15
Using watchtower with Digital Ocean container registry 8 2021-12-02 2022-08-30
1.4.1 release candidate 1 2021-11-01 2022-02-03
Support StyleSheet.create for Theming Customization 3 2021-07-06 2022-09-21
`to_numpy_recarray` throws `KeyError: weight` 2 2022-02-07 2022-09-14
how to set node value by using xpath(no ID or name for some element) 0 2021-05-19 2022-09-26
Failure on Alpine Linux due to ELF parsing 2 2022-08-24 2022-09-19
[Bug]: Resetting list widget causes unexpected page size behaviour 2 2022-06-22 2022-09-26
OMV 6.x OneDrive Plugin - omv-onedrive-auth missing ? 1 2022-04-16 2022-09-19
[4.x] Router had or has a bug? 12 2022-03-16 2022-09-29
An error was reported in integration 1.1.1 configuration 1 2022-06-20 2022-09-30
Unignore `F401 module imported but unused` in flake8 1 2021-12-11 2022-09-23
Bump github.com/tendermint/tendermint from 0.33.3 to 0.34.7 in /incubator/nft 1 2021-02-18 2022-02-10
Raspberry pi 3 gpu not used or poor performance 2 2022-03-18 2022-09-14
crc32 scope hoisting issue 2 2022-03-31 2022-08-21
Boundings error with text break line on words and characters 1 2021-08-28 2022-09-30
createSubscriptionHandshakeLink documentation is not accurate with MQTT deprecation 3 2021-03-17 2022-09-20
object prop can't populate options but string? 0 2022-05-22 2022-09-17
krew.sh install is broken 4 2021-10-20 2022-09-22
Bump @actions/github version for GitHub Enterprise Server support 0 2022-05-13 2022-09-18
Problems about flashing speed (IDFGH-7099) 1 2022-04-02 2022-09-20
Ability to show HasMany count on Show page 2 2022-05-13 2022-09-14
在dureader上提交榜单两天一直在calculating 2 2019-07-31 2022-09-14
c3c9d6b15f breaks build with clang++ as C-compiler 3 2022-03-10 2022-08-11
BBC: Blead Breaks Regexp::Grammars 5 2022-03-11 2022-08-28
BBC: Blead Breaks Excel::Writer::XLSX 5 2022-03-11 2022-09-27
BBC: Blead Breaks Test::Smoke 4 2022-03-11 2022-09-26
BBC: Blead Breaks Set::CrossProduct 8 2022-03-11 2022-08-29
sidebar navigation dropdown in react.js argon dashboard 1 2021-07-04 2022-09-30
chore(deps): bump @serverless-stack/resources from 0.37.1 to 0.37.2 0 2021-08-10 2022-06-22
Cloud run django app cannot contain "authentication" in path 6 2021-10-15 2022-09-22
Quarantine InvalidProcessPath_ExpectServerError 1 2022-05-25 2022-09-30