Add ability to run performance benchmarks on engine PR

This issue has been created since 2022-09-22.

We should add and document the ability to run the full gamut of performance benchmarks on an engine PR.

This came up when I created a potential performance improving PR that works indiscriminately across the whole codebase (https://github.com/flutter/engine/pull/36221/files). We'd like to see the results of benchmark testing before landing it since its benefit is complicated by optimizations the compiler can take.

I talked with @dnfield and @christopherfujino and it sounds like there may be something close to this available where it is launched with led, but it may need to be cleaned up and documented.

keyonghan wrote this answer on 2022-09-23

Related bugs:

  1. recipes support to run a devicelab test: #111119
  2. framework tests that should run in engine: #83731
christopherfujino wrote this answer on 2022-09-23

So, we do have it wired up that you can run a "local" engine from an arbitrary engine commit:

The download is happening via the recipe_engine/cas module: https://flutter.googlesource.com/recipes/+/refs/heads/main/recipe_modules/flutter_deps/api.py#34, I'm not sure exactly how this works (@whesse do you know?)

When triggering an engine build via led, we upload to an experimental namespace in our storage bucket: https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine/engine.py#94. If we could add another property that told the flutter_deps.flutter_engine module code to download from the experimental namespace, I think we could have a workflow like:

  1. trigger a led build on your PR for the particular engine target you are interested in
  2. trigger a led build of the benchmark you are interested in, specifying you want to download experimental binaries and the commit from your engine PR, which would then download the PR binaries and pass them to the tool with --local-engine
More Details About Repo
Owner Name flutter
Repo Name flutter
Full Name flutter/flutter
Language Dart
Created Date 2015-03-06
Updated Date 2022-10-05
Star Count 145516
Watcher Count 3569
Fork Count 23400
Issue Count 11194

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
Newbie How Read Value To Variable - try read step 1 2022-03-28 2022-09-17
Add Cash By Mail payment method 6 2021-03-08 2022-07-22
Add KRW as a fiat currency. 0 2021-03-08 2022-07-22
Unclear about adding existing plots and already on the pool? 1 2021-03-29 2022-07-22
Deposit/Dispute Dialogue box not wide enough to accommodate XLA addresses 1 2021-03-15 2022-07-22
segmentation fault when importing instaboost 6 2019-08-27 2022-07-22
Silent failure with website titles containing slashes 0 2022-04-10 2022-05-21
Better documentation with Sphinx / ReadTheDocs 1 2020-12-20 2022-09-25
Data Generation results 22 2020-05-28 2022-08-10
Add a checkbox "save automatically" to the save dialog when running a simulation 4 2017-03-22 2022-09-08
Having body causes duplicate log entries 7 2022-06-16 2022-08-18
Any plan or pointer to have lurch as a internal library function 0 2022-09-04 2022-09-21
Tool icons disappear after shape is created in Godot 3.4+ Linux x64 3 2021-12-09 2022-02-03
the launcher has exited with a non-zero error code; I only get this error when I install very heavy shaders or textures 2 2021-11-25 2022-07-24
CHECK_EQUAL_C_DOUBLE 0 2021-09-06 2022-09-29
NFS K8s development FailedMount 0 2021-02-02 2022-08-02
Support polls 6 2018-12-23 2022-01-15
one of the params is missing or invalid 26 2018-03-07 2022-07-13
[BUG] : Failed to download song 0 2022-01-17 2022-10-03
Allowlist of required metrics for federated setup 17 2021-06-09 2022-08-30
Upgrade PHP (move away from v7) 12 2021-11-29 2022-10-02
在这里,我看到了世界各国洗脑其民众的现状. 10 2022-05-19 2022-09-24
麻烦来点美国生物实验室和发起乌克兰战争的实锤,其他国爆料也可 4 2022-05-19 2022-09-24
How to take a screenshot for guake 2 2021-10-27 2022-08-16
External Storage not all store setttings from scratch 0 2021-09-12 2022-09-05
Cannot Install - Errors with C++ Compilation 8 2021-08-17 2022-08-23
An error of the process of converting to xmodel with Vitis_ai_pytorch 2 2022-02-24 2022-09-13
[RFC] Support easy-to-use Storage in Multi-process on the single server. 8 2021-09-23 2022-10-01
VoLTE got broken with latest v313 patches 2 2021-09-19 2022-09-22
[Bug] modin-on-ray's unwrap_partitions is 100x slower on Mac than on Windows 64 2021-11-15 2022-09-17
Why in composer.json: platform: php: 7.3.0? 4 2022-01-30 2022-09-09
Rarely wrong case of only_full_group_by check 0 2022-04-27 2022-06-09
_malloc(NaN) is hard to debug 1 2022-07-01 2022-09-12
How to specify the payee with the android-checkout-sdk? 5 2021-08-19 2022-09-28
storescp application's efficiency is too low 3 2021-09-27 2022-10-04
[Jammy] App version 2.8.3 is installed on Ubuntu 22.04 instead of 2.9 using the latest build from main branch 2 2022-06-03 2022-09-14
importing from dist folder gives error 0 2021-09-27 2022-09-28
[DOC] include a bug URL for every API 1 2022-04-14 2022-09-16
Using Netlify as a host 1 2021-06-01 2022-09-26
Telegram 8.8.1: SIGABRT on MacOS 12.5 (arm64) 0 2022-08-06 2022-09-13
Support ESLint 8.x 2 2021-08-28 2022-01-07
Crash when starting video 2 2021-04-29 2022-09-06
Confusing pwa- prefix in documentation 3 2022-09-30 2022-09-28
Add wasm target 4 2018-05-22 2022-01-10
have any plans for admin_create_group like admin_create_topic? 1 2021-07-27 2022-09-25
Firefox tabs crash. 0 2019-03-15 2022-01-04
add plugin description and OBP icons 3 2021-04-09 2022-08-21
运行实例程序时undefined: beego.Run 0 2020-12-07 2022-09-16
compile error 7 2021-11-15 2022-01-16
Allow namespaces to provide a description for display in root help 3 2017-08-07 2022-09-04