Thanks (again) for this report!

We ideally need to be sure that this change won't:
- seriously change the binary size of Chrome on Android/Linux (CrOS probably doesn't care much about these benchmark-based profiles anymore :) )
- seriously increase the time it takes to run our AFDO pipeline.

Luis -- are there any other concerns you can think of here? If not, I'm happy to try to evaluate this (especially if it's easy for me to kick off e.g. a trybot that generates a new profile for me :) )

George, You can modify https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py to run the new test.

Please also set gs_dest unconditionally to GS_TEST_DEST (https://chromium.googlesource.com/chromiumos/third_party/autotest/+/master/server/site_tests/telemetry_AFDOGenerate/telemetry_AFDOGenerate.py#327).

Once done and assuming you have a local build for chell, you can run the test on one of the lab machines.

my concern is that we are moving away from benchmark based profiles. 
Can we evaluate this with the profiles generated from the field?

+1 to llozano's concerns about field-based profiles. 

In any case, it looks like the smoothness benchmarks were removed entirely a few months ago ( issue 868756 ), so we'll need to find something else. Given the content of that issue, it sounds like rendering.desktop would be a good choice, but it has ~240 independent pages it benchmarks, which would presumably take forever(*).

Next step for me is to figure out what pieces of rendering.desktop came to replace smoothness.*, and which of those it's feasible to run as part of our pipeline. Stay tuned. :)

(*) - I tried to quantify this, but apparently even trying to just run them all as part of the AFDO pipeline is broken.

 Hi George and llozano， we evaluated field-based AFDO impact, and found field-based AFDO can't benefit canvas cases(CanvasMark) and benefit Speedometer2 less than benchmarkset-based AFDO (4.72% vs. 14.01%). details in https://docs.google.com/document/d/1T9uiNi9J5TAD8s0q-aB9-Ad9zi-vyViHbywLl_effkY/edit#heading=h.cf2whog89982
 We also observed smoothness is migrated to rendering.desktop. but some cases are kept, such as canvas related workloads(smoothness.tough_canvas_cases). @George, you can try it through 
 ./run_benchmark --browser=cros-chrome --remote=DUT_IP --story-tag-filter=tough_canvas rendering.desktop 

@George, how is rendering.desktop evaluation?

Thanks for that!

Just got back from my vacation; I hope to look into it later this week. Like said, as long as it's not a meaningful size regression on Android, I think we should be good. :)

~> du ARM*/libchrome.so
46780   ARMStandardAFDO/libchrome.so
46804   ARMStandardAFDONewProf/libchrome.so

So, +24K of binary size on Android/ARM's side. The AFDO pipeline is also slowed by ~15min by the addition of rendering.desktop with `--story-tag-filter=tough_canvas`. This makes the AFDO_Record stage of said pipeline take approximately 45 minutes to execute, start to finish.

I think both of these are OK. I'm going to fire off a few pinpoint jobs, but if those come back clean, I think we should be good here.

CLs:
- https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1366415
- https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1366336

I don't plan on landing these CLs today, so that people have time to speak up if they believe that +15min or +24KB are too high of prices to pay for better canvas (and apparently loading.*?) performance.

can we limit the execution time a little more? is there a story or stories that we can execute instead of story tags? 

A while back, integration was not very happy with us reaching the 60 mins for the AFDO collection. NOt sure if that is still the case. 

the 24KB should not be a problem.

I don't know offhand. FWIW, the stories we ran were:

[ RUN      ] rendering.desktop/bouncing_balls_15
[       OK ] rendering.desktop/bouncing_balls_15 (38072 ms)
[ RUN      ] rendering.desktop/bouncing_balls_shadow
[       OK ] rendering.desktop/bouncing_balls_shadow (20572 ms)
[ RUN      ] rendering.desktop/bouncing_clipped_rectangles
[       OK ] rendering.desktop/bouncing_clipped_rectangles (20624 ms)
[ RUN      ] rendering.desktop/bouncing_gradient_circles
[       OK ] rendering.desktop/bouncing_gradient_circles (21802 ms)
[ RUN      ] rendering.desktop/bouncing_png_images
[       OK ] rendering.desktop/bouncing_png_images (16542 ms)
[ RUN      ] rendering.desktop/bouncing_svg_images
[       OK ] rendering.desktop/bouncing_svg_images (28011 ms)
[ RUN      ] rendering.desktop/canvas_animation_no_clear
[       OK ] rendering.desktop/canvas_animation_no_clear (23503 ms)
[ RUN      ] rendering.desktop/canvas_arcs
[       OK ] rendering.desktop/canvas_arcs (19859 ms)
[ RUN      ] rendering.desktop/canvas_font_cycler
[       OK ] rendering.desktop/canvas_font_cycler (21069 ms)
[ RUN      ] rendering.desktop/canvas_lines
[       OK ] rendering.desktop/canvas_lines (23690 ms)
[ RUN      ] rendering.desktop/canvas_to_blob
[       OK ] rendering.desktop/canvas_to_blob (22050 ms)
[ RUN      ] rendering.desktop/chip_tune
[       OK ] rendering.desktop/chip_tune (34189 ms)
[ RUN      ] rendering.desktop/crafty_mind
[       OK ] rendering.desktop/crafty_mind (22971 ms)
[ RUN      ] rendering.desktop/effect_games
[       OK ] rendering.desktop/effect_games (22722 ms)
[ RUN      ] rendering.desktop/fill_shapes
[       OK ] rendering.desktop/fill_shapes (20474 ms)
[ RUN      ] rendering.desktop/geo_apis
[       OK ] rendering.desktop/geo_apis (16583 ms)
[ RUN      ] rendering.desktop/hakim
[       OK ] rendering.desktop/hakim (22324 ms)
[ RUN      ] rendering.desktop/jarro_doverson
[       OK ] rendering.desktop/jarro_doverson (25465 ms)
[ RUN      ] rendering.desktop/kevs_3d
[       OK ] rendering.desktop/kevs_3d (21892 ms)
[ RUN      ] rendering.desktop/man_in_blue
[       OK ] rendering.desktop/man_in_blue (22020 ms)
[ RUN      ] rendering.desktop/many_images
[       OK ] rendering.desktop/many_images (28603 ms)
[ RUN      ] rendering.desktop/megi_dish
[       OK ] rendering.desktop/megi_dish (18383 ms)
[ RUN      ] rendering.desktop/microsoft_asteroid_belt
[       OK ] rendering.desktop/microsoft_asteroid_belt (22995 ms)
[ RUN      ] rendering.desktop/microsoft_fireflies
[       OK ] rendering.desktop/microsoft_fireflies (26722 ms)
[ RUN      ] rendering.desktop/microsoft_fish_ie_tank
[       OK ] rendering.desktop/microsoft_fish_ie_tank (24360 ms)
[ RUN      ] rendering.desktop/microsoft_snow
[       OK ] rendering.desktop/microsoft_snow (22820 ms)
[ RUN      ] rendering.desktop/microsoft_speed_reading
[       OK ] rendering.desktop/microsoft_speed_reading (21886 ms)
[ RUN      ] rendering.desktop/microsoft_tweet_map
[       OK ] rendering.desktop/microsoft_tweet_map (22132 ms)
[ RUN      ] rendering.desktop/microsoft_video_city
[       OK ] rendering.desktop/microsoft_video_city (33541 ms)
[ RUN      ] rendering.desktop/microsoft_worker_fountains
[       OK ] rendering.desktop/microsoft_worker_fountains (23492 ms)
[ RUN      ] rendering.desktop/mix_10k
[       OK ] rendering.desktop/mix_10k (20636 ms)
[ RUN      ] rendering.desktop/put_get_image_data
[       OK ] rendering.desktop/put_get_image_data (21029 ms)
[ RUN      ] rendering.desktop/runway
[       OK ] rendering.desktop/runway (18740 ms)
[ RUN      ] rendering.desktop/smash_cat
[       OK ] rendering.desktop/smash_cat (22822 ms)
[ RUN      ] rendering.desktop/spielzeugz
[       OK ] rendering.desktop/spielzeugz (19710 ms)
[ RUN      ] rendering.desktop/stroke_shapes
[       OK ] rendering.desktop/stroke_shapes (19884 ms)
[  PASSED  ] 36 tests.

I'd be surprised if all of these are necessary, but I'll defer to our Intel friends if they have any thoughts :)

Thanks George for your evaluation. Depending on your feedback, we tried to reduce AFDO profiling time. Through preliminary analysis, we simplified rendering.desktop tough_canvas to bouncing_* + canvas_* + microsoft_* (19 subcases), which covers the most of H5 Canvas features. The profiling time introduced by rendering.desktop tough_canvas can reduce a half(from 15mins to 8mins) without performance impact for Speedometer2 and CanvasMark in Android Chromium (70.0.3532.0) on Electro ARC++. Do you think ~8 mins overhead is acceptable?

Command for running rendering.desktop w/ selected stories:
./run_benchmark --browser=cros-chrome --remote=DUT_IP --story-tag-filter=tough_canvas rendering.desktop --story-filter="bouncing*|canvas*|microsoft*"

Performance comparison 
              whole rendering.desktop 36 stories | selected rendering.desktop 19 stories
Speedometer2           16.57                     |          16.69
CanvasMark             4644                      |          4674
Binary size (k)        85265                     |          85261

Thanks!

I chatted offline with Luis, and he thinks +8mins is fine. I'll put together CLs to do this later today.

CLs:

- Add tough_compositor_cases resources to Chrome, so we'll copy them over to where we need them:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1376874

- Actually add the benchmarks: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1376891

Since we're not in a rush to do this (branch point isn't for a while :) ), I'll do a final vet of the second CL after the first one lands. Assuming that goes well, I'll ask for reviewers and such then.

Got it, thanks very much

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/214485a4475207bf194b6daee7f5838defddfe28

commit 214485a4475207bf194b6daee7f5838defddfe28
Author: George Burgess IV <gbiv@chromium.org>
Date: Sat Dec 15 06:40:38 2018

chrome: add tough_compositor_cases to chrome's test resources

We need this so that we can run pieces of rendering.desktop as a part of
our AFDO pipeline.

`du` tells me that this directory is around 44KB large, so carrying it
with us probably shouldn't be a problem.

BUG=chromium:882349
TEST=Ran said pipeline with these changes. Appears to work.

Change-Id: Ibd5c3a103844902c9102f44090bf581416ff9851
Reviewed-on: https://chromium-review.googlesource.com/1376874
Commit-Ready: George Burgess <gbiv@chromium.org>
Tested-by: George Burgess <gbiv@chromium.org>
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[modify] https://crrev.com/214485a4475207bf194b6daee7f5838defddfe28/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

As a status update: for some reason, the new benchmarks are slowing down a few of our Android ARM devices on speedometer2 pretty consistently by ~2%. I'm going to try some local experiments, since I suspect we may have many new inlining decisions that aren't playing nicely with the orderfile. If it is, that 2% should disappear once we have a new AFDO profile + new orderfile to go with it. If not, I'll poke it a bit more.

Stay tuned, and happy new year :)

Thanks George. Let's wait for your results. Happy new year!