#CBC-RS/TC-watchlist

Absolutely true.
A Flip "veyron-minnie" from last summer gave me 7200.  Now the new Flip gives Octane considerably lower than the lower level of 12% variation to be expected.

Deleted: Screenshot 2016-06-25 at 10.12.18 PM.png 198 KB

	Screenshot 2016-06-25 at 10.12.18 PM.png 198 KB View Download

The performance regression also is evident on my Acer R11 that is on dev channel (waiting patiently for Android). I have both an R11 and a Flip available for testing. The R11 had a similar drop in Octane score, and resulting sluggish performance.

On the Flip, performance is particularly slow for one particular Android app that uses OpenGL to display weather radar. I can provide details if it would help.

Dropped down to 51.0.2704.106 and seeing scores more inline to be expected.

There's definitely an issue in 53.0.2773.3 and higher.

Deleted: Screenshot 2016-07-18 at 9.23.57 PM.png 295 KB

	Screenshot 2016-07-18 at 9.23.57 PM.png 295 KB View Download

To further document (or confuse?) the issue, I just ran the benchmark on both the R11 and Flip.

The Flip scored 6942, in line with #3.

The R11 scored 5554.

Version 53.0.2785.15 dev (64-bit)
Platform 8530.13.0 (Official Build) dev-channel cyan
ARC Version 3050391
Firmware Google_Cyan.7287.57.64

Version 53.0.2785.15 dev
Platform 8530.13.0 (Official Build) dev-channel veyron_minnie
ARC Version 3050391
Firmware Google_Veyron_Minnie.6588.197.0

 Issue 629576  has been merged into this issue.

I can see the renderer process running this tab reaches about 1GB of RAM by the end of the test. We have a regression in V8 GC tracked by  issue 624456 . But I have to confirm that they're related. 

Could you please (willg76x@ or jim.dantin@) record a trace from chrome://tracing while the benchmark is running and focused, and then send the report file?

Make sure to select "Manually Select Settings" > Select "All" from the "Record Categories".

Here you go, but it's throwing errors and might not be useful.

Deleted: Screenshot 2016-07-22 at 8.34.52 PM.png 112 KB

	Screenshot 2016-07-22 at 8.34.52 PM.png 112 KB View Download

Deleted: trace_octane_benchmark.json.gz 19.2 MB

trace_octane_benchmark.json.gz 19.2 MB Download

Here is mine from the Acer R11 (Octane 5608)

Deleted: trace_jimdantin.json.gz 11.4 MB

trace_jimdantin.json.gz 11.4 MB Download

Here is the trace from minnie (Octane 6536)

Deleted: trace_jimdantin_minnie.json.gz 9.5 MB

trace_jimdantin_minnie.json.gz 9.5 MB Download

I reran the tests and captured the screenshots of the results. Note the individual test scores - Cyan should score slightly better than Minnie on all of them. Note, however, that some of the Cyan tests are MUCH slower, while others are where they should be.

Hope this helps identify where the problem is.

Deleted: Cyan_on_dev.png 69.7 KB

	Cyan_on_dev.png 69.7 KB View Download

Deleted: Minnie_on_dev.png 68.2 KB

	Minnie_on_dev.png 68.2 KB View Download

rebooted and reran, success!

Deleted: trace_octane_benchmark_2.json.gz 9.7 MB

trace_octane_benchmark_2.json.gz 9.7 MB Download

I think it's important to note that not all R11's come with the Celeron N3150. Some have the dual-core N3050, which should test a little bit lower. My R11 has the N3150, and seems to be testing pretty close to normal.

My standard testing method is: log in to Guest, run Octane benchmark, exit Guest, repeat. Over three runs I saw an average of 7957, with a maximum variance of 220, overall.

dev version: 53.0.2785.23

Deleted: Screenshot 2016-07-23 at 11.43.42 AM.png 70.1 KB

	Screenshot 2016-07-23 at 11.43.42 AM.png 70.1 KB View Download

Deleted: Screenshot 2016-07-23 at 11.40.16 AM.png 71.6 KB

	Screenshot 2016-07-23 at 11.40.16 AM.png 71.6 KB View Download

Deleted: Screenshot 2016-07-23 at 11.36.23 AM.png 70.1 KB

	Screenshot 2016-07-23 at 11.36.23 AM.png 70.1 KB View Download

N3150 in mine. Unless more logs or tests are needed, I will try a Recovery next week. 

Just for the record, I ran the above tests in my profile and in Guest mode and after resetting flags to default. Happy to document/try anything else that might help.

Re #12: I looked at the trace, and I can see that V8 is doing a LOT of expensive GC operations for the renderer running Octane (see attached). I'm tempted to dupe this into  Issue 624456 , but I need to confirm.

Someone mentioned earlier that om M51 he can get a higher score. Can I ask if possible that someone records a similar trace on M51 (or any non-regressed version) on the same device so that we can compare and confirm it's the same V8 GC regression tracked by  Issue 624456 ?

Deleted: Selection_089.png 118 KB

	Selection_089.png 118 KB View Download

I'll bounce back down to stable and do another run. That run will probably be R52 because Stable had a major update today.

Version 52.0.2743.85
Platform 8350.60.0 (Official Build) stable-channel veyron_minnie
Firmware Google_Veyron_Minnie.6588.197.0

Deleted: trace_stable_run.json.gz 14.9 MB

trace_stable_run.json.gz 14.9 MB Download

What was the difference in scores between #12 and #18?

#18 was close to expected in the high 6000's (close to 7000) and 12 was lower.

Just ran an octane on today's Dev update on Minnie and gotten 6770,6384 and 6588.

Starting to feel like I'm taking my car to a mechanic and it won't make they weird noise when looked at... 

This was consistently bad weeks ago (high 4000's & low 5000's) and now seems a little better.

Note that my cyan benchmarks are still below 6000, and they should be above 7000.

Odds Bodkins, Dept:

staying abreast of this thread, I broke out my trusty, and old, Acer C-710.  It updated to this Dev:53.0.2785.29 (Official Build) dev (64-bit).  It was on an earlier Dev M53.

Back at Dev 51, I noticed it dropped in Octane from >8000, to under [sometimes well under], 7000.  Along with this  drop, came a loss on New Tab Page of the mini-bar, upper right [Gmail and the 4 other tokens]; my start page is NTP.

After today's update, the mini-bar is back, and Octane is 8300.  
So: the poor performance is not tied to the Flip and Arc++ [I am a TT for that device].

De minimis, the slowdown is tied to a failure to launch the mini-bar, and the associated gummy stuff that goes along with the failure to launch it.  

I have a cyan device. I'll make the comparison between 51 and 53.

This is quite confusing. I ran the benchmarks on M51 and M53 on a cyan device several times. Most times M53 will score higher than M51! [attached screenshots]

I also recorded a perf for all runs. For the lower-scored M51 run, the perf report shows a high number of samples in V8 GC functions, whereas not so much on M53. [attached reports].

How accurate these benchmarks are? It seems it all depends on how V8 will behave, and what the user and the machine are doing at the time of the run.

Earlier while looking at willg76x's M52 and M53 traces, I couldn't see much differences in regards to V8's GC samples. It could be a difference in how V8 caches and executes the code?
V8 experts need to chime in.

Deleted: cyan-m53-final.png 388 KB

	cyan-m53-final.png 388 KB View Download

Deleted: cyan-m51-final.png 400 KB

	cyan-m51-final.png 400 KB View Download

Deleted: Octane-perf-reports.zip 1.9 MB

Octane-perf-reports.zip 1.9 MB Download

Based on the lack of any other directions, I went ahead and did a Recovery on my R11

Octane scores:

51 stable right after the Recovery - 7626
52 stable after OTA update - 7695
These scores aer as expected for the device

moved to dev channel

53 dev before any Android apps installed - 5587
rebooted and ran the benchmark again in Guest mode, just to be sure - 5716

I will run any other tests/logs that might help.


So the problem is still somehow associated with 53 dev on my R11.

That's a pretty low score!
Ok, let's try to look at various things:
- Please go to chrome://version and get me all the values under "Variations", if you have any. Also get me the exact version and platform numbers.
- Run "ls -la /tmp/" and paste the output here.
- Hopefully you are on dev mode. Do the following:
  - echo -1 > /proc/sys/kernel/perf_event_paranoid 
  - echo 0 > /proc/sys/kernel/kptr_restrict
  - cd /home/chronos/u-<user hash>/Downloads/
  - Run "perf record -a --". Start running the benchmark immediately. Ctrl+C out of the "perf record" once the benchmark ends.
  - The previous step will result in a "perf.data" file in your Downloads folder. Please share it here. 

Preferably do this on a cyan device that shows a low score on M53.

First item:


Google Inc.
Copyright 2016 Google Inc. All rights reserved.
Google Chrome	53.0.2785.29 (Official Build) dev (64-bit)
Revision	d1d264988e8e3e14ac8cb2eea35746f2b96e59d1-refs/branch-heads/2785@{#349}
Platform	8530.30.0 (Official Build) dev-channel cyan
ARC	3091973
Blink	537.36 (@d1d264988e8e3e14ac8cb2eea35746f2b96e59d1)
JavaScript	V8 5.3.332.22
Flash	22.0.0.209-r1
User Agent	Mozilla/5.0 (X11; CrOS x86_64 8530.30.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.29 Safari/537.36
Command Line	/opt/google/chrome/chrome --ppapi-flash-path=/opt/google/chrome/pepper/libpepflashplayer.so --ppapi-flash-version=22.0.0.209-r1 --ppapi-flash-args=enable_hw_video_decode=1 --ui-prioritize-in-gpu-process --use-gl=egl --gpu-sandbox-failures-fatal=yes --enable-arc --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --max-unused-resource-memory-usage-percentage=5 --login-profile=user --has-chromeos-keyboard --enable-touchview --ash-enable-power-button-quick-lock --enable-centered-app-list --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/oem_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/oem_small.jpg --default-wallpaper-is-oem --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --enable-prefixed-encrypted-media --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --vmodule=screen_locker=2,webui_screen_locker=2,lock_state_controller=2,webui_login_view=2,power_button_observer=2,*ui/display/chromeos*=1,*ash/display*=1,*ui/ozone*=1,*zygote*=1,*plugin*=2,*chromeos/login/*=1,*arc/*=1 --login-manager
Executable Path	/opt/google/chrome/chrome
Profile Path	/home/chronos/u-bc1e945878ffca80caea410da6ed10e888f16851
Variations	6a89113b-e4b26eb4
16e0dd70-3f4a17df
31101bd6-3f4a17df
4a449931-f23d1dea
6345b824-3f4a17df
8739b412-4f7edc42
4a0cbfc9-ca7d8d80
7c1bc906-f55a7974
ba3f87da-92cc81ec
f049a919-3f4a17df
31362330-ca7d8d80
290c251-3f4a17df
f15c1c09-ca7d8d80
811bc6bc-d93a0620
dd4da2fc-ca7d8d80
43d0dd1e-f23d1dea
2e109477-bcf405c8
6d340565-ca7d8d80
9e5c75f1-f5d7252d
6488ba84-3f4a17df
c3ad0f6b-ca7d8d80
6b121ae7-f23d1dea
f5dd6118-2f5721af
f79cb77b-3f4a17df
b7786474-64e7d9a
74df3f1-f23d1dea
7382e39a-3f4a17df
868bda90-3f4a17df
4ea303a6-249c509
d5b671a5-f23d1dea
fe9bec35-80f9a33e
9736de91-3f4a17df
dbffab5d-ca7d8d80
de03e059-3f4a17df
30e679f-3f4a17df
867c4c68-3f4a17df
3ac60855-486e2a9c
f296190c-3a443afd
4442aae2-a90023b1
ed1d377-e1cc0f14
75f0f0a0-d7f6b13c
e2b18481-6754d7b7
e7e71889-e1cc0f14
fe05be5f-4001c964
61b920c1-e2a9a88d
46567c16-f23d1dea
cd9fec2f-786eec3
828a5926-d8f52f32

I am not in dev mode. If there is no other option, I will do that.

I don't think you will be able to run perf in non-dev mode.

Dev mode enabled. Octane benchmark is now 7128!

No Variations are shown when I do chrome://version.

Google Inc.
Copyright 2016 Google Inc. All rights reserved.
Google Chrome	53.0.2785.29 (Official Build) dev (64-bit)
Revision	d1d264988e8e3e14ac8cb2eea35746f2b96e59d1-refs/branch-heads/2785@{#349}
Platform	8530.30.0 (Official Build) dev-channel cyan
ARC	3091973
Blink	537.36 (@d1d264988e8e3e14ac8cb2eea35746f2b96e59d1)
JavaScript	V8 5.3.332.22
Flash	22.0.0.209-r1
User Agent	Mozilla/5.0 (X11; CrOS x86_64 8530.30.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.29 Safari/537.36
Command Line	/opt/google/chrome/chrome --ppapi-flash-path=/opt/google/chrome/pepper/libpepflashplayer.so --ppapi-flash-version=22.0.0.209-r1 --ppapi-flash-args=enable_hw_video_decode=1 --ui-prioritize-in-gpu-process --use-gl=egl --gpu-sandbox-failures-fatal=yes --enable-arc --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --max-unused-resource-memory-usage-percentage=5 --system-developer-mode --login-profile=user --has-chromeos-keyboard --enable-touchview --ash-enable-power-button-quick-lock --enable-centered-app-list --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/oem_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/oem_small.jpg --default-wallpaper-is-oem --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --enable-prefixed-encrypted-media --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --vmodule=screen_locker=2,webui_screen_locker=2,lock_state_controller=2,webui_login_view=2,power_button_observer=2,*ui/display/chromeos*=1,*ash/display*=1,*ui/ozone*=1,*zygote*=1,*plugin*=2,*chromeos/login/*=1,*arc/*=1 --login-manager --first-exec-after-boot --silent-launch
Executable Path	/opt/google/chrome/chrome
Profile Path	/home/chronos/u-74fc0b94ed2f1b6ffa89e4c9e029f06cb094a178

Next, ls -la /tmp/ ---

chronos@localhost / $ ls -la /tmp/
total 144
drwxrwxrwt.  3 root    root     880 Jul 28 17:56 .
drwxr-xr-x. 21 root    root    4096 Jul 26 03:31 ..
drwx------.  2 chronos chronos   80 Jul 28 17:47 .com.google.Chrome.WCXhiF
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-boot-complete
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-chrome-exec
-rw-r--r--.  1 chronos chronos    0 Jul 28 17:49 disk-chrome-first-render
-rw-r--r--.  1 chronos chronos    0 Jul 28 17:47 disk-chrome-main
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-lockbox-cache-end
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-lockbox-cache-start
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-login-prompt-visible
-rw-r--r--.  1 chronos chronos    0 Jul 28 17:49 disk-login-success
-rw-r--r--.  1 chronos chronos    0 Jul 28 17:48 disk-login-wait-for-signin-state-initialize
-rw-r--r--.  1 root    root     198 Jul 28 17:48 disk-network-wifi-association
-rw-r--r--.  1 root    root      99 Jul 28 17:48 disk-network-wifi-configuration
-rw-r--r--.  1 root    root      99 Jul 28 17:48 disk-network-wifi-failure
-rw-r--r--.  1 root    root      99 Jul 28 17:48 disk-network-wifi-idle
-rw-r--r--.  1 root    root      99 Jul 28 17:48 disk-network-wifi-online
-rw-r--r--.  1 root    root      99 Jul 28 17:48 disk-network-wifi-ready
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-post-startup
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-pre-startup
-rw-r--r--.  1 root    root      99 Jul 28 17:47 disk-shill-start
-rw-r--r--.  1 root    root       6 Jul 28 17:47 firmware-boot-time
-rw-r--r--.  1 root    root    1377 Jul 28 17:47 mount-encrypted.log
-rw-------.  1 root    root     146 Jul 28 17:49 .org.chromium.Chromium.LgDWQS
prw-------.  1 dlm     root       0 Jul 28 17:47 PmMessagesPort_in
prw-------.  1 dlm     root       0 Jul 28 17:47 PmMessagesPort_out
-rw-r--r--.  1 root    root      12 Jul 28 17:47 uptime-boot-complete
-rw-r--r--.  1 root    root      11 Jul 28 17:47 uptime-chrome-exec
-rw-r--r--.  1 chronos chronos   14 Jul 28 17:49 uptime-chrome-first-render
-rw-r--r--.  1 chronos chronos   11 Jul 28 17:47 uptime-chrome-main
-rw-r--r--.  1 root    root      11 Jul 28 17:47 uptime-lockbox-cache-end
-rw-r--r--.  1 root    root      11 Jul 28 17:47 uptime-lockbox-cache-start
-rw-r--r--.  1 root    root      12 Jul 28 17:47 uptime-login-prompt-visible
-rw-r--r--.  1 chronos chronos   14 Jul 28 17:49 uptime-login-success
-rw-r--r--.  1 chronos chronos   13 Jul 28 17:48 uptime-login-wait-for-signin-state-initialize
-rw-r--r--.  1 root    root      26 Jul 28 17:48 uptime-network-wifi-association
-rw-r--r--.  1 root    root      13 Jul 28 17:48 uptime-network-wifi-configuration
-rw-r--r--.  1 root    root      13 Jul 28 17:48 uptime-network-wifi-failure
-rw-r--r--.  1 root    root      13 Jul 28 17:48 uptime-network-wifi-idle
-rw-r--r--.  1 root    root      13 Jul 28 17:48 uptime-network-wifi-online
-rw-r--r--.  1 root    root      13 Jul 28 17:48 uptime-network-wifi-ready
-rw-r--r--.  1 root    root      10 Jul 28 17:47 uptime-post-startup
-rw-r--r--.  1 root    root      10 Jul 28 17:47 uptime-pre-startup
-rw-r--r--.  1 root    root      11 Jul 28 17:47 uptime-shill-start
chronos@localhost / $ 

Error at the next step: Permission denied

chronos@localhost / $ echo -1 > /proc/sys/kernel/perf_event_paranoid
bash: /proc/sys/kernel/perf_event_paranoid: Permission denied

Since the benchmark is close to where it should be, do we need to continue? Seems like the Variations may have been the issue?

Deleted: Screenshot 2016-07-28 at 5.54.41 PM.png 64.9 KB

	Screenshot 2016-07-28 at 5.54.41 PM.png 64.9 KB View Download

Yes, the variations seem to be the issue, but we need to determine which one. I was using a test image when I ran my test. Test images get no variations.

Please do the following:
- Reboot your machine. Now you should get some variations. Confirm by going to chrome://version, and paste them again here.
- Sorry I missed telling you that you need to run "sudo su" before running the commands that require root privileges. So please do that, and re do the experiment. You can skip the /tmp/ directory step.

Google Inc.
Copyright 2016 Google Inc. All rights reserved.
Google Chrome	53.0.2785.29 (Official Build) dev (64-bit)
Revision	d1d264988e8e3e14ac8cb2eea35746f2b96e59d1-refs/branch-heads/2785@{#349}
Platform	8530.30.0 (Official Build) dev-channel cyan
ARC	3091973
Blink	537.36 (@d1d264988e8e3e14ac8cb2eea35746f2b96e59d1)
JavaScript	V8 5.3.332.22
Flash	22.0.0.209-r1
User Agent	Mozilla/5.0 (X11; CrOS x86_64 8530.30.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.29 Safari/537.36
Command Line	/opt/google/chrome/chrome --ppapi-flash-path=/opt/google/chrome/pepper/libpepflashplayer.so --ppapi-flash-version=22.0.0.209-r1 --ppapi-flash-args=enable_hw_video_decode=1 --ui-prioritize-in-gpu-process --use-gl=egl --gpu-sandbox-failures-fatal=yes --enable-arc --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --max-unused-resource-memory-usage-percentage=5 --system-developer-mode --login-profile=user --has-chromeos-keyboard --enable-touchview --ash-enable-power-button-quick-lock --enable-centered-app-list --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/oem_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/oem_small.jpg --default-wallpaper-is-oem --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --enable-prefixed-encrypted-media --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --vmodule=screen_locker=2,webui_screen_locker=2,lock_state_controller=2,webui_login_view=2,power_button_observer=2,*ui/display/chromeos*=1,*ash/display*=1,*ui/ozone*=1,*zygote*=1,*plugin*=2,*chromeos/login/*=1,*arc/*=1 --login-manager --first-exec-after-boot
Executable Path	/opt/google/chrome/chrome
Profile Path	/home/chronos/u-74fc0b94ed2f1b6ffa89e4c9e029f06cb094a178
Variations	6a89113b-e4b26eb4
16e0dd70-3f4a17df
4a449931-3f4a17df
6345b824-3f4a17df
8739b412-4f7edc42
4a0cbfc9-ca7d8d80
7c1bc906-f55a7974
ba3f87da-a6404135
f049a919-3f4a17df
31362330-ca7d8d80
290c251-3f4a17df
f15c1c09-ca7d8d80
811bc6bc-d93a0620
dd4da2fc-ca7d8d80
43d0dd1e-3f4a17df
2e109477-e2e291f1
6d340565-ca7d8d80
9e5c75f1-78bc645e
6488ba84-3f4a17df
c3ad0f6b-ca7d8d80
6b121ae7-3f4a17df
f79cb77b-3f4a17df
b7786474-64e7d9a
74df3f1-3f4a17df
7382e39a-3f4a17df
868bda90-f23d1dea
4ea303a6-3d47f4f4
d5b671a5-3d47f4f4
fe9bec35-186f5907
9736de91-ca7d8d80
de03e059-3f4a17df
30e679f-3f4a17df
867c4c68-3f4a17df
3ac60855-486e2a9c
f296190c-17e37270
4442aae2-7158671e
ed1d377-e1cc0f14
75f0f0a0-4ad60575
e2b18481-6bdfffe7
e7e71889-e1cc0f14
fe05be5f-486e2a9c
61b920c1-e2a9a88d
46567c16-f23d1dea
cd9fec2f-786eec3
828a5926-3d47f4f4

n00b issues - took me a while, sorry

No worries. While you're running your experiment, I'm also running mine. I got a much lower score this time (6040) with a non-test image and lots of variations enabled. 

The perf.data file has a problem. It's too big to upload (40 MB) and won't ZIP - the zip is only 118 bytes. Won't upload to Drive, either.

You can't copy it to drive from the "Files" app?

I get an error "File unreadable"

Oh, yes, sorry, the file was written while you were root.

> sudo su
> chown chronos perf.data

Here you go.

Deleted: perf.zip 4.6 MB

perf.zip 4.6 MB Download

Awesome. Thanks!

Jim's perf report is pretty similar to mine, so I will use my test data.

Here are my enabled experiments:
AppBannerTriggering-Aggressive
AutofillFieldMetadata-Enabled
BrowserHangFixesExperiment-Enabled
ChildAccountDetection-Enabled
ChromeOSMemoryPressureHandling-conservative
ChromeOSWideProfilingCollection-Default
ClientSideDetectionModel-Model2
DisallowFetchForDocWrittenScriptsInMainFrame-DocumentWriteEvaluatorGroup
EnableMediaRouter-Enabled
ExtensionDeveloperModeWarning-Default
FontCacheScaling-Enabled
GFE-Default
GoogleNowExtension-Enable
LocalNTPSuggestionsService-Default
NetworkTimeQueries-NetworkTimeQueriesDisabled
NewMediaPlaybackUi-Default
NonValidatingReloadOnNormalReload-Disabled
OmniboxBundledExperimentV1-HQPAllowOverlappingMatchesControl_Dev
OptimizeLoadingIPCForSmallResources-Enabled
PaintOptimizations-Default
ParseHTMLOnMainThread-Enabled
PasswordGeneration-Enabled
PasswordManagerSettingsMigration-Disable
PluginPowerSaverTiny-Enabled
PointerEvent-Enabled
PreconnectMore-Enabled
QUIC-EnabledAdaptiveTimeLossDetection
RenderingPipelineThrottling-Default
SSLPostQuantum-enabled
SafeBrowsingIncidentReportingService-Enabled
SafeBrowsingUpdateFrequency-Default
SafeBrowsingV4LocalDatabaseManagerEnabled-Control
SchedulerExpensiveTaskBlocking-Enabled
StrictSecureCookies-Enabled
UMA-Population-Restrict-normal
UMA-Uniformity-Trial-1-Percent-group_81
UMA-Uniformity-Trial-10-Percent-group_01
UMA-Uniformity-Trial-100-Percent-group_01
UMA-Uniformity-Trial-20-Percent-group_02
UMA-Uniformity-Trial-5-Percent-group_17
UMA-Uniformity-Trial-50-Percent-group_01
V8CacheStrategiesForCacheStorage-normal
V8Ignition-Ignition_Lazy
V8_ES2015_TailCalls-Enabled
WeakMemoryCache-WeakMemoryCache_Control
WebFontsInterventionV2-Enabled-slow2g
use-new-media-cache-Disabled


Top samples are as follow:
- Browser Process:
  - User space from chrome:
     - device_event_log::DeviceEventLogImpl::AddLogEntry(device_event_log::DeviceEventLogImpl::LogEntry const&)    
     - logging::VlogInfo::GetVlogLevel(base::BasicStringPiece<std::string> const&) const
     - cc::AnimationTimeline::PushAttachedPlayersToImplThread(cc::AnimationTimeline*) const 
     - cc::AnimationHost::HasAnyAnimationTargetingProperty(cc::ElementId, cc::TargetProperty::Type) const
     - cc::DisplayItemList::ApproximateMemoryUsage() const
     - tracked_objects::ThreadData::TallyRunOnNamedThreadIfTracking(base::TrackingInfo const&, tracked_objects::TaskStopwatch const&)  
     - cc::ResourceProvider::ReceiveFromChild(int, std::vector<cc::TransferableResource, std::allocator<cc::TransferableResource> > const&)
  - Kernel:
     - smaps_pte_entry.isra.23 
     - copy_page_rep
     - native_apic_mem_read
     - unmap_single_vma
     - get_page
     - avc_lookup 
     - do_raw_spin_lock
     
- Renderer Process Running the Benchmarks:
  - User space from chrome:
     - v8::internal::interpreter::BytecodeArrayWriter::EmitBytecode(v8::internal::interpreter::BytecodeNode const*)
     - v8::internal::StaticScavengeVisitor<(v8::internal::PromotionMode)0>::VisitPointer(v8::internal::Heap*, v8::internal::HeapObject*, v8::internal::Object**).isra.534
     - void v8::internal::RelocInfo::Visit<v8::internal::IncrementalMarkingMarkingVisitor>(v8::internal::Heap*) 
     - v8::internal::BinaryOpWithAllocationSiteStub::MajorKey() const 
  - Kernel:
     - clear_page_c_e 
     - page_fault 
     - get_page_from_freelist 
     - __alloc_pages_nodemask
     - do_raw_spin_lock
     
- Globally on the whole system:
  - User space from Chrome:
     - v8::internal::interpreter::BytecodeArrayWriter::EmitBytecode(v8::internal::interpreter::BytecodeNode const*)
     - logging::VlogInfo::GetVlogLevel(base::BasicStringPiece<std::string> const&) const
     - v8::internal::StaticScavengeVisitor<(v8::internal::PromotionMode)0>::VisitPointer(v8::internal::Heap*, v8::internal::HeapObject*, v8::internal::Object**).isra.534
     - void v8::internal::RelocInfo::Visit<v8::internal::IncrementalMarkingMarkingVisitor>(v8::internal::Heap*)
  - Kernel:
     - clear_page_c_e
     - page_fault
     - get_page_from_freelist 
     - smaps_pte_entry.isra.23
     - do_raw_spin_lock

+danakj@ Is there a recent regression in the cc? As you can see a lot of samples on the browser process are in the cc. Is there a related finch experiment from the ones on this DUT?

Looking at the global top samples from chrome, V8 is showing up a lot. A lot of page faults are present as well. possibly due to thrashing caused by GC? +semenzato@.
Can the following experiments be related?
V8CacheStrategiesForCacheStorage-normal
V8Ignition-Ignition_Lazy
V8_ES2015_TailCalls-Enabled

Deleted: Perf_Reports.zip 1.9 MB

Perf_Reports.zip 1.9 MB Download

I haven't seen anything else go by for cc. More samples could also mean we're making more frames (something is being animated in the browser ui?)

But +ajuma for cc animations

I'd suggest taking a trace before/after to see differences if you want to narrow down what behaviour changed.

+ross/danno the device in question runs with the v8ignition-lazy experiment

Comment #45 seems to verify the experiment was active on my Cyan.

I disabled the flag and Octane benchmark immediately increased to 7724.

I confirm setting the flag #enable-ignition to disabled causes the benchmarks score to rise to 7979 after it was 6028.

Should we keep this as a stable blocker? I assume this experiment was never promoted to stable, correct?

set flag to Disable
Octane in Incognito window; jumped from 6300 to 7100+. screenshot.

for the record:
Google Chrome	53.0.2785.29 (Official Build) dev (32-bit)
Revision	d1d264988e8e3e14ac8cb2eea35746f2b96e59d1-refs/branch-heads/2785@{#349}
Platform	8530.30.0 (Official Build) dev-channel veyron_minnie
ARC	3091973

and these Variations:
Variations	6a89113b-e4b26eb4
16e0dd70-3f4a17df
31101bd6-3f4a17df
4a449931-3f4a17df
6345b824-3f4a17df
8739b412-4f7edc42
4a0cbfc9-ca7d8d80
7c1bc906-f55a7974
ba3f87da-92cc81ec
f049a919-3f4a17df
31362330-ca7d8d80
290c251-3f4a17df
f15c1c09-ca7d8d80
811bc6bc-d93a0620
dd4da2fc-ca7d8d80
43d0dd1e-3f4a17df
2e109477-bcf405c8
6d340565-ca7d8d80
9e5c75f1-1b7dc08b
6488ba84-3f4a17df
c3ad0f6b-ca7d8d80
6b121ae7-3f4a17df
f79cb77b-3f4a17df
b7786474-64e7d9a
74df3f1-f23d1dea
7382e39a-3f4a17df
868bda90-3f4a17df
4ea303a6-408058f9
d5b671a5-3d47f4f4
fe9bec35-80f9a33e
9736de91-ca7d8d80
dbffab5d-ca7d8d80
de03e059-f23d1dea
30e679f-3f4a17df
867c4c68-3f4a17df
3ac60855-486e2a9c
f296190c-f433a0a7
4442aae2-e1cc0f14
ed1d377-e1cc0f14
75f0f0a0-a5822863
e2b18481-7158671e
e7e71889-e1cc0f14
fe05be5f-97e7f871
46567c16-f23d1dea
cd9fec2f-362fd33d
828a5926-d8f52f32

Deleted: Screenshot 2016-07-29 at 12.33.52 PM.png 161 KB

	Screenshot 2016-07-29 at 12.33.52 PM.png 161 KB View Download

I'm not sure what more can be done on the Chrome OS side.

rmcilroy@:
1: Please turn off the experiment ASAP.
2: Don't turn it back on until the performance regression is understood and fixed.
3: DEFINITELY don't promote to Stable.

Thanks.

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Apologies for the delay, I'm just back from vacation.

The Ignition interpreter does cause some regressions on peak benchmark performance (e.g. Octane). We are working on these issues now, however the Ignition interpreter will provide a number of advantages on real-world web pages which are not necessarily captured by a throughput based benchmark like Octane (e.g., reduced memory consumption and improved startup time).

To answer your specific comments:

> 1: Please turn off the experiment ASAP.
> 2: Don't turn it back on until the performance regression is understood and fixed.
> 3: DEFINITELY don't promote to Stable.

We want to keep this experiment on for Dev+Canary while we stabilize and optimize the Ignition interpreter. This experiment is important for collecting performance data and crash reports from the wild for this significant change to V8 as we make progress towards stable.

We are shipping to low-end Android devices (<=512MB of RAM) in m53 since the benefits of the reduced memory usage out-weigh the reduction in raw JS performance and lead to a better user-experience for real-world pages on these devices. However we aren't going to promote to Beta / Stable for any other platforms (including all other ChromeOS devices) until we have addressed the performance regressions.

As mentioned above, Canary/Dev users who need high Octane scores can explicitly disable the experiment in about://flags. 

Thank you for your reply. I'm going measure page load times and memory usage with it enabled and disabled.

I'll keep it enabled if the results seem favorable. After all, real world usage is more important than benchmarks.

rmcilroy@ thanks for the update. Improved memory consumption and startup time is worth quite a bit in tradeoff.

I definitely agree it makes sense to keep this on for Canary/Dev.  Any chance we could do a smaller percentage while working out the performance regressions?

Is there a list where I could keep in the loop on experiments that may lead to regressions in dev?  Our users tend to notice these things and I'd rather not have to have someone waste a few days tracing down the cause when it happens.

re #54: Thanks for that. Regarding page load time - the startup improvements haven't landed yet (hence only enabling it on low-end Android devices), they should come over the next couple of quarters.

re #55: 
hablich@ - any thoughts on reducing the percentage with Ignition enabled?

abodenha@: Unfortunately I don't know of a mailing list which shows experiments which could affect performance, I agree this would be useful. motek@, do you know of any such list?

You can set all finch experiments to default with --enable-benchmarking. This is especially advised when running performance analysis on Canary and Dev.

Can you please elaborate on "Our users tend to notice these things..."? Who are the users? I assume those are corp users with ChromeOS on their laptops? How do they notice it? By running a benchmark or in their day-to-day work. The latter would be interesting feedback for us!

Hi hablitch,

Albert is most likely there referring to Chromebook Central Top Contributors who are helping to test the Play Store feature in Dev (and now Beta). The poor Octane benchmark scores were noticed. That this might have not been reflected in the real-world experience has been noted and is being reassessed currently. I for one have not noticed a degradation in performance in real-world use so all might be well in this regard.

- Mike  

hablitch - as one of the testers who noticed a problem, it was not an issue with benchmarks that started this. 

I noticed significantly poor performance on an Android app that is familiar to me. I have it running on many devices and the performance on Minnie was significantly slower than cyan. Graphical features take 3x as long on minnie to refresh.

That started the search to determine WHY. Based on other benchmarks, we expected minnie to be  10-15% slower, not 50% slower. That led to a round of comparative tests and revealed some minnie and cyan devices to be running significantly lower benchmark scores than others.

Contact me directly if you need details.

Just to point out, the Ignition experiment (and V8 performance in general) shouldn't have any impact on Android apps run through the ChromeOS Play Store feature.

#60, thanks for the clarification. We do understand about Ignition, and think the entire investigation got a bit misdirected by the benchmark results.

Other than observing performance of random apps, we have no tools other than Octane to do comparative testing.

Looking at the individual tests in the Octane suite - do any test OpenGL or other graphics features that would possibly indicate influence from some other experiment?

re: OpenGL - no Octane doesn't don't test any graphics operations, only pure JavaScript code execution.

OK, so we are dealing with two distinct issues -
Degraded JavaScript performance (as observed in Octane benchmarks), and OpenGL performance (as observed in apps)

+kbr regarding OpenGL performance.

re #57: Chrome OS dev channel is available to the public. In this case (as jim pointed out in #59) it was our external trusted testers who noticed the octane numbers while investigating what turned out to be an unrelated issue.

Beyond the TCs there are a fair number of external users who shift to the dev channel for whatever reason.

Thanks for the tip on --enable-benchmarking.  Unfortunately, there's no way to change Chrome's command line flags in Chrome OS short of putting the device in developer mode and manually editing config files.

Enthusiast users are going to switch to developer channel and run benchmarks on their devices. It's just a thing people do when they care about the product we build.  I really want to make it easy for such people to help us make a better product.

Thanks.

Excellently put.

hablich@ would it be possible to expose --enable_benchmarking via chrome://flags?

"+1" for #c67

Good suggestion. isherman@ and asvitkine@ is it feasible to enable --enable_benchmarking in about:flags? I am aware that this looks like a chicken & egg problem :-).

Won't that interfere with finch tests if users become more aware of them and easily enable that flag though? 

I can picture the tech blog and reddit posts already: 

"How to opt out of being a guinea pig in the Canary/Dev channel of Chrome with one simple flag change"

Indeed, we do not want to make it easy for a tech blog or reddit post to encourage a large swath of users to disable all field trials, since our data would immediately become less representative, and remain that way indefinitely.

Honestly, I think that the fundamentals of this bug are "working as intended":  A trusted tester noticed a *real* performance regression, that affects some, but not all, users.  (At least some of the) ChromeOS team leadership were previously unaware of this issue.  There is now greater awareness.

I think the two big issues encountered here were (1) a majorly performance impacting change was under-communicated, and (2) it was hard to identify which experiment was responsible.

Regarding (1), all experiments already need to have launch bugs associated with them, and get approval prior to launch.  We can probably do more to increase awareness without just creating a new low signal-to-noise email list, but I'm not exactly sure what that would look like.  Would be good to brainstorm.

Regarding (2), I think we could, and should, provide some easier way for developers and enthusiastic testers to explore how field trials affect user state.  I don't think that exposing a single --enable-benchmarking hammer more broadly is the right way to go.  It's a very crude measure, and still doesn't help identify *which* experiment is at fault.  It is already a best practice for each field trial to have an associated entry under about:flags.  So, I'd be inclined to explore solutions that leverage that existing UI surface.

A particular challenge in this case was simply the sheer number of experiments that are running. 

My dev-channel non-corp enrolled chromebook currently shows 54 variations.  I find it hard to believe that any of those can yield meaningful results with that many other things active. It makes the process of isolating WHICH of those is causing a problem a nightmare, especially since there's no easy way to disable them in order to bisect.

@ 71 &  72;  we TC's always knew that users were little sandbox ed experiments, and we always marveled at how extensive the list of experiments was. Since I retired from teaching Design of Experiments during the time of my first Chromebook, I kept fingers crossed that you all knew how to screen for lurking variables and primary, secondary, and tertiary interactions. And that you knew how to block and randomize all of that. 
:) 

#72 - can we compare the variations list from two devices and be able to say that if the a particular string exists on two devices, then they are both running a particular experiment?

I am trying to understand if we can logically look at a group of devices, some of which have a problem, and then correlate to the variations list.

#77 Can you add any details about what was actually done to fix this issue? 

I would have assumed a wontfix due to a finch test. For example, I can reduce an octane score by being placed in by default (or enabling) chrome://flags/#enable-v8-future

Wontfix is more appropriate here.

That makes sense. Thanks.