I found that this is because smoothness.map was using the TBMv1 smoothness.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4efbba622fd20e67facea12c3b2679a16a3888db

commit 4efbba622fd20e67facea12c3b2679a16a3888db
Author: nednguyen <nednguyen@google.com>
Date: Mon Nov 13 21:20:29 2017

Make sure that smoothness.maps benchmark use the smoothness measurement

Previously, smoothness.maps benchmark uses TBMv1 measurement to collect the data
(unlike others smoothness tests which use smoothness measurement).

With https://chromium-review.googlesource.com/764536, measuring smoothness
only supported through smoothness measurement, which breaks smoothness.maps.

This CL fixes the breakage by switching smoothness.maps benchmark to use smoothness
measurement like others.


Bug:  784418 ,760553
Change-Id: Id423b5210f0741254ead967a266cf8610fc9a883
Reviewed-on: https://chromium-review.googlesource.com/766482
Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#516051}
[modify] https://crrev.com/4efbba622fd20e67facea12c3b2679a16a3888db/tools/perf/benchmarks/smoothness.py

I found that the benchmark runs much faster now, the code we have for Maps page is currently as follow:

  def RunNavigateSteps(self, action_runner):
    super(MapsPage, self).RunNavigateSteps(action_runner)
    action_runner.Wait(3)

  def RunPageInteractions(self, action_runner):
    with action_runner.CreateInteraction('MapAnimation'):
      action_runner.WaitForJavaScriptCondition(
        'window.testMetrics != undefined', timeout=120)


So basically we navigate to the map page, then wait for 3s, then start taking the measurement through creating 'MapAnimation' record, and wait for 'window.testMetrics' to be available.

In my local run, this leads to 'MapAnimation' taking only a few milliseconds. If I remove the 3s wait in RunNavigateSteps method, then it's taking 2s and contain enough frames for a meaningful measurement. Though there is some missing frames in the beginning of 'MapAnimation' probably due to JS resources still being loaded.


I am guessing that either the page is loading faster, or the animation is rendered faster, hence making 'window.testMetrics' available much faster than we anticipated.

The ideal fix to me would be changing the map test so that the animation can be controlled explicitly.

The Map Page code would then become:

  def RunNavigateSteps(self, action_runner):
    super(MapsPage, self).RunNavigateSteps(action_runner)
    action_runner.WaitForJavaScriptCondition('test.runner != undefined')

  def RunPageInteractions(self, action_runner):
     # kick off the animation run explicitly instead of relying on arbitrary timing
    action_runner.EvaluateJavascript('test.runner.start()')

    with action_runner.CreateInteraction('MapAnimation'):
      action_runner.WaitForJavaScriptCondition(
        'window.testMetrics != undefined', timeout=120)

Reassign this bug to Ken for deciding what should we do next with the map test. 

We can remove the "action_runner.Wait(3)" as shown in #3 for easy fix, but that would make the test less reliable.

I looked into this. If we build the right target (see cl/176724781) and edit config.js to remove the "nobudget=true" query arg, and if we can re-record that WPR, it should start working because the benchmark will take longer to run. There are two issues:

1) We have to temporarily hack the "http://map-test/performance.html" URL in https://cs.chromium.org/chromium/src/tools/perf/page_sets/maps.py to be http://localhost:8000/performance.html while recording the WPR.

2) Once that recording's done, we need to edit the wprgo archive to replace instances of "localhost:8000" with "map-test".

This was run into once before in Issue 757014 and https://github.com/catapult-project/catapult/issues/3787 was filed but never fixed. That feature has to be added to wprgo before the maps benchmark's WPR can be re-recorded.

Filed  Issue 788060 . Fixing this bug is blocked on that one.

Alternatively, if there were some way to tell record_wpr that it should map the hostname "map-test" to "localhost:8000" when making its requests, that would work, too. As it stands, running:
  tools/perf/record_wpr smoothness_maps --browser=system
hangs, because all the requests go to the non-existent host "map-test".

Note for my future reference: work-in-progress CL:
https://chromium-review.googlesource.com/786998

The wprgo hasn't been uploaded yet; it won't work until the remapping functionality is added for wprgo archives.

Note also: currently working with the Maps team to get a new version of their benchmark which will continue to have an 800x600 viewport (for best compatibility with Android devices), use nobudget=false during the recording so it runs for longer, and adds a start() method so we can control more precisely when the benchmark starts to run.

Urgently need the wprgo host rewriting functionality.

Got instructions from Ned on how to edit the wprgo archive by hand:

$ go get github.com/urfave/cli 
$ cd third_party/catapult/web_page_replay_go
$ set GOPATH=`pwd`
Then modify https://github.com/catapult-project/catapult/blob/master/web_page_replay_go/src/httparchive.go#L21 to just "webpagereplay", otherwise go run will complain about not being able to find "github.com/catapult-project/catapult/web_page_replay_go/src/webpagereplay"

Then run:
$ go run src/httparchive.go edit ~/projects/chromium/src/tools/perf/page_sets/data/maps_005.wprgo modified_map.wprgo 

Unfortunately there's a problem, which is that the re-recorded wprgo archive doesn't contain any of the requests to localhost, so editing the archive is fruitless.

Ned, can you please help me? I've sent you the Maps team's new benchmark and updated this CL:
https://chromium-review.googlesource.com/786998

with the code changes needed to work with the new benchmark.

Re-recording the WPR produced a tiny wprgo containing only a couple of safebrowsing requests to ssl.gstatic.com and none of the data requests that went to localhost.

Ken: the good news is we don't this WPRGO for static pages like this at all. I will make a CL for you shortly.

I explained to kbr@ how to switch the test to static local files. Reassign the bug to him.

Apologies for the delay, but https://chromium-review.googlesource.com/828541 is finally up for review with the Maps team's revised benchmark, and serving up the files statically instead of from a wprgo archive.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/077c8d9b539f8b2c344f6932fc6d52057a57cbf2

commit 077c8d9b539f8b2c344f6932fc6d52057a57cbf2
Author: Kenneth Russell <kbr@chromium.org>
Date: Sat Dec 16 02:52:14 2017

Revise Maps smoothness benchmark and correctness test.

Thanks to the Maps team for a new benchmark that supports manual
control and which can be checked in, and Ned from the Telemetry team
for illustrating how this would work, both the benchmark and
correctness test have been updated to be simpler and more robust.

The new benchmark required new pixel expectations to be sampled for
the various colors on the map.

The old wprgo archive's SHA1 and associated JSON file have been
deleted.

BUG= 784418 

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I662f85e3e8262957231266b8bd1988accca6a750
Reviewed-on: https://chromium-review.googlesource.com/828541
Reviewed-by: Daniel Cheng <dcheng@chromium.org>
Reviewed-by: Lei Zhang <thestig@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#524573}
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/PRESUBMIT.py
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/chrome/test/BUILD.gn
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/content/test/gpu/gpu_tests/maps_integration_test.py
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/content/test/gpu/gpu_tests/maps_pixel_expectations.json
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/checklicenses/checklicenses.py
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/benchmarks/smoothness.py
[delete] https://crrev.com/a3910459eec10cdff27cebeaca59a79446afb65c/tools/perf/page_sets/data/maps.json
[delete] https://crrev.com/a3910459eec10cdff27cebeaca59a79446afb65c/tools/perf/page_sets/data/maps_005.wprgo.sha1
[modify] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/page_sets/maps.py
[add] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/page_sets/maps_perf_test/config.js
[add] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/page_sets/maps_perf_test/load_dataset.sha1
[add] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/page_sets/maps_perf_test/performance.html
[add] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/page_sets/maps_perf_test/tracked.js
[add] https://crrev.com/077c8d9b539f8b2c344f6932fc6d52057a57cbf2/tools/perf/page_sets/maps_perf_test/worker.js

This should be fixed. Would appreciate help verifying.