What changed? I see this error in this shard:
https://chromium-swarm.appspot.com/task?id=3869b15e7dfdba10&refresh=10&show_raw=1

of the smoothness.top_25_smooth failures from https://sheriff-o-matic.appspot.com/chromium.perf :

(INFO) 2017-09-05 07:29:29,862 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 06a9b365003beeb1 shell 'rm -f /data/local/tmp/temp_file-879413e599343.png'
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 105, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 331, in RunStory
    self._current_page.Run(self)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 108, in Run
    shared_state.page_test.RunNavigateSteps(self, current_tab)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 152, in RunNavigateSteps
    page.RunNavigateSteps(action_runner)
  File "/b/swarming/w/ir/tools/perf/page_sets/top_pages.py", line 89, in RunNavigateSteps
    self.credentials_path)
  File "/b/swarming/w/ir/tools/perf/page_sets/login_helpers/google_login.py", line 69, in LoginGoogleAccount
    action_runner.ClickElement(selector=_EMAIL_NEXT_SELECTOR)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 309, in ClickElement
    selector=selector, text=text, element_function=element_function))
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 56, in _RunAction
    action.RunAction(self._tab)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/javascript_click.py", line 25, in RunAction
    element_function=self.element_function)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/actions/page_action.py", line 130, in EvaluateCallbackWithElement
    return tab.EvaluateJavaScript(code)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 168, in EvaluateJavaScript
    return self._inspector_backend.EvaluateJavaScript(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 237, in EvaluateJavaScript
    return self._EvaluateJavaScript(expression, context_id, timeout)
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 36, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 501, in _EvaluateJavaScript
    return self._runtime.Evaluate(expression, context_id, timeout)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 54, in Evaluate
    description=details.get('exception', {}).get('description'))
EvaluateException: UncaughtError:
Error: Cannot find element: using selector "#identifierNext,#gaia_firstform #next"
    at callback (<anonymous>:7:19)
    at <anonymous>:11:16
    at <anonymous>:12:9

Also: the smoothness.maps failures that started on the Android Nexus5 Perf bot in this build:
https://luci-milo.appspot.com/buildbot/chromium.perf/Android%20Nexus5%20Perf/224

are due to shards expiring because there isn't enough capacity.

Could the re-recording of some of the WPR archives caused the apparent mismatch between the test and the recorded page in https://bugs.chromium.org/p/chromium/issues/detail?id=762165#c1 ?

This is quite surprising. Did they start to fail since last Thursday or today?

I'll try to repro this locally. Will revert if confirmed to be a bad conversion.

Is the regression range by sheriff-o-matic accurate? If so, the regression started long after the conversion has been committed. Could it be one of the catapult roll?

Re #2: That benchmark's device was offline and was fixed earlier today in t/28507470. The latest build should start getting results in.

We are seeing the top_25_smooth failure on ChromeOS - https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/testDetails?testName=telemetry_Benchmarks.smoothness.top_25_smooth&suite=&daysBack=30&board=&architecture=&boardFamily=&buildConfig=&reason=&version=&milestone=63&dut=

On the following bots:
https://build.chromium.org/p/chromium.perf/builders/Mac%2010.11%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Mac%2010.12%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Mac%20Retina%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Mac%20Pro%2010.11%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Mac%20Air%2010.11%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Mac%20Mini%208GB%2010.12%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Win%2010%20High-DPI%20Perf?numbuilds=200
https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf?numbuilds=200
(probably more, I stopped looking after all of those)

The smoothness.top_25_smooth failures started when https://chromium-review.googlesource.com/c/chromium/src/+/546615 landed.

The failures are all google pages:
Unexpected Failures:
* smoothness.top_25_smooth/Docs (1 open document tab)
* smoothness.top_25_smooth/http://www.youtube.com
* smoothness.top_25_smooth/https://plus.google.com/110031535020051778989/posts
* smoothness.top_25_smooth/https://www.google.com/calendar/
* smoothness.top_25_smooth/https://www.google.com/search?q=cats&tbm=isch


CC-ing owner of CL.

https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_10-2017-09-12_10-19-1135701.png

Screenshot from the failed youtube test. Looks like it is indeed getting stuck trying to log in.

fkackr@: seems like we need to revert your CL

Reverting CL in https://chromium-review.googlesource.com/c/chromium/src/+/663946

 Issue 764547  has been merged into this issue.

The revert didn't go through cleanly :-(

Robert, can you fix pages whose login broken because of your CL? 

 Issue 764756  has been merged into this issue.

The smoothness.key_silk_cases seem to be an unrelated problem. Spun off into its own bug at  crbug.com/764825 

This seems to account for quite a bit of the redness on the waterfall right now...

Just looking at the last failure on each bot this accounts for (note, this is just me quickly going through all the bots to estimate impact of this bug. I'm not saying that all these failures are 100% caused by this, I just highly suspect that they are):

Nexus 5x: 3/6 failures
Nexus 5: 2/6 failures
Nexus 6: 3/7 failures
Nexus 7: 3/9 failures
A1: 3/10 failures
Nexus 5X webview: 2/5 failurs
Nexus 6 Webview: 2/5 failures

(problem with dromeao benchmarks causing high failure rates on windows)
win 10 high-dpi: 2/10 failures
win 10 perf: 2/10 failures
win 8 perf: 2/8 failures
win 7 perf: 2/8 failures
win 7 x64: 2/9 failures
win ATI: 2/8 failures
win intel: 2/8 failures
win nvidia: Exceptions on bot make it hard to tell.. estimated 2 benchmarks though

mac 10.11: 2/2 failures
mac 10.12: 1/1 failures
mac retina: 1/2 failures
mac pro 10.11: 2/3 failures
mac air 10.11: 1/5 failures
mac mini 8GB: 2/3 failures

Linux: Exceptions on the bot make it hard to tell.. estimated 2/3 failures

Sorry I was on a plane and haven't had a chance to look into this yet. It's the weekend here but I will try to have a look at this soon. Do we know why we're just now noticing that my July 1st patch broke the tests? Is it that we recently started running it again? Or was it just rolled in now?

Hmmm. I think the real cause was this cl:

https://chromium-review.googlesource.com/c/chromium/src/+/648252

which contains a link to your cl:
https://chromium-review.googlesource.com/c/chromium/src/+/546615

So.. I'm not sure if its your CL, or this other CL that was based off your CL.
CCing the author of that CL. Sorry that no one noticed this earlier.

Oh yes, that patch updated the auth call for YouTube without rerecording it which seems like it may have been the problem.

I will revert Walter's CL given that rasterize_and_record_micro now uses static HTML 

I'm re-recording these sites and will upload a patch soon.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc

commit 26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc
Author: Victor Miura <vmiura@chromium.org>
Date: Sat Sep 16 06:43:40 2017

telemetry: Re-record top_25 google sites, and fix login flow.

This fixes and re-enables loading of the following page stories:

* top_25_smooth/Docs (1 open document tab)
* top_25_smooth/http://www.youtube.com
* top_25_smooth/https://plus.google.com/110031535020051778989/posts
* top_25_smooth/https://www.google.com/calendar/
* top_25_smooth/https://www.google.com/search?q=cats&tbm=isch

TBR=nednguyen@google.com
BUG= 762165 

Change-Id: I4c39637b1c02a40653e2c11dcb3cca0ea6d34070
Reviewed-on: https://chromium-review.googlesource.com/670064
Reviewed-by: Victor Miura <vmiura@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#502496}
[modify] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/benchmarks/smoothness.py
[add] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/data/top_25_007.wprgo.sha1
[add] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/data/top_25_008.wprgo.sha1
[modify] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/data/top_25_009.wprgo.sha1
[add] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/data/top_25_010.wprgo.sha1
[modify] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/data/top_25_011.wprgo.sha1
[modify] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/data/top_25_smooth.json
[modify] https://crrev.com/26f31fe066aa26cbbd9cd2a1ea2cf32d4b52edfc/tools/perf/page_sets/login_helpers/google_login.py

 Issue 770907  has been merged into this issue.

I believe this can be closed?

 Issue 775634  has been merged into this issue.

I'm still seeing failures on Win 7 Perf and Win 7 x64 Perf:

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20x64%20Perf/builds/1391
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_7_x64_Perf%2F1391%2F%2B%2Frecipes%2Fsteps%2Fsmoothness.top_25_smooth_on__102b__GPU_on_Windows_on_Windows-2008ServerR2-SP1%2F0%2Fstdout

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20Perf?numbuilds=200
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20Perf/builds/1409
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_7_Perf%2F1409%2F%2B%2Frecipes%2Fsteps%2Fsmoothness.top_25_smooth_on__102b__GPU_on_Windows_on_Windows-2008ServerR2-SP1%2F0%2Fstdout

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b45d15ba8b6fa1e66d1aa24b636b44792a804e61

commit b45d15ba8b6fa1e66d1aa24b636b44792a804e61
Author: Charlie Andrews <charliea@chromium.org>
Date: Wed Nov 15 22:11:36 2017

Disable failing smoothness.top_25_smooth stories

TBR=rnephew@chromium.org, vmiura@chromium.org

Bug:  762165 , 736099
Change-Id: I8c0cfb765015a3efdc26ae4a548dd5164b4d483f
Reviewed-on: https://chromium-review.googlesource.com/769431
Commit-Queue: Charlie Andrews <charliea@chromium.org>
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Reviewed-by: rnephew <rnephew@chromium.org>
Cr-Commit-Position: refs/heads/master@{#516851}
[modify] https://crrev.com/b45d15ba8b6fa1e66d1aa24b636b44792a804e61/tools/perf/benchmarks/smoothness.py

Moving Infra>Client>Android -> Infra>Client>Chrome+OS=Android

The following pages have now been re-recorded, and the corresponding expectations removed:
rendering.desktop/google_calendar
rendering.desktop/google_plus
rendering.desktop/youtube
rendering.desktop/google_image_search
rendering.desktop/google_docs