All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=627113

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZynQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im9DaHJvbWl1bVBlcmYvd2luLXplbmJvb2svc21vb3RobmVzcy50b3BfMjVfc21vb3RoL2ZpcnN0X2dlc3R1cmVfc2Nyb2xsX3VwZGF0ZV9sYXRlbmN5L2h0dHBfX19uZXdzLnlhaG9vLmNvbV9yZWYMCxINU3RvcHBhZ2VBbGVydBjPxxgM,agxzfmNocm9tZXBlcmZyjwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImFDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLXdpbjctZ3B1LWF0aS9zbW9vdGhuZXNzLnRvcF8yNV9zbW9vdGgvZnJhbWVfdGltZXMvV2lraXBlZGlhICgxIHRhYilfcmVmDAsSDVN0b3BwYWdlQWxlcnQYg8YYDA,agxzfmNocm9tZXBlcmZyiAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50IlpDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLXdpbjEwL3Ntb290aG5lc3MudG9wXzI1X3Ntb290aC9mcmFtZV90aW1lcy9XaWtpcGVkaWEgKDEgdGFiKV9yZWYMCxINU3RvcHBhZ2VBbGVydBiCuBgM


Bot(s) for this bug's original alert(s):

chromium-rel-win10
chromium-rel-win7-gpu-ati
win-zenbook

I'm having a hard time figuring out what caused the ref builds to stop reporting data here.

The buildbot stdio has expired, which makes this tricky.

It looks like wikipedia isn't producing any results at all for the ref build, but it is producing results for the non-ref build?

I wonder why wikipedia would be failing, but only on the ref build...

Here's a trace for the non-ref build:
https://00e9e64bacd119245edca6ba1259402818fc9acf2c66c46f0b-apidata.googleusercontent.com/download/storage/v1_internal/b/chrome-telemetry-output/o/trace-file-id_11-2016-07-08_14-06-39-25973.html?qk=AD5uMEssRh5GA6uZ6mZ_9zKwqVUVX7oiAXtM88zsW6ePpF27WBS3bMEkkWjRiLLC174yicXdluIP_r6adItKHncX9gepR0ODwu9X-5eb1dZf3y5Sd_GTetkJVRbpRiuTUU1t9cQ_duPqp9-R7SuPIyzAlY3ONOIGMqAqUkNzVtUN6kvqChYFOeCP2Ce7Ua1fZpguyxejN16Jo5oqmfhjrqFbwfzqUaQKh820EKfCLl5yaRNgYSQ2UoyOzwgdtd2AFndlbxWeh__uDL0Osyxn9VyJSiQf9ZTCMNLQeTFQfLAYO0D0ZVP6RYxNRuo93DqnHiM62DZebbWImx9Gn5nm5zDblBZ_VjqEd4-7J5YK-kd1mog2mzmkxdIYUepdyCh5R9Wi_ENj5m45fhrliOAQSp1x96ajXELyng9xCpV0rLuXarz7SqSigjo22khOj3F90puGp9hrSCIpwv4gAqQWrjzoPR4eSbe1ghN4-96hucRtyGg6ah9W6nrRSJ9ggGo8dpIsQr0M5wscQ0q4_Sgy-fGGgQWshrE9-oDZCbzrDn-1WjsXUeQUV9aRMm8ksDxYz10rtP-ufSGp7TLhZLrC9nKOdwEuN-DklnjpY6FB4j98-iohRYB3fkaLe5xveaa85iT4ZYA-k805X4OYLR_HKPk5IyFfQGOn0YSx61iyhp41_ZFpL3XtPNqQ-N9Jjx_XEPGY9qfj0c56lzei2pB2cFGdOd6ZH7ihwDFJYimlrNpTVgA8n61jgaXxgvSq9f_Sn1rMRrlf8YxESYJFnmeZC8gsfZPCFauQ_gph5OapXfyGxlDUJtcYOi0GrrXZ9UkNpQPnMCYvpPMyDe6epv-sou0Vouj1sfvFloBvXGZEVbmrVk6Uy8BxOpEHVB-39asHANbKEzwLar1N

And a trace for the ref build:
https://00e9e64bac80f684f020b28185b420ca4dd01cb8495faa0bd7-apidata.googleusercontent.com/download/storage/v1_internal/b/chrome-telemetry-output/o/trace-file-id_11-2016-07-08_14-13-10-11602.html?qk=AD5uMEsrbpwtBnpUe7_in0rcI3vVVmnOsUJ0VGy2A_Dvc9N1hswKAI_blxw9QaTqPrfFWNZi8NghAVDqd90djIfLWRC894O7WRaij6iON4EG-DLwOWZ3v7H29RDt4TaDP_Fc5WS4EQlthIrimg0uhocsTfWV0eenWue0f-9bA1e1zTn579La0bnkMqKBNyPShMvlmEYI5VNkvdN0AvgqEierz1itL1TlwvEz93cLpBirF70tmWT_SMgC7JjLM1yOe3vbk_WydIp4mHi3fW7ltSQnW1mgU6Vu14TS3F5gI-BZeKsRJtseXNK4HDh2jKzz-nLl3pjDm1-1aYcwt6ryXMRXBR9eT3QwQHRh_eBAbe4LhGt-iP2dqk8uPDss8gEtkhpBwRYqdxd-cvcSrEwtDSmcz6Z8TG6WHoT7gD0XxReTHI8bs4RIRzFgwtbEhIiCcORbnptieWNYE0FiqcIx8OopmS96FCftPOKftPJ5uj17ERDIQ34TiLQKi1wvSLUspCkoQVVtCsnUT5wFDP7UqFkIgbh7u_zPwHT3AnEH-m8oaTTALd_t6GqA3II4mNfuzUj8FhcZxGktJeooafjhbl5PnC2SIh_WLePOBYUz0AnbsOAtXlzXfRegH2ugGlJkSxLox8B5IlMRL6qPtZziN_liULmuLJvM6DgdxnbjGhMex85PKQPhF5jiEU5qurpLDaLsK8Ol0Xxn2r71TZDkViCm_zXI1aBxmS0etarfdaHgUX50INUCMJcVWnAHH2RGZtfoZMYJpTAjXLGU4aufASrOWrwohPjFBHspmeZuuRBKthNiFrpnEaj3v54_1wTvUkUvHD7alAfRyfbCijBnuQoD0TQTWBhjpcC52Ul8bkX-hDM0C6Jy7l765WpnTt0abApsP_2ys8Ff

The ref build trace looks completely broken.

Annie, any thoughts on next steps here? 

What I'm seeing:

Data coming back in for Wikipedia on win10 and win7-dual-ati:
https://chromeperf.appspot.com/report?sid=74599e89dd763335cdf6d1697d1d4836c0758e76a389b09a4fe1331ed58b903f&rev=400386
https://chromeperf.appspot.com/report?sid=2d6df9e355dd46cd149a1648134ff47b4de848d48f1c0c33c31cad7e674b1866&rev=400386

Broken ref build for news.yahoo.com on zenbook:
https://chromeperf.appspot.com/report?sid=1379e4aa2273c44b79f6a5b223dc1938b015b7af88f63d8742a8de2c290feb13

Looking at the log of a recent build on zenbook, news.yahoo.com looks the same as the other sites, but there is a warning on all of them for no importer found:
https://build.chromium.org/p/chromium.perf/builders/Win%20Zenbook%20Perf%20%283%29/builds/2115/steps/smoothness.top_25_smooth.reference/logs/stdio

[ RUN      ] http://news.yahoo.com
(INFO) 2016-07-08 17:52:17,630 cache_temperature.EnsurePageCacheTemperature:55  PageCacheTemperature: any
(INFO) 2016-07-08 17:52:18,203 chrome_tracing_agent._CreateTraceConfigFile:248  Trace config file string: {"trace_config": {"included_categories": ["webkit.console", "benchmark", "blink.console", "trace_event_overhead"], "record_mode": "record-as-much-as-possible"}}
(INFO) 2016-07-08 17:52:18,223 tracing_backend.StartTracing:131  Start Tracing Request: {'params': {'transferMode': 'ReturnAsStream', 'options': 'record-as-much-as-possible', 'categories': 'benchmark,blink.console,trace_event_overhead,webkit.console'}, 'method': 'Tracing.start'}
(INFO) 2016-07-08 17:52:24,438 trace._GetTempFileHandle:65  Trace (TraceDataPart("traceEvents")) of size 1315136 bytes saved.
(INFO) 2016-07-08 17:52:24,440 trace._GetTempFileHandle:65  Trace (TraceDataPart("tabIds")) of size 40 bytes saved.
(INFO) 2016-07-08 17:52:24,444 trace._GetTempFileHandle:65  Trace (TraceDataPart("telemetry")) of size 946 bytes saved.
c:\users\chrome~1.lab\appdata\local\temp\tmp56j5nm.html
(WARNING) 2016-07-08 17:52:33,563 model._CreateImporters:275  No importer found for TraceDataPart("telemetry")

Ned, do you know what that error means?

In the chartjson for the ref build, we get:
        "http://news.yahoo.com": {
          "description": "First gesture scroll update latency measures the time it takes to process the very first gesture scroll update input event. The first scroll gesture can often get delayed by work related to page loading.", 
          "important": true, 
          "improvement_direction": "down", 
          "name": "first_gesture_scroll_update_latency", 
          "none_value_reason": "Merging values containing a None value results in a None value.", 
          "page_id": 17, 
          "std": null, 
          "type": "list_of_scalar_values", 
          "units": "ms", 
          "values": null
        }, 

That "No importer found for TraceDataPart("telemetry")" basically mean we don't implement a python importer for telemetry trace, which can be safely ignore because we are pushing everything to TBM2

As you pointed out, the page that which blows up the whole thing is news.yahoo.com. Its first_gesture_scroll_update_latency is null due to some reason.

Trace of news.yahoo.com:
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_17-2016-07-08_17-55-04-60162.html

There are no gesture scroll update latency slices in that trace, but we are seeing a bunch of touch move latency slices.

Win 7 Intel is missing data for wikipedia:
        "Wikipedia (1 tab)": {
          "description": "First gesture scroll update latency measures the time it takes to process the very first gesture scroll update input event. The first scroll gesture can often get delayed by work related to page loading.", 
          "important": true, 
          "improvement_direction": "down", 
          "name": "first_gesture_scroll_update_latency", 
          "none_value_reason": "Merging values containing a None value results in a None value.", 
          "page_id": 11, 
          "std": null, 
          "type": "list_of_scalar_values", 
          "units": "ms", 
          "values": null
        }, 

but has data for yahoo

        "http://news.yahoo.com": {
          "description": "First gesture scroll update latency measures the time it takes to process the very first gesture scroll update input event. The first scroll gesture can often get delayed by work related to page loading.", 
          "important": true, 
          "improvement_direction": "down", 
          "name": "first_gesture_scroll_update_latency", 
          "page_id": 17, 
          "std": 0.0, 
          "type": "list_of_scalar_values", 
          "units": "ms", 
          "values": [
            11.45
          ]
        }, 

This is the wikipedia trace: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_11-2016-07-08_17-54-51-63667.html

It looks very wrong. Maybe the page no longer load & 404 page is showing? :-/

 Issue 627432  has been merged into this issue.

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

On ChromiumPerf/chromium-rel-win7-gpu-intel/smoothness.top_25_smooth / frame_times / Blogger, we stopped getting ref data at the same time as we started getting non-ref data.

That seems kinda suspicious.

 Issue 627214  has been merged into this issue.

I suspect this could be an issue with the WPR recording or telemetry change.

Most suspicious thing is this catapult roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/9f8a0a8c0d45..71bab5766221

Nothing is really jumping out at me though.

Ned, do you think there's anything in the catapult roll that could be causing this?

No. I don't think the catapult roll is causing this. The best way to diagnose this is to run the benchmark locally on win-zenbook & see what happening on the page.

I don't have access to that device - could someone who does take this?

Can we get an owner for this? Any volunteers who have access to win-zenbook?

Assigning to sullivan@ to delegate for the moment.

This has recovered.