This appears to be failing on android N5 and N7v2 bots only (both common and memory benchmarks).

Screenshots:
N5: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_47-2017-06-21_05-42-0862876.png
N7v2: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_47-2017-06-21_05-36-3364824.png

First failure on N5 bot was:
https://luci-milo.appspot.com/buildbot/chromium.perf/Android%20Nexus5%20Perf/112

With CL range:
http://test-results.appspot.com/revision_range?start=477961&end=478050

Will try a return code bisect.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8976174610306875248

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c77c4ac96a3363dd902f65a738de74b55ed2593

commit 6c77c4ac96a3363dd902f65a738de74b55ed2593
Author: Juan A. Navarro Perez <perezju@chromium.org>
Date: Wed Jun 21 14:22:22 2017

[system health] Disable browse:chrome:newtab story

Story is causing a critical failure, preventing other stories in the
suite to run.

NOTRY=True
TBR=nednguyen@google.com

Bug:  735405 
Change-Id: Ic46b1815ae8599927b9c2366f7fde6b51a0595b0
Reviewed-on: https://chromium-review.googlesource.com/543116
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Cr-Commit-Position: refs/heads/master@{#481198}
[modify] https://crrev.com/6c77c4ac96a3363dd902f65a738de74b55ed2593/tools/perf/page_sets/system_health/expectations.py

=== Auto-CCing suspected CL author twellington@google.com ===

Hi twellington@google.com, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : Theresa Wellington
  Commit : ce2be9d4ba31aa44b6d18716746647537e535546
  Date   : Thu Jun 08 15:57:02 2017
  Subject: [Android] Use absolutely positioned PopupWindow for app menu

Bisect Details
  Configuration: android_nexus5_perf_bisect
  Benchmark    : system_health.common_mobile
  Metric       : cpuTimeToFirstMeaningfulPaint_avg/browse_chrome/browse_chrome_newtab

Revision             Exit Code      N
chromium@477960      0 +- N/A       5      good
chromium@477972      0 +- N/A       5      good
chromium@477978      0 +- N/A       5      good
chromium@477979      0 +- N/A       5      good
chromium@477980      1 +- N/A       5      bad       <--
chromium@477981      1 +- N/A       5      bad
chromium@477983      1 +- N/A       5      bad
chromium@478005      1 +- N/A       5      bad
chromium@478050      1 +- N/A       5      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.chrome.newtab system_health.common_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8976174610306875248

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5221737114894336


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

The change in layout above appears to have broken the following bit of code on the story:

    app_ui.WaitForUiNode(resource_id='menu_button').Tap()
    app_ui.WaitForUiNode(resource_id='menu_item_text')  # <-- this fails

https://cs.chromium.org/chromium/src/tools/perf/page_sets/system_health/chrome_stories.py?rcl=863ce9d480be2fa787462ea8da9bfd9b49070047&l=76

That is, we tap on the app menu button and (I assume) wait for some "item text" to appear.

Ted, any idea why this only broke on some configurations and not others? Could you suggest a more robust way to test for the menu to be visible?

Actually, I'm not quite sure why we tap on the app menu at all; ssid could we just remove that bit at the end of the story?

I will remove the second line. We do not click the item just wait for it to appear.

Where was menu_item_text even defined?  I didn't see it removed in that patch, so I'm wondering how I would have diagnosed it.

I am trying to find what the issue is.

"menu_item_text" is the resource id of the TextView for items in the menu: https://cs.chromium.org/chromium/src/chrome/android/java/res/layout/menu_item.xml?q=menu_item_text&sq=package:chromium&dr=C&l=13

The menu contents didn't change. Rather than using a ListPopupWindow, which shows a PopupWindow at an anchor location, I changed the app menu to use a PopupWindow displayed at an absolute location.

Since the test was disabled, I think this is no longer a P0 (but still a priority to get fixed soon).

I just tried to run the benchmark locally and it seems to work! I took the apk from the buildbot and ran the benchmark.
I can't understand what caused the benchmark to fail since the UI ID of the element remains the same: "org.chromium.chrome:id/menu_item_text"

Juan, am I doing anything different? I will try to run on the bot.

What device type and build of android did you test on locally?

I am using Nexus 7 and ChromePublic.apk that was built by the failed build link in the first comment.

Try to flash your device to K (the bots use KOT49H); not sure why, but the internal bots running M did not reproduce the failure.

Hm makes sense. This fails in Kitkat.

The issue seems to be a bug in uiautomator where it misses the ID of the UI elements in some cases. The device monitor shows the right IDs, but uiautomator dump does not dump the ID.
I am fixing the story to look for the button text rather than the ID.
https://codereview.chromium.org/2961653002/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a2849be12be0a4b84f45dc9185019f93b1c317b4

commit a2849be12be0a4b84f45dc9185019f93b1c317b4
Author: ssid <ssid@chromium.org>
Date: Mon Jun 26 19:43:03 2017

Fix browse:chrome:newtab story

The uiautomator does not give the right element IDs for new popup
window. So, use the text to search for the button instead.

BUG= 735405 

Review-Url: https://codereview.chromium.org/2961653002
Cr-Commit-Position: refs/heads/master@{#482362}

[modify] https://crrev.com/a2849be12be0a4b84f45dc9185019f93b1c317b4/tools/perf/page_sets/system_health/chrome_stories.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4d09478c03cfc9c735b46af22aad84589e23911b

commit 4d09478c03cfc9c735b46af22aad84589e23911b
Author: siddhartha sivakumar <ssid@chromium.org>
Date: Mon Jun 26 21:53:51 2017

Revert "[system health] Disable browse:chrome:newtab story"

This reverts commit 6c77c4ac96a3363dd902f65a738de74b55ed2593.

Reason for revert: This issue was fixed in codereview.chromium.org/2961653002

Original change's description:
> [system health] Disable browse:chrome:newtab story
> 
> Story is causing a critical failure, preventing other stories in the
> suite to run.
> 
> NOTRY=True
> TBR=nednguyen@google.com
> 
> Bug:  735405 
> Change-Id: Ic46b1815ae8599927b9c2366f7fde6b51a0595b0
> Reviewed-on: https://chromium-review.googlesource.com/543116
> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
> Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#481198}

TBR=perezju@chromium.org,nednguyen@google.com

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  735405 
Change-Id: Ia9b6fecf7076356a3faf60024457282753f317ef
Reviewed-on: https://chromium-review.googlesource.com/549062
Reviewed-by: siddhartha sivakumar <ssid@chromium.org>
Commit-Queue: siddhartha sivakumar <ssid@chromium.org>
Cr-Commit-Position: refs/heads/master@{#482424}
[modify] https://crrev.com/4d09478c03cfc9c735b46af22aad84589e23911b/tools/perf/page_sets/system_health/expectations.py

Looking good. Thanks!