New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 828378 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug

Blocking:
issue 779637



Sign in to add a comment

Reference build not able to run on Android Go

Project Member Reported by eyaich@chromium.org, Apr 3 2018

Issue description

Currently all reference builds are failing on android go.  Although we don't monitor them in the test results, we do upload the data to the perf dashboard.  

Link to the most recent task where tests are failing (the shard is still passing because we ignore reference build failures): https://chrome-swarming.appspot.com/task?id=3ca37dae07043f10&refresh=10&show_raw=1

The benchmark is failing to run with the following error: 

(INFO) 2018-04-03 09:23:41,560 battor_wrapper.IsBattOrConnected:39  BattOr power monitor is not connected.
(WARNING) 2018-04-03 09:23:41,560 binary_manager._WrapDependencyManagerFunction:59  Cannot find path for chrome_stable on platform android_n_armeabi-v7a. Falling back to android_armeabi-v7a.
(INFO) 2018-04-03 09:23:41,561 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s A6R9K17A17901177 shell '( pm list packages );echo %$?'
(WARNING) 2018-04-03 09:23:43,028 desktop_browser_finder.FindAllAvailableBrowsers:244  Chrome build location for linux_x86_64 not found. Browser will be run without Flash.
(WARNING) 2018-04-03 09:23:43,124 desktop_browser_finder.FindAllAvailableBrowsers:378  Found (reference), but you do not have a DISPLAY environment set.
Cannot find browser of type reference. To list out all available browsers, rerun your command with --browser=list
(INFO) 2018-04-03 09:23:43,124 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s A6R9K17A17901177 shell '( dumpsys battery );echo %$?'
(INFO) 2018-04-03 09:23:43,256 android_power_monitor_controller._ReenableChargingIfNeeded:13  Charging status checked at exit.
Traceback (most recent call last):
  File "/b/swarming/w/ir/testing/scripts/run_telemetry_benchmark_as_googletest.py", line 130, in run_benchmark
    with open(tempfile_name) as f:
IOError: [Errno 2] No such file or directory: '/b/swarming/w/itzGULyB/tmpHDnEuNtelemetry/results-chart.json'
/ir/.swarming_module/bin:/b/swarming/cipd_cache/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'NO_GCE_CHECK': 'False', 'CHROME_HEADLESS': '1', '_': '/usr/bin/python', 'SWARMING_TASK_ID': '3ca37dae07043f11', 'DOCKER_HOST_HOSTNAME': 'build30-a7.labs.chromium.org', 'ISOLATED_CACHE_SIZE': '88041570304', 'PWD': '/b/swarming', 'SWARM_URL': 'https://chrome-swarming.appspot.com/bot_code', 'MAIL': '/var/mail/chrome-bot'})
Command ['/b/swarming/w/ir/.swarming_module_cache/vpython/09eff0/bin/python', '../../tools/perf/run_benchmark', u'thread_times.key_idle_power_cases', '-v', '--browser=reference', '--output-format=chartjson', '--max-failures=5', '--output-trace-tag=_ref', '--output-dir', '/b/swarming/w/itzGULyB/tmpHDnEuNtelemetry', '--output-format=json-test-results'] returned exit code 1

Juan do you have any thoughts on what might be going wrong with android and telemetry not being able to find the browser here?  
 
Ned, any thoughts? Not sure how the reference browser lookup works on Android.

Is this specific to the Android Go bot? Has this been always broken, or started failing at some point?
This is a new failure to android go.

We saw the error on linux when we weren't running with xvfb, but I am wondering if that was a red herring since if that were the case it would fail for both the reference and non-reference build, and here all the non-reference build cases are passing.
I think this log is relevant: 
(WARNING) 2018-04-03 09:23:41,560 binary_manager._WrapDependencyManagerFunction:59  Cannot find path for chrome_stable on platform android_n_armeabi-v7a. Falling back to android_armeabi-v7a.

I am suspecting that we don't have a reference chrome binary for the ANdroid Go platform checked in cloud storage yet.

The list of reference CHrome binaries are in https://github.com/catapult-project/catapult/blob/master/common/py_utils/py_utils/chrome_binaries.json
I think this is not an infra failure of any kind, we simply didn't put the required Chrome APK on cloud storage yet. 

The fix for this is also easy, you just need to add android_armeabi-v7a to https://github.com/catapult-project/catapult/blob/master/common/bin/update_chrome_reference_binaries#L80 then run update_chrome_reference_binaries script
Sounds reasonable, I will try that.  Wouldn't it be android_n_armeabi-v7a that we would need to add given the log statement? 

Comment 6 by eyaich@chromium.org, Apr 16 2018

Cc: eyaich@chromium.org
Owner: ----
Ned and I looked at this script to upgrade the binary and it is currently not able to find the right binaries in cloud storage, needs more investigation, but de-prioritizing in Q2 while correct hardware is being investigated.

Comment 7 by eyaich@chromium.org, Apr 16 2018

Blocking: 779637
Project Member

Comment 8 by bugdroid1@chromium.org, Apr 16 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/869da49b098afbb43aca70c25a40ab807b912443

commit 869da49b098afbb43aca70c25a40ab807b912443
Author: Nghia Nguyen <nednguyen@google.com>
Date: Mon Apr 16 20:11:24 2018

Update update_chrome_reference_binaries to use 'win-clang' and 'win64-clang' in cloud storage directory

This fixes the issue with this script crashes when trying to fetch reference
build on Chrome win.

Bug:  chromium:828378 

TBR=eyaich@chromium.org

Change-Id: I7c5c0827f6f2f0b88b705a1060d0302a6cea3627
Reviewed-on: https://chromium-review.googlesource.com/1014297
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/869da49b098afbb43aca70c25a40ab807b912443/common/bin/update_chrome_reference_binaries

Project Member

Comment 9 by bugdroid1@chromium.org, Apr 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/87c43b293fdf5361edcf7fdff9f3e0cfe2705156

commit 87c43b293fdf5361edcf7fdff9f3e0cfe2705156
Author: catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Tue Apr 17 00:56:02 2018

Roll src/third_party/catapult/ 5d0c16d72..69a36a622 (3 commits)

https://chromium.googlesource.com/catapult.git/+log/5d0c16d725ca..69a36a622381

$ git log 5d0c16d72..69a36a622 --date=short --no-merges --format='%ad %ae %s'
2018-04-16 nednguyen Update Chrome reference builds (Chrome 63 to 65)
2018-04-16 simonhatch Dashboard - Remove whitespace from index.yaml
2018-04-16 nednguyen Update update_chrome_reference_binaries to use 'win-clang' and 'win64-clang' in cloud storage directory

Created with:
  roll-dep src/third_party/catapult
BUG= chromium:828378 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: Ia8d35f0ab940b07231ee7520ec2e56e700fe9a39
Reviewed-on: https://chromium-review.googlesource.com/1014625
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#551197}
[modify] https://crrev.com/87c43b293fdf5361edcf7fdff9f3e0cfe2705156/DEPS

Labels: -Pri-1 Pri-2
Components: Infra
Components: -Infra Infra>Client>Perf
Project Member

Comment 13 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/55650fadabf7af12ce6c594c83d29b520cfb14d3

commit 55650fadabf7af12ce6c594c83d29b520cfb14d3
Author: Nghia Nguyen <nednguyen@google.com>
Date: Wed Apr 25 13:33:16 2018

Add android_n_armeabi-v7a platform to chrome-stable

Bug:  chromium:828378 
Change-Id: I1d739a04ff4ff0d785410a88ff12bba9c5a232b4
Reviewed-on: https://chromium-review.googlesource.com/1024242
Commit-Queue: Emily Hanley <eyaich@chromium.org>
Reviewed-by: Emily Hanley <eyaich@chromium.org>

[modify] https://crrev.com/55650fadabf7af12ce6c594c83d29b520cfb14d3/common/bin/update_chrome_reference_binaries
[modify] https://crrev.com/55650fadabf7af12ce6c594c83d29b520cfb14d3/common/py_utils/py_utils/chrome_binaries.json

Project Member

Comment 14 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5b91b126d511591dac91e0e0b1eb950397e498bd

commit 5b91b126d511591dac91e0e0b1eb950397e498bd
Author: catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Wed Apr 25 15:44:24 2018

Roll src/third_party/catapult/ 0d6f84866..55650fada (1 commit)

https://chromium.googlesource.com/catapult.git/+log/0d6f84866778..55650fadabf7

$ git log 0d6f84866..55650fada --date=short --no-merges --format='%ad %ae %s'
2018-04-23 nednguyen Add android_n_armeabi-v7a platform to chrome-stable

Created with:
  roll-dep src/third_party/catapult
BUG= chromium:828378 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: I1f318c23f328cc38459617e1a43d6be253bcb1bf
Reviewed-on: https://chromium-review.googlesource.com/1028133
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#553572}
[modify] https://crrev.com/5b91b126d511591dac91e0e0b1eb950397e498bd/DEPS

Status: Fixed (was: Untriaged)
Owner: nednguyen@chromium.org

Sign in to add a comment