New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 680174 link

Starred by 1 user
Status: Fixed
Owner:
bpastene@chromium.org
Closed: Jan 2017
Cc:
█ mikec...@chromium.org
martiniss@chromium.org
█ rnep...@chromium.org
█ benhenry@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 678046



Sign in to add a comment

loading.mobile failure on chromium.perf.fyi on android swarming bot

Project Member Reported by eyaich@chromium.org, Jan 11 2017 
This is a failure on our testing android bot for android swarming.  It has been timing out since this bot has been up.  But this is the last failure on android swarming!!

Here is the link to the build status page with the failure: 

https://uberchromegw.corp.google.com/i/chromium.perf.fyi/builders/Android%20Swarming%20N5X%20Tester

Link to the most recent failing task: 

https://chromium-swarm.appspot.com/task?id=33a53c1e49ff2a10&refresh=10&show_raw=1

Oddly it just starts not being able to find directories and then hits the swarming timeout.  Its almost as if something is being deleted on the swarming bot.  

There was a bug on this test on the main waterfall, another timeout, but they seem slightly different: crbug.com/676612

cc'ing a few people that might have some more android knowledge or might have seen something like this before.

Comment 1 by rnep...@chromium.org, Jan 11 2017 

This test has an insane run time. It is basically running the benchmark twice, once with tsproxy and once without (if I understand eveyrthing correctly). Maybe we should be having a discussion about splitting it into 2 benchmarks, loading.mobile w/ tsproxy and loading.mobile w/o tsproxy. I'm not sure if this is the correct decision, but it seems like a discussion worth having since we keep having timeout issues with loading.mobile. 

Here is an android test runner trace from a run that had loading.mobile in it to give a visualization of just how long it is compared to other tests:
https://storage.cloud.google.com/chromium-testrunner-trace/Android%20Nexus5X%20Perf%20(3)/trace_3159.html

Comment 2 by bpastene@chromium.org, Jan 11 2017 

The test runner isn't the best at handling the timeout-induced sigterm that swarming sends. The test just ignores it and continues on, but swarming eventually gets tired of waiting for the test to exit and starts clearing its work directory. The test loses the adb binary at some point and starts complaining about missing files.

So, IMO the issue here is that swarming's timeout threshold is too low for that task. According to Randy's numbers in #1 (that's a cool link btw) it normally takes ~3 hours. Let's just up the timeout for loading.mobile. We should be able to do that in the recipe. I'll poke around.

Comment 3 by eyaich@chromium.org, Jan 11 2017 

To do that for just this test you would have to update the src side json timeout.  This script has a generic one for all tests, and I don't think we want to change this for all tests.  We just want it this test.  

This is where we set it now: https://cs.chromium.org/chromium/src/tools/perf/generate_perf_json.py?q=generate_perf_json&sq=package:chromium&l=451 but we will need to update this script to be able to set it for just one test and then re-generate and check in.

Comment 6 by bpastene@chromium.org, Jan 14 2017

Status: Fixed (was: Untriaged)

Comment 7 by benhenry@chromium.org, Mar 14 2017

Labels: Performance-Loading
►

Sign in to add a comment