New issue
Advanced search Search tips

Issue 794582 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Most benchmarks timing out when uploading to the test-results dashboard

Project Member Reported by charliea@chromium.org, Dec 13 2017

Issue description

If you go to https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5%20Perf/builds/772 and search for "Upload to test-results", it looks like the vast majority of benchmarks are failing to upload their test results. 

Going to https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=system_health.common_desktop seems to corroborate this: the most recent data point is from 12/5.

Ned, any idea who we should route this issue to?
 
Cc: -nednguyen@chromium.org nedngu...@google.com
go/right-ned
Components: Infra>Flakiness>Dashboard
Owner: seanmccullough@chromium.org
The failing is:

Traceback (most recent call last):
  File "/b/rr/tmpL5O7OW/rw/checkout/scripts/slave/recipe_modules/test_results/resources/upload_test_results.py", line 199, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/rr/tmpL5O7OW/rw/checkout/scripts/slave/recipe_modules/test_results/resources/upload_test_results.py", line 194, in main
    options.test_results_server, attrs, files, 120)
  File "/b/rr/tmpL5O7OW/rw/checkout/scripts/slave/recipe_modules/test_results/resources/test_results_uploader.py", line 51, in upload_test_results
    timeout_secs)
  File "/b/rr/tmpL5O7OW/rw/checkout/scripts/slave/recipe_modules/test_results/resources/test_results_uploader.py", line 131, in _retry_exp_backoff
    raise TimeoutError()

(https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf%2F772%2F%2B%2Frecipes%2Fsteps%2FUpload_to_test-results__blink_perf.bindings.reference_%2F0%2Fstdout)


Sean: are you the right person to look at this? If not, can you reroute this bug? 
Labels: Performance-Sheriff-BotHealth
Status: Available (was: Untriaged)
Yep, right person.

Looks like this is getting 500s on the server when it tries to process the upload:

updateAggregate: merge for master: "chromium.perf", builder: "Android Nexus5 Perf", file: "results-small.json" :: {"error":"model: Merge: expected *AggregateTestLeaf, but got: model.AggregateTest{\"artifacts\":(*model.AggregateTestLeaf)(0xc0201fcf00)}"}


Mergedinto: 772134
Status: Duplicate (was: Available)
Thought this sounded familiar.

Sign in to add a comment