Many "dashboard upload" step failed on many perf bots |
||||||
Issue descriptionhttps://build.chromium.org/p/chromium.perf/builders/Mac%2010.11%20Perf/builds/750 Log: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FMac_10.11_Perf%2F750%2F%2B%2Frecipes%2Fsteps%2Fmedia.tough_video_cases_tbmv2.reference_Dashboard_Upload%2F0%2Fstdout ending result 1 of 2 to dashboard. !@@@STEP_LINK@Results Dashboard@https://chromeperf.appspot.com/report?masters=ChromiumPerf&bots=chromium-rel-mac11&tests=media.tough_video_cases_tbmv2&rev=475820@@@ HTTPError: 500. Reponse: <html><head> <meta http-equiv="content-type" content="text/html;charset=utf-8"> <title>500 Server Error</title> </head> <body text=#000000 bgcolor=#ffffff> <h1>Error: Server Error</h1> <h2>The server encountered an error and could not complete your request.<p>Please try again in 30 seconds.</h2> <h2></h2> </body></html>
,
May 31 2017
This also happen on: https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf/builds/736 https://build.chromium.org/p/chromium.perf/builders/Mac%20Mini%208GB%2010.12%20Perf/builds/1621 https://build.chromium.org/p/chromium.perf/builders/Win%207%20ATI%20GPU%20Perf/builds/801 ... I suspect this is dashboard being overloaded?
,
May 31 2017
I just checked the logs, and except for some CrOS problems, all the 500 errors on /add_point are "Deadline exceeded". Ethan, Simon, any ideas what could cause that? logs: go/bug-728203-500s
,
May 31 2017
So some notes looking at this: All 3 bots started failing dashboard uploads roughly around the same time, noon'ish May 26. We deployed a new dashboard the day before, and there's no failures on the 25th, so probably not that.
,
Jun 5 2017
I'm seeing a lot of errors like this today. Has anyone made progress on figuring out why this is happening?
,
Jun 5 2017
Unfortunately not no, I can dig into this today.
,
Jun 5 2017
Thanks Simon! This is making us losing lots of perf data, so I really appreciate your help on this.
,
Jun 5 2017
https://docs.google.com/spreadsheets/d/1ANJ_yNYO-H8uGVmzGtRDQ9rN3Mq0I4-0RPmmZE5XBvQ/edit#gid=30199547 is a list of the failures on all our bots, if anyone is curious. Looks fairly widespread, and not really isolated to a particular metric.
,
Jun 5 2017
Holy cow. I pump the priority of this bug to P0 due to the the number of failures. Thanks Stephen for that data!
,
Jun 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7c4b7842b0e593a4b165438a4d1281a4c006c4b0 commit 7c4b7842b0e593a4b165438a4d1281a4c006c4b0 Author: eakuefner <eakuefner@chromium.org> Date: Mon Jun 05 20:23:36 2017 [Perf] Filter memory metrics in media benchmarks Recently, memory metrics were turned on for media benchmarks, but the filter used to reduce the volume of memory data sent to the perf dashboard was not used. This CL adds the filter to these benchmarks to reduce the chance of the perf dashboard being overloaded. BUG= 728203 NOTRY=true Review-Url: https://codereview.chromium.org/2917423003 Cr-Commit-Position: refs/heads/master@{#477069} [modify] https://crrev.com/7c4b7842b0e593a4b165438a4d1281a4c006c4b0/tools/perf/benchmarks/media.py
,
Jun 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/752a2952974b6db1244b916a3ce51165f2d69ba0 commit 752a2952974b6db1244b916a3ce51165f2d69ba0 Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Mon Jun 05 21:58:41 2017 Roll src/third_party/catapult/ 6171fd4dd..e7bf345be (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/6171fd4dd88d..e7bf345be18c $ git log 6171fd4dd..e7bf345be --date=short --no-merges --format='%ad %ae %s' 2017-06-05 sullivan Log what was being stored before datastore errors. Created with: roll-dep src/third_party/catapult BUG= 728203 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: Ia1d15024d84de309c836b234ec683b1dcc96eb7c Reviewed-on: https://chromium-review.googlesource.com/524235 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#477090} [modify] https://crrev.com/752a2952974b6db1244b916a3ce51165f2d69ba0/DEPS
,
Jun 8 2017
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by nedngu...@google.com
, May 31 2017