New issue
Advanced search Search tips

Issue 785966 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Issues with JSON decoding in results_dashboard.py

Project Member Reported by eakuefner@chromium.org, Nov 16 2017

Issue description

Looks like this is an error that json.loads can raise. May or may not be related to the histogram pipeline; we should investigate.

From sullivan@:

I dug a bit deeper. We don't seem to be getting a dashboard error because the data is uploading correctly. Here is sample output:
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus6_WebView_Perf%2F661%2F%2B%2Frecipes%2Fsteps%2Fblink_perf.bindings_Dashboard_Upload%2F0%2Fstdout
Sending result 1 of 19 to dashboard. Sending result 2 of 19 to dashboard. ValueError: Extra data: line 1 column 5 - line 5 column 4 (char 4 - 114) Error uploading to dashboard.
 
Here's the first failing build from Mac Retina: https://build.chromium.org/p/chromium.perf/builders/Mac%20Retina%20Perf/builds/1407

The weird thing is, the loading.desktop upload _did_ actually fail, but yet the dashboard upload step is green. That's probably an additional issue.
Project Member

Comment 2 by bugdroid1@chromium.org, Mar 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/d75aeafec43eb0d98cd2fac345785644bd35513e

commit d75aeafec43eb0d98cd2fac345785644bd35513e
Author: Simon <simonhatch@chromium.org>
Date: Wed Mar 07 00:14:14 2018

Refactor uploading histograms and don't retry as often.

Previously, the retry logic retried for just about any error. This
changes the logic to only retry in certain conditions. Motivation for
this was bad data getting cached and retried over and over, meaning
someone eventually had to login to every bot and manually clear it out.

For exceptions from httplib2, retry.

For dashboard errors (500's, 400's), very likely it's not worth trying
again since that means we hit an exception dashboard side (malformed
data that we don't handle properly), or timed out. We can potentially
recover from a 403 (auth error), so we only retry on that currently.


Bug:  785966 
Change-Id: I4beac8ee1cec37a4a1b7835990b5ed432ff7c109
Reviewed-on: https://chromium-review.googlesource.com/930025
Commit-Queue: Simon Hatch <simonhatch@chromium.org>
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/d75aeafec43eb0d98cd2fac345785644bd35513e/scripts/slave/unittests/results_dashboard_test.py
[modify] https://crrev.com/d75aeafec43eb0d98cd2fac345785644bd35513e/scripts/slave/results_dashboard.py

Owner: simonhatch@chromium.org
Status: Fixed (was: Assigned)
Project Member

Comment 4 by bugdroid1@chromium.org, Mar 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2f0acfa98f2863f297d1eb0f96efd96afc1a30d9

commit 2f0acfa98f2863f297d1eb0f96efd96afc1a30d9
Author: Ashley Enstad <ashleymarie@chromium.org>
Date: Tue Mar 13 14:02:47 2018

Merging changes to src side

https://chromium-review.googlesource.com/c/chromium/tools/build/+/930025

Bug:  785966 
Change-Id: I3e0e501abf7d97deb95151afc3b08a94924815e0
Reviewed-on: https://chromium-review.googlesource.com/948922
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Reviewed-by: Simon Hatch <simonhatch@chromium.org>
Commit-Queue: Ashley Enstad <ashleymarie@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542788}
[modify] https://crrev.com/2f0acfa98f2863f297d1eb0f96efd96afc1a30d9/tools/perf/core/results_dashboard.py
[modify] https://crrev.com/2f0acfa98f2863f297d1eb0f96efd96afc1a30d9/tools/perf/core/upload_results_to_perf_dashboard.py

Project Member

Comment 5 by bugdroid1@chromium.org, Apr 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/a83fc01423d3eec5f9db1a2bacf69527fa2617f0

commit a83fc01423d3eec5f9db1a2bacf69527fa2617f0
Author: Chong Zhang <chongz@chromium.org>
Date: Fri Apr 13 18:13:32 2018

Add 'HTTP Error 400' matching string and print unmatched error

Adding 'HTTP Error 400' in addition to 'HTTPError: 400' as observed in
the Mojo Linux Perf bot through pdb:
```
Error while uploading chartjson data: Traceback (most recent call last):
  File "/b/rr/tmpDItjLG/rw/checkout/scripts/slave/results_dashboard.py", line 485, in _SendResultsJson
    urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: Bad Request
```

Also print unmatched error for the ease of debugging.

Bug:  785966 
Change-Id: I93c881ae14355eba11d42b31ec19c56ae7e4bd49
Reviewed-on: https://chromium-review.googlesource.com/1011300
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Commit-Queue: Chong Zhang <chongz@chromium.org>

[modify] https://crrev.com/a83fc01423d3eec5f9db1a2bacf69527fa2617f0/scripts/slave/results_dashboard.py

Sign in to add a comment