New issue
Advanced search Search tips

Issue 747560 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Oct 4
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Perf bugs keep having failed bisects

Project Member Reported by enne@chromium.org, Jul 21 2017

Issue description

I'm doing triage for my team, and I have a pile of what feel like unactionable perf bugs because bisects fail quite frequently:

https://bugs.chromium.org/p/chromium/issues/detail?id=741705
https://bugs.chromium.org/p/chromium/issues/detail?id=741717
https://bugs.chromium.org/p/chromium/issues/detail?id=741736
https://bugs.chromium.org/p/chromium/issues/detail?id=743053

Is there anything that can be done to fix the bisect success rate?

Is there some way to cut down on the noise of perf waterfall bugs without any extra information?
 
Owner: simonhatch@chromium.org
Thanks for reporting! We're currently finishing off a ground-up rewrite of the bisect system to make it more reliable and easier to debug. I took a look at the failures in the short term; there are a couple different things going on:

Simon, can you take a look at the failure in  bug 741705 :
https://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_10_perf_bisect/builds/1318
Traceback (most recent call last):
  File "C:\b\rr\tmpezdhri\rw\checkout\scripts/slave\recipe_modules\crrev\resources\crrev_client.py", line 77, in <module>
    print main(sys.argv[1:])
  File "C:\b\rr\tmpezdhri\rw\checkout\scripts/slave\recipe_modules\crrev\resources\crrev_client.py", line 73, in main
    return json.dumps(crrev_get(args.path, params, args.attempts), indent=2)
  File "C:\b\rr\tmpezdhri\rw\checkout\scripts/slave\recipe_modules\crrev\resources\crrev_client.py", line 52, in crrev_get
    return json.loads(response.text)
  File "C:\b\depot_tools\python276_bin\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\b\depot_tools\python276_bin\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\b\depot_tools\python276_bin\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I remember seeing a similar error for  bug 743053  last week:
https://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/7419
Traceback (most recent call last):
  File "/tmp/tmpj6nHmT.py", line 4, in <module>
    shutil.copy(sys.argv[1], sys.argv[2])
  File "/usr/lib/python2.7/shutil.py", line 119, in copy
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/b/c/b/linux_perf_bisect/src/testing/buildbot/tryserver.chromium.perf.json'
Simon, do you know what caused that? I'll re-kick the bisect to see if it was transient.

 Bug 741717  and  bug 741736  were timing out because the monitoring was set up for the summary, and not the individual pages, so the bisect is not running on the page that regressed. I fixed the monitoring and re-kicked more specific bisects.
Oops, for  bug 743053  I went to go re-kick the bisect and it looks like the sheriff shouldn't have filed that; the ref build jumps at the same time as ToT, which also corresponds to labs team changing the linux perf testers. I WontFix-ed that one.

Comment 3 by enne@chromium.org, Jul 24 2017

Thanks!  I'm glad to hear the bisect system is getting some love.  When it finds something, it's such a helpful tool.
From the looks of it, the error in 741705 was some sort of transient crrev problem, I don't see any other instances of that before/after.
Status: Assigned (was: Untriaged)
Status: Archived (was: Assigned)

Sign in to add a comment