Ethan - could this be related to things I heard you and Ben talking about today?

re: #c2

Responded to each of these, nexus 5x issue (or at least one of them) was the p0 from last week where all the bisects were failiong. The android one failure seemed to be a gclient sync issue, and the last the device was unresponsive.

Another one (android_nexus6_perf_bisect <-- new bot):

https://bugs.chromium.org/p/chromium/issues/detail?id=729760

I _think_ simonhatch@ is working on it, so updating the bug status accordingly. Let me know if this is incorrect.

What I see in Android bisect is a few retires like:
devtools_client_backend._IsInspectorWebsocketAvailable:57  Websocket at port 35552 not yet available: [Errno 104] Connection reset by peer

should it maybe retry for longer than 1sec? websocket is known to start up slowly at times

re: #c10

pasko@ which runs were you seeing those errors?


I removed a bunch of the blocking issues, as they were either the bisect not repro'ing or specifying chromium revisions for non chromium bisects.

I was looking at issue 729738, lemme try to start new bisects there.

All bisects I started there failed with "tests failed to produce values". Example log: https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-phone-perf-bisect/builds/461

Please take a look.

So in those runs it looks like there are no values generated:

Here's an example json output from one of the runs:
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-phone-perf-bisect/builds/461/steps/Gathering%20reference%20values.Reading%20chartjson%20results%20%288%29/logs/json.output

It's nearly empty, and has no perf data in there.

There are seemingly no errors during the run, and the output seems to indicate the pages ran successfully.

this is strange, just a couple of days ago the same benchmark generated nice chartjson results with messageloop_start_time in my local test runs.

When you ran it, did you use pageset-repeat=1? I vaguely recall there being a bug with startup tests not producing metrics in that case, no idea if that's been fixed or not.

oh, you are right, I ran with pageset-repeat=20 and got 18 results. Sounds like badness on the benchmark side. If it does not work with pageset-repeat=1, we need to fix the benchmark.

Huge thanks for the insight!!

Do you know how to send a bisect with extra commandline parameters from the per dashboard UI? 

I wanted to start a bisect manually using tools/auto_bisect/bisect.cfg, did something, but not sure I succeeded. I then wrote some feedback on documentation in issue 737557.

We have a hack on the dashboard specifically for that startup bug (crbug.com/677843), might have to extend it to cover this benchmark until the benchmark itself is fixed.

nice hack! though I would set something around 20-40, since startup metrics are quite noisy and small amount of runs would likely lead bisect in wrong directions

Sure, although the bisect runs the test a minimum of 6 times, so seems like I'd only need to set the repeat to a few to get 20-40 total samples.

do you mean that each revision is benchmarked 6 times? I was worried that one noisy datapoint may diverge bisection into a sub-range of commits that does not have a regression, hence the bisect won't be useful. Glad that it's not the case.

Is it ready for another triple of bisects or you launched them already?

Yeah each revision is run a min of 6 times.

Let me put up a cl first with the hack to change the repeat, and then deploy that dashbaord. I'll post back here once that's done.

thank you, will wait for dashboard update

New dashboard is up, go for it.

thanks! Done.

Can I close this? Seems like the issues here are covered by individual bugs.

Closing in favor of more specific bugs.