Yeah, I can't see any GPU process deaths on the bots that still work, so appears our win7 bots have broken GL drivers since a couple days back. I'm trying to figure out if I can see what has changed on the bots. It does not appear to be a patch that broke this, since the bots have different blamelists but start failing at almost exactly the same time.

Ok, appears the tests are just timing out for the 720p tests. So maybe someone did an upgrade that broke GL on the bots, they're now falling back on slow software graphics causing them to time out?

This is tricky so I'll disable the tests on win 7 for now to green up the bots.

sgtm

I checked if the test has just gradually grown slower, but that's not the case. The HD test cases took around 200 seconds to execute before they started breaking, far from the timeout of 350 seconds. Now that I hand-run them on the bot they take about 400 seconds to execute.

Dear labs: can you help me understand what happened to our poor win7 bots? It appears the GPU process dies on them now, and that causes our tests to time out (maybe because they fall back on software GL)?

The two bots are
build83-a1
build130-a1

Is there any log somewhere about what upgrades or changes have been made to the machines?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9547bb36ce85227ef99e6ac31d923669d97b2177

commit 9547bb36ce85227ef99e6ac31d923669d97b2177
Author: phoglund <phoglund@chromium.org>
Date: Tue May 24 10:57:00 2016

Disabling HD video quality tests on win7.

A labs engineer probably needs to look at the GL drivers on the bot.
The problem only affects HD tests on Win 7.

BUG= 613947 
R=hbos@chromium.org

Review-Url: https://codereview.chromium.org/2008583004
Cr-Commit-Position: refs/heads/master@{#395562}

[modify] https://crrev.com/9547bb36ce85227ef99e6ac31d923669d97b2177/chrome/browser/media/webrtc_video_quality_browsertest.cc

Henrik asked around in a separate thread and the answer is "nothing has been done to the machines". It's hard to say if it's cosmic rays / goblins / other unexpected changes. That's annoying. If I'm lucky maybe the bots haven't tossed the logs from other bots around that time; I'll see if I can find a corresponding massive increase in execution time from around that time.

They take 247 seconds to execute on the Mac bot. That's slower than what I would expect. Unfortunately we don't have any logs from back then. The mac bot doesn't say anything about GL failures at least.

Wow, the Linux tests run in like 55 seconds. That mostly shoots down the theory that this is a general slowdown.

When I hand-ran on the bots, it appears we entered the processing stage pretty quickly, but the processing stage (copy+write+analyze) was incredibly slow. It could be the V8 update made the javascript code we use a lot slower on Windows. I should probably loop them in.

There was a V8 roll in the blame list which I initially wrote off, but it's increasingly likely it's actually the "culprit". Well, our javascript code could surely be more efficiently written.

Assigning to bmeurer, who will make an attempt at making String.fromCharCode faster to mitigate the regression.

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/7554360f28adebd086761c3636c004a1013ebbb5

commit 7554360f28adebd086761c3636c004a1013ebbb5
Author: bmeurer <bmeurer@chromium.org>
Date: Tue May 31 11:37:26 2016

[builtins] Migrate String.fromCharCode to TurboFan code stub.

When we moved the String.fromCharCode builtin to C++, we slightly
regressed the fast single character code argument case. Recovered some
of the performance by implementing the builtin using the TurboFan
CodeStubAssembler.

Drive-by-fix: Make sure the stack trace from the implicit ToNumber
conversion in String.fromCharCode includes the builtin by adding a
regression test for that.

R=yangguo@chromium.org
BUG=chromium:609831, chromium:613947 ,v8:5049

Review-Url: https://codereview.chromium.org/2021143003
Cr-Commit-Position: refs/heads/master@{#36611}

[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/bootstrapper.cc
[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/builtins.cc
[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/builtins.h
[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/code-stub-assembler.cc
[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/code-stub-assembler.h
[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/runtime/runtime-internal.cc
[modify] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/src/runtime/runtime.h
[add] https://crrev.com/7554360f28adebd086761c3636c004a1013ebbb5/test/mjsunit/regress/regress-string-from-char-code-tonumber.js

The mac bots now take 150 seconds to execute the test, down from 250, so I think your fix had the desired effect :) I'll try to re-enable the tests now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d5eb5fc29849c3ddb1f96d87f84c448cd65286cd

commit d5eb5fc29849c3ddb1f96d87f84c448cd65286cd
Author: phoglund <phoglund@chromium.org>
Date: Thu Jun 02 11:07:08 2016

Revert of Disabling HD video quality tests on win7. (patchset #1 id:1 of https://codereview.chromium.org/2008583004/ )

Reason for revert:
The charCode regression has been fixed, so the tests should work now.

Original issue's description:
> Disabling HD video quality tests on win7.
>
> A labs engineer probably needs to look at the GL drivers on the bot.
> The problem only affects HD tests on Win 7.
>
> BUG= 613947 
> R=hbos@chromium.org
>
> Committed: https://crrev.com/9547bb36ce85227ef99e6ac31d923669d97b2177
> Cr-Commit-Position: refs/heads/master@{#395562}

TBR=hbos@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 613947 

Review-Url: https://codereview.chromium.org/2032993002
Cr-Commit-Position: refs/heads/master@{#397361}

[modify] https://crrev.com/d5eb5fc29849c3ddb1f96d87f84c448cd65286cd/chrome/browser/media/webrtc_video_quality_browsertest.cc

The tests are back up so we're good. They take around 200 seconds on the FYI win7 bot and 334 (!) on the regular win7 bot. That's too close for comfort but it seems to stay under the limit consistently. We're good for now. Thanks bmeurer!