This always seems to fail at the same point: After benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:amazon, the swarming bot is unable to remove the run directory, and crashes/times out.

I'm disabling that test; hopefully it's that one (and not, say, the one immediately after it) which is actually timing out.

Ben, I've seen you name in this file's logs a lot; are you the right owner for this one?

(Removing from general sheriff queue)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fb205333f06c99935c86da532f084b740c993da9

commit fb205333f06c99935c86da532f084b740c993da9
Author: Ian Clelland <iclelland@chromium.org>
Date: Thu Feb 22 19:35:32 2018

Disable flaky smoke test

benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:amazon
is flaky on Win7, causing shard timeouts. Disabling for now.

Bug: 814791
Change-Id: I9d715a8fc77781f4aa23bdf4cc7ff022500eb84c
TBR: benjhayden@chromium.org
Reviewed-on: https://chromium-review.googlesource.com/931751
Reviewed-by: Ian Clelland <iclelland@chromium.org>
Commit-Queue: Ian Clelland <iclelland@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538525}
[modify] https://crrev.com/fb205333f06c99935c86da532f084b740c993da9/tools/perf/benchmarks/system_health_smoke_test.py

Ned, it looks like a lot of these smoke tests are disabled. Maybe worth a deeper look?

Oops. Looks like experimental mode isn't working properly on the waterfall yet. I think I will revert my CL enabling tpu here

uh, it definitely works, e.g. https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%28dbg%29%281%29/66498.

SoM might not be respecting it, though.

Right, I think it was SOM that didn't skip the failure.

+Sean: is it possible to tell SOM to black list telemetry_perf_unittests on "chromium.win/Win7 Tests (dbg)(1)"?

er, I should rephrase: I don't think we'd indicate this to SoM in any way *other* than having "experimental" in the step name; the results JSON isn't currently modified to include any sort of indication as to the suite's experimental state.

in any event: we shouldn't be disabling experimental tests. you should be able to revert #4.

I will revert #4 & my CL added TPU in win 7 tests for now until SOM support ignoring experimental step.

I would lean toward leaving TPU on win7 dbg for now, tbh.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0206c3113a19953ea962a6b1cdfd639bbc4b6b7c

commit 0206c3113a19953ea962a6b1cdfd639bbc4b6b7c
Author: Ned Nguyen <nednguyen@google.com>
Date: Thu Feb 22 20:33:56 2018

Revert "Disable flaky smoke test"

This reverts commit fb205333f06c99935c86da532f084b740c993da9.

Reason for revert: offending CL is reverted.

Original change's description:
> Disable flaky smoke test
> 
> benchmarks.system_health_smoke_test.SystemHealthBenchmarkSmokeTest.system_health.memory_desktop.load:search:amazon
> is flaky on Win7, causing shard timeouts. Disabling for now.
> 
> Bug: 814791
> Change-Id: I9d715a8fc77781f4aa23bdf4cc7ff022500eb84c
> TBR: benjhayden@chromium.org
> Reviewed-on: https://chromium-review.googlesource.com/931751
> Reviewed-by: Ian Clelland <iclelland@chromium.org>
> Commit-Queue: Ian Clelland <iclelland@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#538525}

TBR=iclelland@chromium.org

Change-Id: I13a4195ef70784e6cc87af2d79ef2c7e986ba668
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 814791
Reviewed-on: https://chromium-review.googlesource.com/932523
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#538549}
[modify] https://crrev.com/0206c3113a19953ea962a6b1cdfd639bbc4b6b7c/tools/perf/benchmarks/system_health_smoke_test.py

I think you can exclude this step in gatekeeper config to make SoM ignore it. https://cs.chromium.org/chromium/build/scripts/slave/gatekeeper.json?q=gatekeeper&sq=package:chromium&l=493

I'd rather not have SoM try to extract any information from step names.

I filed  issue 815006  for SOM

It's still happening https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20Tests%20%28dbg%29%281%29/68367

Looks like a timeout.

telemetry_perf_unittests (experimental) has been mostly failing on Win7 dbg for more than a week:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20Tests%20%28dbg%29%281%29?limit=200

*Note that this doesn't change the status code of the build as it's an experimental build step.

telemetry_perf_unittests (experimental) still fails 75% percent of times, but the result seem to be ignored now.

sample builds:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20Tests%20%28dbg%29%281%29/68933
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20Tests%20%28dbg%29%281%29/68926

#22: that's the point of experimental.

This is cluttering sheriff-o-matic. Can it be fixed/removed/something?

Okay, so now the error is painted green for "Win7 Tests (dbg)(1)".
But it also fails on these:
"Win 7 Tests x64 (1)" - first failing build is 38811 [1]
"Win7 Tests (1)" - first failing build is 80169 [2]

And they fail consistently (or are they just incredibly flaky?). Now we are in a horrible state: the tests fail and we have no way of telling whether its the test or a change we made at some point.
I vote for reverting/disabling this experimental test.

If you absolutely insist on having failing and flaky tests, please share the way you painted them green and document heavily why this is absolutely necessary.

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win%207%20Tests%20x64%20%281%29/38811
[2] https://ci.chromium.org/buildbot/chromium.win/Win7%20Tests%20%281%29/80169

#25, those issues are new and separated from this bug. I just file bug for them in  issue 846289 

This test suite is failing very often on win7 tests dbg still - see <https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20Tests%20%28dbg%29%281%29/70777> for example. The symptoms are the same as in #1.