Build failure started at: https://build.chromium.org/p/chromium.win/builders/Win%207%20Tests%20x64%20%281%29/builds/13247

I cannot find a suspicious CL in the range. This could be a bot issue?


===== Sample of failure log =====

[102/999] telemetry.core.memory_cache_http_server_unittest.MemoryCacheHTTPServerTest.testBasicHostingAndRangeRequests failed unexpectedly 0.0470s:
  Chrome build location for win_AMD64 not found. Browser will be run without Flash.
  Failure while starting browser backend.
  Traceback (most recent call last):
    File "e:\b\swarm_slave\work\isolated\runp1kzkd\third_party\catapult\telemetry\telemetry\internal\browser\browser.py", line 55, in __init__
      self._browser_backend.Start()
    File "e:\b\swarm_slave\work\isolated\runp1kzkd\third_party\catapult\telemetry\telemetry\internal\backends\chrome\desktop_browser_backend.py", line 278, in Start
      args, stdout=self._tmp_output_file, stderr=subprocess.STDOUT, env=env)
    File "e:\b\depot_tools\python276_bin\lib\subprocess.py", line 709, in __init__
      errread, errwrite)
    File "e:\b\depot_tools\python276_bin\lib\subprocess.py", line 957, in _execute_child
      startupinfo)
  WindowsError: [Error 14001] The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail
  Traceback (most recent call last):
    File "e:\b\swarm_slave\work\isolated\runp1kzkd\third_party\catapult\telemetry\telemetry\testing\browser_test_case.py", line 46, in setUpClass
      current_browser = browser_to_create.Create(options)
    File "e:\b\swarm_slave\work\isolated\runp1kzkd\third_party\catapult\telemetry\telemetry\internal\backends\chrome\desktop_browser_finder.py", line 68, in Create
      browser_backend, self._platform_backend, self._credentials_path)
    File "e:\b\swarm_slave\work\isolated\runp1kzkd\third_party\catapult\telemetry\telemetry\internal\browser\browser.py", line 55, in __init__
      self._browser_backend.Start()
    File "e:\b\swarm_slave\work\isolated\runp1kzkd\third_party\catapult\telemetry\telemetry\internal\backends\chrome\desktop_browser_backend.py", line 278, in Start
      args, stdout=self._tmp_output_file, stderr=subprocess.STDOUT, env=env)
    File "e:\b\depot_tools\python276_bin\lib\subprocess.py", line 709, in __init__
      errread, errwrite)
    File "e:\b\depot_tools\python276_bin\lib\subprocess.py", line 957, in _execute_child
      startupinfo)
  WindowsError: [Error 14001] The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail

I see the same failures on win_chromium_x64_rel_ng try server, but only once. The next build is green.
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/206776

The next run was green but with retry. https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/206777

The next next run is green w/o retry.
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/206778

It's flaky

No sorry. #206778 doesn't include the telemetry tests.

In the last 200 runs of win_chromium_x64_rel_ng, I can view,
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_rel_ng/builds/206629
is the first run with the failures (retry succeeded). The revision of the run is r390845.

 Issue 608293  has been merged into this issue.

The win_chromium_rel_ng tryserver is OK, so this isn't affecting the CQ.

I suspect it's probably caused by a build or GN switch on this bot. The isolate is configured incorrectly. I think it's likely that manifest files are missing. Dirk, is there any way you could help look into this?

I'm sheriffing today, so sure, I can help. I think dtu@ is looking into this at the moment as well, since I pinged him about it.

The waterfall bots are busted as well.

However, there haven't been any GN/GYP/MB related changes for some time; these bots have been on GN for a week or more.

brucedawson@ has been fixing a bunch of manifest-related issues w/ the win build (see  bug 602505 ). Maybe he has some insight into this, also. My understanding is that if manifests are missing things can indeed be flaky.

it looks like the problem is that we bumped the revision number on Saturday, but that only triggered a partial rebuild, and didn't rebuild everything that needed to be rebuilt (i.e., the resource linking to the manifest didn't get rebuilt).

It looks like the manifest rules are missing a dependency such that when the *.manifest file is updated, the *.res file doesn't get updated, only a bunch of stamp files.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/30eadcf8a08630d39c3df6001834572ef4f1932f

commit 30eadcf8a08630d39c3df6001834572ef4f1932f
Author: brettw <brettw@chromium.org>
Date: Tue May 03 00:55:34 2016

Recompile .rc files when manifests change in GN.

Previously changes in the .manifest file would not cause the .rc file that references it to recompile, giving stale builds.

BUG= 608228 

Review-Url: https://codereview.chromium.org/1943103002
Cr-Commit-Position: refs/heads/master@{#391131}

[modify] https://crrev.com/30eadcf8a08630d39c3df6001834572ef4f1932f/build/config/win/manifest.gni

I believe this is now fixed.

Yes, it is. Thank you!