The build archive does not seem to contain the "icudtl.dat" file.

Assigning to Machenbach@ to take a look. Any idea why this does not exist. Anyway, does the new standalone builder on windows have it, is it time to switch ?

I added the missing --verify-heap to windows_asan_d8 job type, could that be triggering these ? Anyway, why is icudtl.dat missing here.

They will have it. Windows isn't there yet. Marking as blocking.

ClusterFuzz has detected this issue as fixed in range 485869:485889.

Detailed report: https://clusterfuzz.com/testcase?key=6094616251334656

Fuzzer: inferno_js_fuzzer_c
Job Type: windows_asan_d8
Platform Id: windows

Crash Type: CHECK failure
Crash Address: 
Crash State:
  normalizer != nullptr in builtins-intl.cc
  v8::platform::PrintStackTrace
  v8::internal::Builtin_Impl_StringPrototypeNormalizeIntl
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=485825:485869
Fixed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=485869:485889

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6094616251334656


See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

ClusterFuzz testcase 6094616251334656 is verified as fixed, so closing issue.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

Digging. Here is what I found out:

1. The builder that built the archive used in this issue did indeed not pack the icudtl.dat file on toplevel:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.lkgr%2FWin_ASan_Release%2F5909%2F%2B%2Frecipes%2Fsteps%2Fzipping%2F0%2Fstdout

This is from build:
https://build.chromium.org/p/chromium.lkgr/builders/Win%20ASan%20Release/builds/5909

2. The most recent build did pack it:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.lkgr%2FWin_ASan_Release%2F5919%2F%2B%2Frecipes%2Fsteps%2Fzipping%2F0%2Fstdout

Here you can see the line:
linking ...\out\Release\icudtl.dat -> ...\asan-win32-release-486296\icudtl.dat

It's from this build:
https://build.chromium.org/p/chromium.lkgr/builders/Win%20ASan%20Release/builds/5919

---

Bottom line: The icudtl.dat file was somehow not there in the build that cause the crashes.

It could be that there was a temporary code change that made the file disappear. But it'd break a bunch of things. A lot of the intl stuff would stop working. 

It could be that the file is flakily missing in some builds? That'd be strange too...

Already in the subsequent build, the ICU data file was back in the zip. But in the compile output of both builds (5909 and 5910) we can see that the file was copied to the output toplevel:

[3889/39281] COPY ../../third_party/icu/common/icudtl.dat icudtl.dat

Digging more...

Somewhere between compile and the filter_builddir step, things are getting screwed up.

The filter step of 5909 does not include the icudtl.dat file:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.lkgr%2FWin_ASan_Release%2F5909%2F%2B%2Frecipes%2Fsteps%2Ffilter_build_dir%2F0%2Flogs%2Fjson.output%2F0

+ Max who optimized the filtering recently.

Here's the diff of filtered files between the two builds 5909 and 5910:
https://paste.googleplex.com/5705359041757184

Actually the output looks a bit scary. I don't know where so many differences come from. Here's the Chromium range of those two builds:
https://chromium.googlesource.com/chromium/src/+log/e069330ac..f66aae2d690

Nothing in there looks like it could cause those differences.

I'd rather guess that something in the filter method is wrong. Maybe windows only or specific to links?

There was no significant change on the infra build repository between the builds.

I checked some of the other files that are missing in the 5909 build and they are all in the compile output. E.g. "First Run", chrome.VisualElementsManifest.xml, SmallLogo.png. All files have in common that they were _copied_ by ninja - not generated by a step.

Is it possible that the subsequent filewalk https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/archive/resources/filter_build_files.py?q=filter_build_files.py&sq=package:chromium&dr&l=122 somehow didn't see those files due to some obscure Windows disk caching thing?

Detailed report: https://clusterfuzz.com/testcase?key=6569636513710080

Fuzzer: inferno_js_fuzzer_c
Job Type: windows_asan_d8
Platform Id: windows

Crash Type: CHECK failure
Crash Address: 
Crash State:
  normalizer != nullptr in builtins-intl.cc
  v8::platform::PrintStackTrace
  v8::internal::Builtin_Impl_StringPrototypeNormalizeIntl
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=486245:486286

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6569636513710080


See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.

We had something similar recently (instrumented libraries often were missing in the builds for no particular reason), issue 726144. We haven't found the culprit that has been deleting them, but adding 'clobber' config resolved the issue for us: https://chromium-review.googlesource.com/c/534678/

A random suggestion: on Windows a path should contain '\\' separators, otherwise things like '\temp' would be parsed as TAB + 'emp'. Source: https://stackoverflow.com/questions/5233814/os-walk-doesnt-walk

If this were some backslash issue, where does the randomness come from?

Maybe we could as a start add some additional debug output to the filter function, in order to know what it sees.

Does ninja in the copy step on windows actually copy the files, or does it create symlinks?

You can't create symlinks on Windows without admin privileges until very recently, so symlinks aren't really a thing on Windows.

Also, ninja just calls programs, it doesn't copy anything itself I believe. You can tell it to run some program to copy something, and then it does whatever that copy program does.

Good to know... I'm just blindly digging here. The compile output prints:

[3889/39281] COPY ../../third_party/icu/common/icudtl.dat icudtl.dat

which should mean icudtl.dat ending up in the output folder toplevel. But the subsequent python script didn't find it in that build.

Before analyzing much further, we should also check it the problem is recurring. Maybe this was a one-time solar-activity-bit-flip :/

ClusterFuzz has detected this issue as fixed in range 486286:486296.

Detailed report: https://clusterfuzz.com/testcase?key=6569636513710080

Fuzzer: inferno_js_fuzzer_c
Job Type: windows_asan_d8
Platform Id: windows

Crash Type: CHECK failure
Crash Address: 
Crash State:
  normalizer != nullptr in builtins-intl.cc
  v8::platform::PrintStackTrace
  v8::internal::Builtin_Impl_StringPrototypeNormalizeIntl
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=486245:486286
Fixed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=486286:486296

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6569636513710080


See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

We see it again, so this is not a one-time occurence, please see https://clusterfuzz.com/v2/testcase-detail/6081375135596544 and its accompanying build https://storage.cloud.google.com/chromium-browser-asan/win32-release/asan-win32-release-487028.zip does not icudtl.dat file in out directory.

Here's the build from archive in comment 19:
https://build.chromium.org/p/chromium.lkgr/builders/Win%20ASan%20Release/builds/5975

The same as in the other build. Compile output shows it, but the filter step doesn't include icudtl.dat. Also misses a bunch of other copied files again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/b985981d9f2ba3104d9e1ef1cd1c8634a1554ff3

commit b985981d9f2ba3104d9e1ef1cd1c8634a1554ff3
Author: Michael Achenbach <machenbach@chromium.org>
Date: Tue Jul 18 08:35:07 2017

Add temporary output for debugging incomplete archives

TBR=mmoroz@chromium.org

Bug:  741603 
Change-Id: I9d3091a4ba3935ffd645aedbebfa180def66fa01
Reviewed-on: https://chromium-review.googlesource.com/575968
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/b985981d9f2ba3104d9e1ef1cd1c8634a1554ff3/scripts/slave/recipe_modules/archive/resources/filter_build_files.py

I've added some extra debug output. Please ping me if this issue happens again. The last 20 builds seem all ok.

I see this today with r490288. Michael, can you please check your debug logs for any clues.

windows_asan_content_shell (r490288)
c:\clusterfuzz\slave-bot\builds\chromium-browser-asan_win32-release_a15986b6468dcac5c0ae9a2e0729fd95d8ce33c8\revisions\asan-win32-release-490288\content_shell.exe --user-data-dir=c:\tmp\user_profile_0 --log-net-log=c:\tmp\net_log_0 --enable-logging --allow-file-access-from-files --disable-gesture-requirement-for-media-playback --use-gl=any --run-layout-test --dump-render-tree 

[844:3228:0728/014006.261:34641546:ERROR:icu_util.cc(174)] Invalid file descriptor to ICU data received.
[844:3228:0728/014006.261:34641546:FATAL:icu_util.cc(306)] Check failed: result.
Backtrace:
base::debug::StackTrace::StackTrace [0x0560D0B1+33] (C:\b\c\b\win_asan_release\src\base\debug\stack_trace_win.cc:217)
base::debug::StackTrace::StackTrace [0x056783BD+13] (C:\b\c\b\win_asan_release\src\base\debug\stack_trace.cc:199)
logging::LogMessage::~LogMessage [0x05457433+579] (C:\b\c\b\win_asan_release\src\base\logging.cc:553)
base::i18n::InitializeICU [0x0573DAA6+1766] (C:\b\c\b\win_asan_release\src\base\i18n\icu_util.cc:306)
content::ContentMainRunnerImpl::Initialize [0x0177CFFA+2138] (C:\b\c\b\win_asan_release\src\content\app\content_main_runner.cc:612)
content::ContentServiceManagerMainDelegate::Initialize [0x0177B695+69] (C:\b\c\b\win_asan_release\src\content\app\content_service_manager_main_delegate.cc:37)
service_manager::Main [0x087A44FA+1242] (C:\b\c\b\win_asan_release\src\services\service_manager\embedder\main.cc:427)
content::ContentMain [0x0177C421+177] (C:\b\c\b\win_asan_release\src\content\app\content_main.cc:19)
main [0x00B111B0+432] (C:\b\c\b\win_asan_release\src\content\shell\app\shell_main.cc:33)
__scrt_common_main_seh [0x0F78C40B+249] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
BaseThreadInitThunk [0x77A27C04+36]
RtlInitializeExceptionChain [0x77D4AD2F+143]
RtlInitializeExceptionChain [0x77D4ACFA+90]

=================================================================
==844:3228==ERROR: AddressSanitizer: breakpoint on unknown address 0x93d48080 (pc 0x0567839c bp 0x18c2df78 sp 0x18c2df78 T0)
==844:3228==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)
==844:3228==*** WARNING: Failed to initialize DbgHelp! ***
==844:3228==*** Most likely this means that the app is already ***
==844:3228==*** using DbgHelp, possibly with incompatible flags. ***
==844:3228==*** Due to technical reasons, symbolization might crash ***
==844:3228==*** or produce wrong results. ***
#0 0x567839b in base::debug::BreakDebugger(void) C:\b\c\b\win_asan_release\src\base\debug\debugger_win.cc:18:7
#1 0x54580c7 in logging::LogMessage::~LogMessage(void) C:\b\c\b\win_asan_release\src\base\logging.cc:784:7
#2 0x573daa5 in base::i18n::InitializeICU(void) C:\b\c\b\win_asan_release\src\base\i18n\icu_util.cc:306:3
#3 0x177cff9 in content::ContentMainRunnerImpl::Initialize(struct content::ContentMainParams const &) C:\b\c\b\win_asan_release\src\content\app\content_main_runner.cc:612:5
#4 0x177b694 in content::ContentServiceManagerMainDelegate::Initialize(struct service_manager::MainDelegate::InitializeParams const &) C:\b\c\b\win_asan_release\src\content\app\content_service_manager_main_delegate.cc:37:32
#5 0x87a44f9 in service_manager::Main(struct service_manager::MainParams const &) C:\b\c\b\win_asan_release\src\services\service_manager\embedder\main.cc:426:29
#6 0x177c420 in content::ContentMain(struct content::ContentMainParams const &) C:\b\c\b\win_asan_release\src\content\app\content_main.cc:19:10
#7 0xb111af in main C:\b\c\b\win_asan_release\src\content\shell\app\shell_main.cc:33:10
#8 0xf78c40a in _scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253
#9 0x77a27c03 (C:\windows\SYSTEM32\KERNEL32.DLL+0x6b817c03)
#10 0x77d4ad2e (C:\windows\SYSTEM32\ntdll.dll+0x6b2dad2e)
#11 0x77d4acf9 (C:\windows\SYSTEM32\ntdll.dll+0x6b2dacf9)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: breakpoint C:\b\c\b\win_asan_release\src\base\debug\debugger_win.cc:18:7 in base::debug::BreakDebugger(void)
==844:3228==ABORTING

[844:3228:0728/014006.261:34641546:ERROR:icu_util.cc(174)] Invalid file descriptor to ICU data received.
[844:3228:0728/014006.261:34641546:FATAL:icu_util.cc(306)] Check failed: result.
Backtrace:
base::debug::StackTrace::StackTrace [0x0560D0B1+33] (C:\b\c\b\win_asan_release\src\base\debug\stack_trace_win.cc:217)
base::debug::StackTrace::StackTrace [0x056783BD+13] (C:\b\c\b\win_asan_release\src\base\debug\stack_trace.cc:199)
logging::LogMessage::~LogMessage [0x05457433+579] (C:\b\c\b\win_asan_release\src\base\logging.cc:553)
base::i18n::InitializeICU [0x0573DAA6+1766] (C:\b\c\b\win_asan_release\src\base\i18n\icu_util.cc:306)
content::ContentMainRunnerImpl::Initialize [0x0177CFFA+2138] (C:\b\c\b\win_asan_release\src\content\app\content_main_runner.cc:612)
content::ContentServiceManagerMainDelegate::Initialize [0x0177B695+69] (C:\b\c\b\win_asan_release\src\content\app\content_service_manager_main_delegate.cc:37)
service_manager::Main [0x087A44FA+1242] (C:\b\c\b\win_asan_release\src\services\service_manager\embedder\main.cc:427)
content::ContentMain [0x0177C421+177] (C:\b\c\b\win_asan_release\src\content\app\content_main.cc:19)
main [0x00B111B0+432] (C:\b\c\b\win_asan_release\src\content\shell\app\shell_main.cc:33)
__scrt_common_main_seh [0x0F78C40B+249] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
BaseThreadInitThunk [0x77A27C04+36]
RtlInitializeExceptionChain [0x77D4AD2F+143]
RtlInitializeExceptionChain [0x77D4ACFA+90]

=================================================================
==844:3228==ERROR: AddressSanitizer: breakpoint on unknown address 0x93d48080 (pc 0x0567839c bp 0x18c2df78 sp 0x18c2df78 T0)
==844:3228==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)
==844:3228==*** WARNING: Failed to initialize DbgHelp! ***
==844:3228==*** Most likely this means that the app is already ***
==844:3228==*** using DbgHelp, possibly with incompatible flags. ***
==844:3228==*** Due to technical reasons, symbolization might crash ***
==844:3228==*** or produce wrong results. ***
#0 0x567839b in base::debug::BreakDebugger(void) C:\b\c\b\win_asan_release\src\base\debug\debugger_win.cc:18:7
#1 0x54580c7 in logging::LogMessage::~LogMessage(void) C:\b\c\b\win_asan_release\src\base\logging.cc:784:7
#2 0x573daa5 in base::i18n::InitializeICU(void) C:\b\c\b\win_asan_release\src\base\i18n\icu_util.cc:306:3
#3 0x177cff9 in content::ContentMainRunnerImpl::Initialize(struct content::ContentMainParams const &) C:\b\c\b\win_asan_release\src\content\app\content_main_runner.cc:612:5
#4 0x177b694 in content::ContentServiceManagerMainDelegate::Initialize(struct service_manager::MainDelegate::InitializeParams const &) C:\b\c\b\win_asan_release\src\content\app\content_service_manager_main_delegate.cc:37:32
#5 0x87a44f9 in service_manager::Main(struct service_manager::MainParams const &) C:\b\c\b\win_asan_release\src\services\service_manager\embedder\main.cc:426:29
#6 0x177c420 in content::ContentMain(struct content::ContentMainParams const &) C:\b\c\b\win_asan_release\src\content\app\content_main.cc:19:10
#7 0xb111af in main C:\b\c\b\win_asan_release\src\content\shell\app\shell_main.cc:33:10
#8 0xf78c40a in _scrt_common_main_seh f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253
#9 0x77a27c03 (C:\windows\SYSTEM32\KERNEL32.DLL+0x6b817c03)
#10 0x77d4ad2e (C:\windows\SYSTEM32\ntdll.dll+0x6b2dad2e)
#11 0x77d4acf9 (C:\windows\SYSTEM32\ntdll.dll+0x6b2dacf9)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: breakpoint C:\b\c\b\win_asan_release\src\base\debug\debugger_win.cc:18:7 in base::debug::BreakDebugger(void)
==844:3228==ABORTING

Detailed report: https://clusterfuzz.com/testcase?key=6098683887353856

Fuzzer: inferno_js_fuzzer_c
Job Type: windows_asan_d8
Platform Id: windows

Crash Type: CHECK failure
Crash Address: 
Crash State:
  normalizer != nullptr in builtins-intl.cc
  v8::platform::PrintStackTrace
  v8::internal::Builtin_Impl_StringPrototypeNormalizeIntl
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=490547:490630

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6098683887353856


See https://github.com/google/clusterfuzz-tools for more information.

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.lkgr%2FWin_ASan_Release%2F6088%2F%2B%2Frecipes%2Fsteps%2Ffilter_build_dir%2F0%2Fstdout says

Didn't find C:\b\c\b\Win_ASan_Release\src\out\Release\icudtl.dat

See also others encountering it https://bugs.chromium.org/p/chromium/issues/detail?id=750308. This is not specific to sanitizer builds then.

 Issue 750308  and the debug output in file filter suggest that the file/files went missing already during or right after compilation.

Maybe we could install a monitoring script on one of the affected builders that tracks when exactly the file goes missing.

Any chance a windows expert can help how to read out windows syscalls to e.g. track exactly what happened to this file?

 Issue 750308  has been merged into this issue.

Need to make this available for now as I've low bandwidth and will be OOO rather often. The problem is also not V8 specific and should probably be owned by windows compilation or ninja owners.

 Issue 750591  has been merged into this issue.

Jungshik, any idea why this is only happening on windows ?

ClusterFuzz has detected this issue as fixed in range 490630:490712.

Detailed report: https://clusterfuzz.com/testcase?key=6098683887353856

Fuzzer: inferno_js_fuzzer_c
Job Type: windows_asan_d8
Platform Id: windows

Crash Type: CHECK failure
Crash Address: 
Crash State:
  normalizer != nullptr in builtins-intl.cc
  v8::platform::PrintStackTrace
  v8::internal::Builtin_Impl_StringPrototypeNormalizeIntl
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=490547:490630
Fixed: https://clusterfuzz.com/revisions?job=windows_asan_d8&range=490630:490712

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=6098683887353856


See https://github.com/google/clusterfuzz-tools for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

Please don't merge more clusterfuzz test cases into this issue to avoid clusterfuzz spam. CF will always quickly recognize the issues as fixed since the missing file shows up again in one of the next builds...

If we detect more clusterfuzz cases, maybe we can just leave a note and set them to wontfix.

> Jungshik, any idea why this is only happening on windows ?

I have no idea. I'm afraid I'm not a good owner for this issue. We need a build infra/Windows expert here. 

Who would that be? 

See also  crbug.com/752218  where we've been investigating it breaking official builders.

(not the same file, but seems like very similar behaviour where one file "goes missing" for a few builds on a windows builder)

This is also randomly breaking compilation, e.g. see:
https://build.chromium.org/p/chromium/builders/Win/builds/57915
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium%2FWin%2F57915%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

Fails in build-installer-archive.py with:
IOError: [Errno 2] No such file or directory: 'icudtl.dat'

Though, some lines above the compile output claims:
[7501/64944] COPY ../../third_party/icu/common/icudtl.dat icudtl.dat

Dirk, can you please help with an owner for this. This seems like some issue on infra side that is causing these broken builds to be generated without icudtl.dat.

This sounds like a build issue, not an infra issue, given that the file appears to be present earlier in the build. I.e., I don't think this is something most infra folks could help with and you'd likely need a build person.

brucedawson@, is this something you might have time to take a look at?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/49172f4395d4b720492322395a48335fd5b6953f

commit 49172f4395d4b720492322395a48335fd5b6953f
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Wed Aug 16 00:38:28 2017

Check whether file copy copies files

There has been a steady stream of reports of icudtl.dat being copied to
the output directory but then showing up missing. This change merely
checks, after the file is copied, to see if it is where it should be.

The problem is probably an anti-virus race condition or some-such
nonsense and it's not clear if this check will help, but it's worth a
try.

R=dpranke@chromium.org
BUG= 741603 

Change-Id: Ia585f51ef038f847c12281e13970b9e2d9b2500b
Reviewed-on: https://chromium-review.googlesource.com/614923
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#494636}
[modify] https://crrev.com/49172f4395d4b720492322395a48335fd5b6953f/build/toolchain/win/tool_wrapper.py

Re 42: Race with anti virus or maybe a disc clean-up service is the right idea I think.

The last three breakages on the Chromium win builds of this sort were all around 12 p.m. (buildbot time zone, I assume PST?). See:
https://build.chromium.org/p/chromium/builders/Win/builds/57841
https://build.chromium.org/p/chromium/builders/Win/builds/57877
https://build.chromium.org/p/chromium/builders/Win/builds/57915

Also win64:
https://build.chromium.org/p/chromium/builders/Win%20x64/builds/13866
https://build.chromium.org/p/chromium/builders/Win%20x64/builds/13800
https://build.chromium.org/p/chromium/builders/Win%20x64/builds/13768

The same goes for the buggy clusterfuzz builds.

@labs team: Any clues about a virus scan or a disc clean up service running daily at midnight on Windows? Specifically build11-m1, build61-m1 and build86-m1.

The patch from comment 42 might not really catch the problem. I think the file is there right after copying and goes missing a bit later. Some of the builds above first fail in the compile confirm no-op step, which indicates that the file was still there when dependent rules needed the file during compilation.

Is the copy procedure copying also the original file stats or does the file get a fresh "modified" time stamp? There was a similar problem on Mac once, where the original stats were kept, and a disc clean-up service treated the files as too old.

> The last three breakages on the Chromium win builds of this sort were all around 12 p.m.

Aha! Crucially, all of the builds that broken went across the 12:30 mark. That is when a disk cleanup task runs.  crbug.com/739282 . The file time stamp does *not* get updated when copying so this fits the symptoms perfectly.

I'm gonna close this as a duplicate.

Ugh, we really need to fix that.

Wow, amazing. Nice debugging!

Cool.. or not cool. But wait, do the timings really fit? Disc cleanup runs at 12:30 and the problems on the win builders occurred already at 12:15. How does that fit? Is the time-zone off by one hour?

Which builds are you seeing where the failure occurred already by 12:15? I'm seeing build end times between 00:31 and 00:38 on the six links in comment #43, typically looking like this:

Start	Thu Aug 10 00:00:08 2017
End	Thu Aug 10 00:34:01 2017

The steps after the failed steps happen almost instantly so the failure time and the end of the build time should be fairly close together.

Lol, yea... looked at the start times apparently. Never mind.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1c74f7a0e1266a6691895a741a30e201533debf4

commit 1c74f7a0e1266a6691895a741a30e201533debf4
Author: Michael Achenbach <machenbach@chromium.org>
Date: Sat Aug 19 07:21:24 2017

Revert "Add temporary output for debugging incomplete archives"

This reverts commit b985981d9f2ba3104d9e1ef1cd1c8634a1554ff3.

Reason for revert: The bug got identified. This output
is no longer needed.

Original change's description:
> Add temporary output for debugging incomplete archives
> 
> TBR=mmoroz@chromium.org
> 
> Bug:  741603 
> Change-Id: I9d3091a4ba3935ffd645aedbebfa180def66fa01
> Reviewed-on: https://chromium-review.googlesource.com/575968
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Michael Achenbach <machenbach@chromium.org>

TBR=machenbach@chromium.org,tandrii@chromium.org,mmoroz@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  741603 
Change-Id: I15dcaaf5f57f1abcea808a7ac1c11fefb369879c
Reviewed-on: https://chromium-review.googlesource.com/620566
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/1c74f7a0e1266a6691895a741a30e201533debf4/scripts/slave/recipe_modules/archive/resources/filter_build_files.py

We have made a bunch of changes on ClusterFuzz side, so resetting ClusterFuzz-Wrong label.