Windows GPU FYI bots periodically fail all tests with ICU failure. |
|||||
Issue descriptionIt appears to fail with this signature in about 20% of builds. Example failures: https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28Intel%20HD%20530%29/builds/885 https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28AMD%20R7%20240%29/builds/973 The failure is flaky per run, either all tests fail with the same crash or all pass. This seems to affect most of the Windows bots but a couple appear to be immune (https://build.chromium.org/p/chromium.gpu.fyi/builders/Win7%20Release%20%28AMD%29). Looks like this started as far back as June 12th. Crash stack: [1184:4772:0613/012832.159:17372312:ERROR:icu_util.cc(173)] Invalid file descriptor to ICU data received. [1184:4772:0613/012832.160:17372312:FATAL:icu_util.cc(292)] Check failed: result. Backtrace: base::debug::StackTrace::StackTrace [0x0015E757+55] base::debug::StackTrace::StackTrace [0x0015E69A+10] base::i18n::InitializeICU [0x003BB552+354] base::TestSuite::Initialize [0x001A6FAE+318] base::TestSuite::Run [0x001A72D0+25] base::internal::Invoker<base::internal::BindState<int (__cdecl*)(base::TestSuite *),base::internal::UnretainedWrapper<base::TestSuite> >,int __cdecl(void)>::Run [0x0013F53F+63] base::internal::Invoker<base::internal::BindState<int (__cdecl*)(base::TestSuite *),base::internal::UnretainedWrapper<base::TestSuite> >,int __cdecl(void)>::Run [0x0013F50E+14] base::UnitTestLauncherDelegate::GetTests [0x001A557C+796] base::LaunchUnitTestsWithOptions [0x001A55D7+73] main [0x0013F5E0+141] __scrt_common_main_seh [0x0047088B+249] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253) BaseThreadInitThunk [0x76A838F4+36] RtlUnicodeStringToInteger [0x77435DE3+595] RtlUnicodeStringToInteger [0x77435DAE+542] Looking for possible regressing CLs now..
,
Jun 20 2017
Noticing another possible issue with locals on the ANGLE CQ Mac bot: https://build.chromium.org/p/tryserver.chromium.angle/builders/mac_angle_rel_ng/builds/5097 seeing [13793:775:0620/101136.912278:FATAL:chrome_browser_main_mac.mm(117)] Check failed: !loaded_locale.empty(). Default locale could not be found https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.angle%2Fmac_angle_rel_ng%2F5097%2F%2B%2Frecipes%2Fsteps%2Ftab_capture_end2end_tests_on_Intel_GPU_on_Mac__with_patch__on_Mac-10.12%2F0%2Fstdout
,
Jun 20 2017
I'm at a loss to understand how all of the jobs in https://build.chromium.org/p/tryserver.chromium.angle/builders/mac_angle_rel_ng/builds/5097 could have failed unless the binary itself was broken. This was a Swarmed task so all of the tests likely ran on different machines. Is it possible that one of the earlier patch sets on https://chromium-review.googlesource.com/c/529707/ was actually buggy? The failure occurred on patch set 21 but I do see that it failed and then passed with the same code. I'm not seeing the same sorts of failures for example on the mac optional tryserver: https://luci-milo.appspot.com/buildbot/tryserver.chromium.mac/mac_optional_gpu_tests_rel/?limit=200 or even on mac_chromium_rel_ng: https://luci-milo.appspot.com/buildbot/tryserver.chromium.mac/mac_chromium_rel_ng/?limit=200 Could this be a problem with how mac_angle_rel_ng is configured?
,
Jun 20 2017
If you run one of the failing task locally or you retry it, does it fail again?
,
Jun 28 2017
Not sure if it's the same issue, having Mac failures again today, eg: https://build.chromium.org/p/tryserver.chromium.angle/builders/mac_angle_rel_ng/builds/5230
,
Jun 28 2017
I've just clicked "Retry" for the context_lost_tests failure in this job: https://luci-milo.appspot.com/buildbot/tryserver.chromium.angle/mac_angle_rel_ng/5230 It was retried here: https://chromium-swarm.appspot.com/task?id=37096495a6cadc10&refresh=10&show_raw=1 It failed again. Something is broken with the binaries that were built from this job. How do we get to the bottom of this?
,
Jul 28 2017
Geoff, could you please find an owner to debug this?
,
Aug 10 2017
Appears to be fixed. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by geoffl...@chromium.org
, Jun 20 2017