Flaky base_unittests failures on fuchsia_x64 trybot |
||||||
Issue descriptionExample failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/58611 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/58619 https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/58597 Failures seen: TimeFormattingTest.TimeFormatTimeOfDayDE NumberFormattingTest.FormatPercent TaskSchedulerSingleThreadTaskRunnerManagerStartTest.PostTaskBeforeStart Couple of log excerpts: [ RUN ] TimeFormattingTest.TimeFormatTimeOfDayDE ../../base/i18n/time_formatting_unittest.cc:218: Failure Expected equality of these values: clock12h_pm Which is: 3:42 nachm. TimeFormatTimeOfDayWithHourClockType(time, k12HourClock, kKeepAmPm) Which is: 3:42 PM Stack trace: #00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810 #01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383 #02: TestBody at ../../base/i18n/time_formatting_unittest.cc:215 [00028.228] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log" [ FAILED ] TimeFormattingTest.TimeFormatTimeOfDayDE (3 ms) [ RUN ] TaskSchedulerSingleThreadTaskRunnerManagerStartTest.PostTaskBeforeStart [80448:6893521:0709/183824.522289:46254667:FATAL:scheduler_worker.cc(92)] Check failed: !should_exit_.IsSet(). #00: base::debug::StackTrace::StackTrace(unsigned long) at ../../base/debug/stack_trace_fuchsia.cc:173 #01: endl<char, std::__2::char_traits<char> > at ../../buildtools/third_party/libc++/trunk/include/ostream:1001 (inlined by) operator<< at ../../buildtools/third_party/libc++/trunk/include/ostream:195 (inlined by) ~LogMessage at ../../base/logging.cc:593 #02: WakeUp at ../../base/task_scheduler/scheduler_worker.cc:93 #03: Release at ../../base/memory/ref_counted.h:385 (inlined by) Release at ../../base/memory/scoped_refptr.h:284 (inlined by) ~scoped_refptr at ../../base/memory/scoped_refptr.h:208 (inlined by) Start at ../../base/task_scheduler/scheduler_single_thread_task_runner_manager.cc:430 #04: TestBody at ../../base/task_scheduler/scheduler_single_thread_task_runner_manager_unittest.cc:655 Marking P1; the CQ must be reliable.
,
Jul 11
Logs from the number formatting tests in 58611 and 58597:
[ RUN ] NumberFormattingTest.FormatPercent
../../base/i18n/number_formatting_unittest.cc:130: Failure
Expected equality of these values:
WideToUTF16(cases[i].expected_arabic)
Which is: ٠٪
FormatPercent(cases[i].number)
Which is: 0%
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/number_formatting_unittest.cc:129
[00027.824] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
../../base/i18n/number_formatting_unittest.cc:130: Failure
[00027.905] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
Expected equality of these values:
[00027.943] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
WideToUTF16(cases[i].expected_arabic)
Which is: ٤٢٪
FormatPercent(cases[i].number)
Which is: 42%
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/number_formatting_unittest.cc:129
[00027.981] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
../../base/i18n/number_formatting_unittest.cc:130: Failure
[00028.008] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
Expected equality of these values:
[00028.098] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
WideToUTF16(cases[i].expected_arabic)
Which is: ١٬٠٢٤٪
FormatPercent(cases[i].number)
Which is: 1,024%
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/number_formatting_unittest.cc:129
[00028.104] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
[ FAILED ] NumberFormattingTest.FormatPercent (6 ms)
[00028.174] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
[635/3112] NumberFormattingTest.FormatPercent (6 ms)
Both NumberFormattingTest.FormatPercent and TimeFormattingTest.TimeFormatTimeOfDayDE were modified as part of an ICU dependency roll, at about the time that those flakes were observed, which is suspicious, possibly indicating a missing dependency e.g. on ICU data files, resulting in stale ones being used for some time immediately after the roll?
Broke out the TaskScheduler report to issue 862582 , since that looks to be something different.
,
Jul 11
jshin: Looking at this flake, it seems to me that the failures are would be consistent with us running an out-of-date base_unittests against the newly-rolled ICU data - could you take a look at the NumberFormattingTest.FormatPercent failures from comment #2 and confirm that assessment? If that fits these failures (which seem to be one-offs; there are no other reports on try-flakes, AFAICT) then we'll need to look for a missing dependency, likely somewhere in our Fuchsia package creation logic.
,
Jul 17
> consistent with us running an out-of-date base_unittests against the newly-rolled ICU data Yes, fuschia's base test is out-of-date. The expected values (for both DateFormat and NumberFormat) in that outdated base_unittests (as used by Fuschia) have to be updated to the trunk. Why does Fuschia have a separate update-cycle for base_unittests? Don't Fuscha trybots use the latest trunk of base_unittests ? > If that fits these failures (which seem to be one-offs; there are no other reports on try-flakes, AFAICT) then we'll need to look for a missing dependency I'm not a good person to deal with the abov. Whoever that is has to take this bug.
,
Jul 17
Re #4: We should have seen our base_unittests rebuilt in-sync with the ICU data-set change, so that things would have matched, but for some reason we didn't, and we're not sure why, but a missing GN dependency seems the obvious issue. Re-assigning to someone on the Chrome-Fuchsia team to look at as a cleanup task, since this is unlikely to keep happening unless that dependency is changed often.
,
Jul 18
It seems that Fuchsia has a separate forked tree of ICU and it is still 61.1 instead of 62.1. I wonder why Fuchsia forked ICU from Chromium. There must be up-sides to that, but there are also downside: Fuchsia's ICU update would lag behind Chromium's as found here. ICU bug fixes/security fixes wouldn't be propagated in time to Fuschia. Of course, there'd be no issue if Fuchsia's copy of ICU is updated as frequently as Chrome's :-)
,
Jul 18
Re #6: Are we actually using the system-provided ICU in the Chromium build, though?
,
Sep 4
To clarify, do we think the cause was an out of date base_unittests, or an updated base_unittests that was running against an old ICU data file? If the latter, I wonder if this could be that we're loading the wrong .so (perhaps only in tests?) We do build icuuc and icui18n, but they're renamed because Fuchsia provides system ones in /system/lib already, that I assume would be 61 or older still.
,
Jan 15
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by w...@chromium.org
, Jul 11Labels: M-69
Owner: w...@chromium.org
Status: Started (was: Untriaged)