New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 861850 link

Starred by 2 users

Issue metadata

Status: Closed
Owner:
Closed: Jan 15
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Fuchsia
Pri: 3
Type: Bug



Sign in to add a comment

Flaky base_unittests failures on fuchsia_x64 trybot

Project Member Reported by kbr@chromium.org, Jul 9

Issue description

Example failures:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/58611
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/58619
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/fuchsia_x64/58597

Failures seen:

TimeFormattingTest.TimeFormatTimeOfDayDE
NumberFormattingTest.FormatPercent
TaskSchedulerSingleThreadTaskRunnerManagerStartTest.PostTaskBeforeStart

Couple of log excerpts:

[ RUN      ] TimeFormattingTest.TimeFormatTimeOfDayDE
../../base/i18n/time_formatting_unittest.cc:218: Failure
Expected equality of these values:
  clock12h_pm
    Which is: 3:42 nachm.
  TimeFormatTimeOfDayWithHourClockType(time, k12HourClock, kKeepAmPm)
    Which is: 3:42 PM
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/time_formatting_unittest.cc:215

[00028.228] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
[  FAILED  ] TimeFormattingTest.TimeFormatTimeOfDayDE (3 ms)


[ RUN      ] TaskSchedulerSingleThreadTaskRunnerManagerStartTest.PostTaskBeforeStart
[80448:6893521:0709/183824.522289:46254667:FATAL:scheduler_worker.cc(92)] Check failed: !should_exit_.IsSet().
#00: base::debug::StackTrace::StackTrace(unsigned long) at ../../base/debug/stack_trace_fuchsia.cc:173
#01: endl<char, std::__2::char_traits<char> > at ../../buildtools/third_party/libc++/trunk/include/ostream:1001
      (inlined by) operator<< at ../../buildtools/third_party/libc++/trunk/include/ostream:195
      (inlined by) ~LogMessage at ../../base/logging.cc:593
#02: WakeUp at ../../base/task_scheduler/scheduler_worker.cc:93
#03: Release at ../../base/memory/ref_counted.h:385
      (inlined by) Release at ../../base/memory/scoped_refptr.h:284
      (inlined by) ~scoped_refptr at ../../base/memory/scoped_refptr.h:208
      (inlined by) Start at ../../base/task_scheduler/scheduler_single_thread_task_runner_manager.cc:430
#04: TestBody at ../../base/task_scheduler/scheduler_single_thread_task_runner_manager_unittest.cc:655


Marking P1; the CQ must be reliable.

 
Components: Internals>PlatformIntegration
Labels: M-69
Owner: w...@chromium.org
Status: Started (was: Untriaged)
Auto-assigning to diagnose & triage.
Logs from the number formatting tests in 58611 and 58597:

[ RUN      ] NumberFormattingTest.FormatPercent
../../base/i18n/number_formatting_unittest.cc:130: Failure
Expected equality of these values:
  WideToUTF16(cases[i].expected_arabic)
    Which is: ٠٪؜
  FormatPercent(cases[i].number)
    Which is: 0‎%‎
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/number_formatting_unittest.cc:129

[00027.824] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
../../base/i18n/number_formatting_unittest.cc:130: Failure
[00027.905] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
Expected equality of these values:
[00027.943] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
  WideToUTF16(cases[i].expected_arabic)
    Which is: ٤٢٪؜
  FormatPercent(cases[i].number)
    Which is: 42‎%‎
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/number_formatting_unittest.cc:129

[00027.981] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
../../base/i18n/number_formatting_unittest.cc:130: Failure
[00028.008] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
Expected equality of these values:
[00028.098] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
  WideToUTF16(cases[i].expected_arabic)
    Which is: ١٬٠٢٤٪؜
  FormatPercent(cases[i].number)
    Which is: 1,024‎%‎
Stack trace:
#00: CurrentOsStackTraceExceptTop at ../../third_party/googletest/src/googletest/src/gtest.cc:810
#01: operator= at ../../third_party/googletest/src/googletest/src/gtest.cc:383
#02: TestBody at ../../base/i18n/number_formatting_unittest.cc:129

[00028.104] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
[  FAILED  ] NumberFormattingTest.FormatPercent (6 ms)
[00028.174] pkgsvr: 2018/07/09 18:31:18 pkgfs:unsupported(/packages/base_unittests/0): dir unlink "app.log"
[635/3112] NumberFormattingTest.FormatPercent (6 ms)

Both NumberFormattingTest.FormatPercent and TimeFormattingTest.TimeFormatTimeOfDayDE were modified as part of an ICU dependency roll, at about the time that those flakes were observed, which is suspicious, possibly indicating a missing dependency e.g. on ICU data files, resulting in stale ones being used for some time immediately after the roll?

Broke out the TaskScheduler report to  issue 862582 , since that looks to be something different.
Cc: sergeyu@chromium.org
Owner: js...@chromium.org
Status: Assigned (was: Started)
jshin: Looking at this flake, it seems to me that the failures are would be consistent with us running an out-of-date base_unittests against the newly-rolled ICU data - could you take a look at the NumberFormattingTest.FormatPercent failures from comment #2 and confirm that assessment?

If that fits these failures (which seem to be one-offs; there are no other reports on try-flakes, AFAICT) then we'll need to look for a missing dependency, likely somewhere in our Fuchsia package creation logic.
> consistent with us running an out-of-date base_unittests against the newly-rolled ICU data 

Yes, fuschia's base test is out-of-date. The expected values (for both DateFormat and NumberFormat) in that outdated base_unittests (as used by Fuschia) have to be updated to the trunk.   

Why does Fuschia have a separate update-cycle for base_unittests? 
Don't Fuscha trybots use the latest trunk of base_unittests ? 

> If that fits these failures (which seem to be one-offs; there are no other reports on try-flakes, AFAICT) then we'll need to look for a missing dependency

I'm not a good person to deal with the abov. Whoever that is has to take this bug. 
Labels: -Pri-1 -M-69 Pri-3
Owner: kmarshall@chromium.org
Re #4: We should have seen our base_unittests rebuilt in-sync with the ICU data-set change, so that things would have matched, but for some reason we didn't, and we're not sure why, but a missing GN dependency seems the obvious issue.

Re-assigning to someone on the Chrome-Fuchsia team to look at as a cleanup task, since this is unlikely to keep happening unless that dependency is changed often.
Cc: cira@chromium.org cira@google.com
It seems that Fuchsia has a separate forked tree of ICU and it is still 61.1 instead of 62.1. I wonder why Fuchsia forked ICU from Chromium.   There must be up-sides to that, but there are also downside: 

Fuchsia's ICU update would lag behind Chromium's as found here. ICU bug fixes/security fixes wouldn't be propagated in time to Fuschia. 

Of course, there'd be no issue if Fuchsia's copy of ICU is updated as frequently as Chrome's :-)
Re #6: Are we actually using the system-provided ICU in the Chromium build, though?
Cc: scottmg@chromium.org
To clarify, do we think the cause was an out of date base_unittests, or an updated base_unittests that was running against an old ICU data file?

If the latter, I wonder if this could be that we're loading the wrong .so (perhaps only in tests?) We do build icuuc and icui18n, but they're renamed because Fuchsia provides system ones in /system/lib already, that I assume would be 61 or older still.
Status: Closed (was: Assigned)

Sign in to add a comment