New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 707515 link

Starred by 4 users

Issue metadata

Status: Verified
Owner:
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug-Regression

Blocking:
issue 677043



Sign in to add a comment

base::TimeDurationFormat() fails for many locales

Project Member Reported by derat@chromium.org, Apr 1 2017

Issue description

base::TimeDurationFormat() fails when the default locale is set to en_GB and various other locales (but not all locales). This is causing  issue 677043 .

This is the failing line in base/i18n/time_formatting.cc:

icu::MeasureFormat measure_format(icu::Locale::getDefault(), u_width, status);

ICU sets |status| to U_RESOURCE_TYPE_MISMATCH.

I've uploaded a trivial test demonstrating the error at https://codereview.chromium.org/2793613004.

I'm guessing that something is wrong with ICU's resource files. I tried to build them locally using the instructions in src/third_party/icu/README.chromium but was unsuccessful.

Some ideas:

- Maybe our resource-building scripts are removing necessary data.
- Looking at the files in src/third_party/icu/source/data/locales, I see that locales like "da" and "fa" (which work) define a "generic" calendar, while en_GB only defines "gregorian". (We should be using the Gregorian calendar here, though, I assume.)
 

Comment 1 by derat@chromium.org, Apr 1 2017

Summary: base::TimeDurationFormat() fails for many locales (was: base::TimeDurationFormat() fails when default locale is en_GB)
I think that this is a full list of broken locales (out of all the ones returned by l10n_util::GetAvailableLocales()):

de-CH
en-150
en-GB
en-IN
en-NZ
en-ZA
es-419
fr-CA

Comment 2 by derat@chromium.org, Apr 1 2017

Labels: -OS-Chrome OS-All
Per try jobs with my test, this looks like it's broken on all platforms (and for the same locales, I think).

Comment 3 by js...@chromium.org, Apr 10 2017

Thank you for the report and sorry for getting to this late. 

I'll look into the ICU data. 


Comment 4 by js...@chromium.org, Apr 10 2017

> en_GB only defines "gregorian"

That should be fine (even if generic is necessary for this API) because the parent of en_GB is en_001 which has 'generic'. 

It's possible that trim_data.sh messes up something as you suspected. 


Comment 5 by js...@chromium.org, Apr 11 2017

Measure unit data come from data/unit. Even before trimming, en_001 (the parent of en_GB) does not have entries for {units,unitsNarrow}.duration.{day,hour,month,second} while it does have unitsShort.duration.*.  
en.txt does have {units, unitsNarrow}.duration.* as well as unitsShort.duration.*.   I guess the fallback path is en_GB => en_001 => en. 

Because unit*.duration.* are kept (not trimmed away), trimming does not make any difference (at least, should not; and I couldn't find any difference). 

Nonetheless, with the full ICU data, Dan's test passes while with Chrome's trimmed data it fails. 

I may be hitting a latent bug in ICU's fallback algorithm (en_GB => en_001 => en)  with Chrome's trimmed ICU data file. 

Comment 6 by js...@chromium.org, Apr 11 2017

I think I found a likely cause. With data trimming, en_001 (the parent of en_GB) ends up having an empty units and unitsNarrow. Because Chrome only needs duration and en_001 does not have any duration entry for units/unitsNarrow to start with, they become empty after trimming.  And, this empty units and unitsNarrow blocks ICU from falling back to en.txt. 

    units{
    }
    unitsNarrow{
    }

The same is true of all other locales listed in comment 1. 

Comment 7 by js...@chromium.org, Apr 11 2017

Status: Started (was: Assigned)
That was it. I rebuilt the ICU data and Dan's test passes for all locales. 

> I tried to build them locally using the instructions in src/third_party/icu/README.chromium but was unsuccessful.

Sorry about that. source/data/trnslit/trnslocal.mk should have root_subset.txt instead of css3transform.txt.  That file was left not updated. 

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/deps/icu.git/+/d5c238dcc2e801210749f3bb421b9227fe1c0948

commit d5c238dcc2e801210749f3bb421b9227fe1c0948
Author: Jungshik Shin <jshin@chromium.org>
Date: Tue Apr 11 21:58:59 2017

Update trim_data to deal with locale fallback failure for units

Delete empty units,units{Narrow,Short} blocks after trimming units data.
Empty units* blocks in en_GB and a few other locales after trimming
causes ICU to fail to fall back to get the duration data for those
locales.

In addition, fix source/data/translit/root_subset.txt. Rule*Ids block has
to be present even though it's empty. When dropping Hans-Hant transform
rules, root_subset.txt was changed to be completely empty, which broke
"components_unittests --g_test_filter=AutofillProfileComparato*" .

With these changes, regenerate ICU data files. The size is slightly smaller.

android/icudtl.dat  6573872 => 6573792
common/icudt*dat    10130560 => 10130480

BUG= 707515 , 677043 , 684609 
TEST=components_unittests --gtest_filter=AutofillProfileComparato*
TEST=ui_base_unittests --gtest_filter=L10nUtilTest.TimeDurationForm*
R=derat@chromium.org

Review-Url: https://codereview.chromium.org/2812943003 .

[modify] https://crrev.com/d5c238dcc2e801210749f3bb421b9227fe1c0948/android/icudtl.dat
[modify] https://crrev.com/d5c238dcc2e801210749f3bb421b9227fe1c0948/common/icudtb.dat
[modify] https://crrev.com/d5c238dcc2e801210749f3bb421b9227fe1c0948/common/icudtl.dat
[modify] https://crrev.com/d5c238dcc2e801210749f3bb421b9227fe1c0948/scripts/trim_data.sh
[modify] https://crrev.com/d5c238dcc2e801210749f3bb421b9227fe1c0948/source/data/translit/root_subset.txt

Project Member

Comment 9 by bugdroid1@chromium.org, Apr 11 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a8550bc1391772685da6e86adc72a587bd964289

commit a8550bc1391772685da6e86adc72a587bd964289
Author: derat <derat@chromium.org>
Date: Tue Apr 11 22:11:31 2017

Add test for TimeDurationFormat() error.

base::TimeDurationFormat() fails due to a
U_RESOURCE_TYPE_MISMATCH error from ICU when called with
certain locals like en_GB due to missing resource data.
Add a test (disabled for now) to verify that it works
for all supported locales.

BUG= 677043 , 707515 

Review-Url: https://codereview.chromium.org/2793613004
Cr-Commit-Position: refs/heads/master@{#463796}

[modify] https://crrev.com/a8550bc1391772685da6e86adc72a587bd964289/ui/base/l10n/l10n_util_unittest.cc

Project Member

Comment 10 by bugdroid1@chromium.org, Apr 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cf65b91bc7496b3dd2c958c890984e73c26ad22c

commit cf65b91bc7496b3dd2c958c890984e73c26ad22c
Author: jshin <jshin@chromium.org>
Date: Wed Apr 12 00:34:12 2017

Roll third_party/icu from 450be73 to b34251f

http://chromium.googlesource.com/chromium/deps/icu.git/+log/450be73..b34251f

Changes include (along with minor data build script updates):

  1. f0449ad Update IANA timezone db to 2017a from 2016i
  2. c781b5f Drops unused Hans-Hant ICU transliteration data.
     ( cuts down the ICU data file size by ~25kB.)
  3. d5c238d Update trim_data to deal with locale fallback failure for units
     and fix root_subset.txt in ICU's transliteration configuration.
  4. b34251f Update timezone db to 2017b

BUG= 684609 ,473288, 707515 , 677043 
TEST=components_unittests --gtest_filter=AutofillProfileComparato*
TEST=ui_base_unittests --gtest_filter=L10nUtilTest.TimeDurationForm*
TEST=For timezone change test, see CL #1 and #4 TEST lines.
TBR=riesa@chromium.org,derat@chromium.org

Review-Url: https://codereview.chromium.org/2755963002
Cr-Commit-Position: refs/heads/master@{#463856}

[modify] https://crrev.com/cf65b91bc7496b3dd2c958c890984e73c26ad22c/DEPS
[modify] https://crrev.com/cf65b91bc7496b3dd2c958c890984e73c26ad22c/ui/base/l10n/l10n_util_unittest.cc

Comment 11 by js...@chromium.org, Apr 12 2017

Will ask for merge to M58 after baking in canary for a day or two. Just a DEPS roll would suffice in M58 branch. 


Comment 12 by js...@chromium.org, Apr 18 2017

Labels: Merge-Request-58
Verified in a canary build (en-GB locale: in the ash-tray, I confirmed that '1% - 2:28 until full' instead of '1% -  until full' is shown;  see  bug 677043 ). I also confirmed it in es-419 locale. 

Requesting for merge to M58.  It's rather late in the cycle, but it has regressed and affects en-GB and es-419 users. (again, see  bug 677043 ). 



Comment 13 by js...@chromium.org, Apr 18 2017

Status: Verified (was: Started)
Project Member

Comment 14 by sheriffbot@chromium.org, Apr 18 2017

Labels: -Merge-Request-58 Merge-Review-58 Hotlist-Merge-Review
This bug requires manual review: We are only 6 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
There is a huge risk in taking a change post branch as we're cutting M58 stable RC today. Can you please confirm if this is critical? Based on comment 12, seems like it's already verified in Canary. Is there enough unit test coverage for this?
And also seems like this bug has been there since M56? My recommendation is to wait until M59, but if you think it's critical and it's a safe merge, please let us know. 

Comment 17 by js...@chromium.org, Apr 18 2017

Sorry that I forgot to give how safe/complicated a cherry-pick is. 

It'll be just a ICU data rebuild with no code change. Dan added a unittest to verify the fix (ICU).  

So,  I think it's pretty safe. It's unfortunate that it's been broken since M56, but it's  bad for en-GB and es-419 users.(no remaining hours for battery. how long to wait until full). 
Cc: bhthompson@chromium.org
Since change is affecting Chrome OS more, adding bhthompson@ for merge review. 
Labels: -Hotlist-Merge-Review -Merge-Review-58 Merge-Approved-58
As we still have a beta before stable for CrOS, if engineering is confident this is a clean cut safe fix, we can merge this if it is only for CrOS. 
Project Member

Comment 20 by bugdroid1@chromium.org, Apr 20 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/buildspec/+/cd0a1e71a466bf114861203e987fb7ed07e5efd6

commit cd0a1e71a466bf114861203e987fb7ed07e5efd6
Author: Jungshik Shin <jungshik@google.com>
Date: Thu Apr 20 19:47:02 2017

Comment 21 by js...@chromium.org, Apr 21 2017

Labels: merge-merged-3029

Comment 22 by js...@chromium.org, Apr 21 2017

Labels: -Merge-Approved-58

Sign in to add a comment