Android: Fix compilation of localized resources that use ISO-639-2 and BCP-47 qualifiers. |
||
Issue descriptionWhile working on https://crbug.com/897056, I discovered that our "compile_resources.py" script will ignore some input localized resource strings. In particular, Chrome / ChromeModern / Monochrome are missing 8 localized strings for the Filipino locale (identified as 'fil' by Chrome). The reason why comes from how localized resources are provided to the "aapt" command. The latter takes a list of directories, each one of them having a structure that looks like: res/values-<qualifier1>/strings.xml // Strings localized for locale // identified by <qualifier1> res/values-<qualifier2>/strings.xml // Localized for <qualifier2> ... res/values-<qualifierN>/strings.xml // Localized for <qualifierN> Where each <qualifier> is used to identify a given locale, according to the rules described at [1], which can be summarized as: * A 2-letter ISO 639-1 code, potentially followed by "-r" and a capitalized region code. Examples: "en" (English) "en-rUS" (American English) "en-rGB" (British English) * A 3-letter ISO 639-2 code, potentially followed by "-r" and a capitalized region code. Examples: "fil" (Filipino) "fil-rPH" (Filipino as spoken in Phillipines) NOTE: This format is only supported since Android L/5.0! * A "b+"prefix following by an ISO 639 code (2 or 3 letters) followed by an optional capitalized region tag and other optional tags, separated with plus signs, e.g.: "b+en+GB" (British English) "b+ja+Latn" (Japanese transliterated to Latin) "b+sr+Latn+fonapi" (Serbian written in Latin using IPA) "b+sr+Cyrl" (Serbian written in cyrillic characters) NOTE: This format is only supported since Android N/7.0! Also note that some random tags can be suffixed to the first two formats, even though this is not documented. A good example is the Android framework resources that provide shorter versions of localized strings adapted to being displayed on a watch, For an example see [2] which shows the following choices: values-hu/ values-hu-rHU/ values-hu-watch/ values-hy-rAM/ values-hy-rAM-watch/ The thing is that compile_resources.py will only support the first format above (i.e. 2-letter language codes with optional region code), and omit any input resource directory corresponding to qualifiers of the two other formats. In practice, this means we are missing 8 strings from the following file that uses "values-fil": https://cs.chromium.org/chromium/src/third_party/feed/src/src/main/java/com/google/android/libraries/feed/basicstream/internal/viewholders/res/values-fil/strings.xml?g=0 The effect is that whenever these strings need to be displayed when the user is using the Filipino locale on their device, the English versions will be displayed instead. [1] https://developer.android.com/guide/topics/resources/providing-resources#LocaleQualifier [2] https://gitlab.e.foundation/e/devices/samsung/i9100/android_frameworks_base/tree/eea06acdbbdcc5a808f4b0c42d5f94b4693df045/core/res/res
,
Dec 10
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d71432dde17693204a30f801a9209548ddfcddca commit d71432dde17693204a30f801a9209548ddfcddca Author: David 'Digit' Turner <digit@google.com> Date: Mon Dec 10 10:08:05 2018 android: Better locale resources selection. This CL modifies compile_resources.py to better select which locale-related resource files should be packaged into the final output archive. Before the patch "aapt -c <locale-list>" would be used to tell aapt to only use resources from a specific locale whitelist, but this turns out to be unreliable (see http://crbug.com/897056). The patch instead removes all non-relevant locale resource files from the temporary resource directories themselves before passing their list to aapt, thus avoiding any surprises. This CL should *not* change the resources embedded in final APKs, and thus not modify their size. In particular the script still fails to embed some Filipino and Indonesian localized strings. This will be addressed by a future CL based on this one. + resource_utils.py: Provide ToAndroidLocaleName() and removes CHROME_TO_ANDROID_LOCALE_MAP from public view. And update calling code in compile_resources.py, create_app_bundle.py and create_ui_locale_resources.py. + resource_utils.py: Add ToChromiumLocaleName() which will be used in a future CL, which does the opposite, and also supports ISO 639-2 language codes (supported since Android 5.0/Lollipop) and BCP 47 language tags (supported since Android 7.0/Nougat). + resource_utils.py: Add IsAndroidLocaleQualifier() and FIndLocaleInStringResourceFilePath() and GetRTxtStringResourceNames() which will be used in a future CL. + resource_utils_test.py: New unit-tests for resource_utils.py to track regressions in the future. Ensure it is run during 'git cl presubmit' BUG= 912044 R=agrieve@chromium.org,estevenson@chromium.org,yfriedman@chromium.org,tedchoc@chromium.org,twellington@chromium.org Change-Id: If7da6990fed5a8c52aa1443863da94f91ecb1b22 Reviewed-on: https://chromium-review.googlesource.com/c/1367665 Commit-Queue: David Turner <digit@chromium.org> Reviewed-by: agrieve <agrieve@chromium.org> Reviewed-by: Theresa <twellington@chromium.org> Cr-Commit-Position: refs/heads/master@{#615075} [modify] https://crrev.com/d71432dde17693204a30f801a9209548ddfcddca/build/android/PRESUBMIT.py [modify] https://crrev.com/d71432dde17693204a30f801a9209548ddfcddca/build/android/gyp/compile_resources.py [modify] https://crrev.com/d71432dde17693204a30f801a9209548ddfcddca/build/android/gyp/create_app_bundle.py [modify] https://crrev.com/d71432dde17693204a30f801a9209548ddfcddca/build/android/gyp/util/resource_utils.py [add] https://crrev.com/d71432dde17693204a30f801a9209548ddfcddca/build/android/gyp/util/resource_utils_test.py [modify] https://crrev.com/d71432dde17693204a30f801a9209548ddfcddca/ui/android/build/create_ui_locale_resources.py
,
Dec 11
Actually fixed by https://chromium-review.googlesource.com/c/chromium/src/+/1367730 which wrongly listed this entry as 921044 instead of 921044. Sorry about that! |
||
►
Sign in to add a comment |
||
Comment 1 by digit@google.com
, Dec 5Additional note: The reason compile_resources.py only supports 2-letter language code comes from several facts: - Chromium uses locale names that can be either <lang> or <lang>-<region> where <lang> is an ISO 639 language code (2 or 3 letters), and <region> a capitalized region code (e.g. 'en-GB' for Chromium instead of 'en-rGB' for Android) - Android uses deprecated codes for a few languages, while Chromium uses newer ones, i.e.: Language Chromium Android Yiddish: yi ji Hebrew: he iw Indonesian: id in - Android didn't support the Filipino language (ISO 639-2 'fil') before Android L, but it supported Tagalog (ISO 639-1 'tl') which is a variant of the same language (but with less speakers in the Philippines). As such, Filipino localized strings are typically associated with the 'tl' locale. - Chrome uses 'es-419' to identify the locale for Spanish as spoken in the US, known on Android as 'es-rUS'. In practice, compile_resources.py translates the list of Chromium locale names (e.g. ['en', 'fil', 'he', 'es-419']) into Android ones (e.g. ['en', 'tl', 'iw', 'es-rUS']) and use the result to filter which resource files go into the final output. E.g. from .xml files under: values-en/ values-tl/ values-iw/ values-es-rUS/ The problem is that if a package provides 'values-fil/' instead of 'values-tl/', then these files are currently simply omitted by 'compile_resources.py'.