New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 897056 link

Starred by 3 users

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 3
Type: Bug



Sign in to add a comment

Fix Android locale-related resources compilation process for Monochrome and SystemWebView

Project Member Reported by digit@google.com, Oct 19

Issue description

I had a very hard time implementing the proper CLs to add extra language strings to Android app bundles (but not regular APKs), during my work for http://crbug.com/882860, only to discover that the way we package locale-related UI string resources does *not* work correctly.

More specifically:

1) Chrome doesn't ship all locales on Android in order to save on disk space for regular APKs (see http://crbug.com/369218).

This is controlled by the android_chrome_omitted_locales list defined in build/config/locales.gni [1], which currently lists 9 locales.

2) However, UI strings for these omitted locales *must* be included in the system webview feature, which is provided by Chrome, as two separate APKs:

   SystemWebView.apk   (for Android L+M)
   Monochrome.apk      (for Android N+)

In practice, this means that:

  SystemWebView.apk should contain resource strings for all
  supported locales.

  Monochrome.apk should not contain resource strings for the omitted
  locales, unless they are used by the system webview feature.

Several places in our build system are impacted by this to implement this. To see them, use code search to look for 'locales', 'aapt_locale_whitelist' and 'android_chrome_omitted_locales' in our GN files) [2]

I added a debugging mode to the compile_resources.py script recently [3] and discovered that the APKs we currently generate do *not* have the right Android resources in their resources.arsc file, i.e.:

  - The UI strings for the 9 omitted locales are *never* included
    in either SystemWebView.apk or Monochrome.apk

  - But _other_ UI strings for the same 9 omitted locales are
    included, and come from the following prebuilt packages that
    are linked into Chromium:

      - com_android_support_appcompat_v7_java
      - com_android_support_mediarouter_v7_java
      - com_android_support_support_compat_java

    Our compile_resources.py script is supposed to remove these
    from the final compiled resources (using the AAPT -c flag), but
    for some reason, this seems to fail horribly.

    These "dead strings" are never used and just add 80 kiB to the
    .apk size needlessly.

You can check this using "aapt dump resources out/Release/apks/SystemWebView.apk" for example, and look for "config bn", which currently lists:

      config bn:
        resource 0x020c0000 com.android.webview:string/abc_action_bar_home_description: t=0x03 d=0x000007f8 (s=0x0008 r=0x00)
        resource 0x020c0001 com.android.webview:string/abc_action_bar_up_description: t=0x03 d=0x000007eb (s=0x0008 r=0x00)
        resource 0x020c0002 com.android.webview:string/abc_action_menu_overflow_description: t=0x03 d=0x000007ea (s=0x0008 r=0x00)
        resource 0x020c0003 com.android.webview:string/abc_action_mode_done: t=0x03 d=0x000007f7 (s=0x0008 r=0x00)
        resource 0x020c0004 com.android.webview:string/abc_activity_chooser_view_see_all: t=0x03 d=0x000007f6 (s=0x0008 r=0x00)
        resource 0x020c0005 com.android.webview:string/abc_activitychooserview_choose_application: t=0x03 d=0x000007ec (s=0x0008 r=0x00)
        resource 0x020c0006 com.android.webview:string/abc_capital_off: t=0x03 d=0x000007f3 (s=0x0008 r=0x00)
        resource 0x020c0007 com.android.webview:string/abc_capital_on: t=0x03 d=0x000007f2 (s=0x0008 r=0x00)
        resource 0x020c0014 com.android.webview:string/abc_search_hint: t=0x03 d=0x000007e9 (s=0x0008 r=0x00)
        resource 0x020c0015 com.android.webview:string/abc_searchview_description_clear: t=0x03 d=0x000007f0 (s=0x0008 r=0x00)
        resource 0x020c0016 com.android.webview:string/abc_searchview_description_query: t=0x03 d=0x000007ee (s=0x0008 r=0x00)
        resource 0x020c0017 com.android.webview:string/abc_searchview_description_search: t=0x03 d=0x000007f1 (s=0x0008 r=0x00)
        resource 0x020c0018 com.android.webview:string/abc_searchview_description_submit: t=0x03 d=0x000007ef (s=0x0008 r=0x00)
        resource 0x020c0019 com.android.webview:string/abc_searchview_description_voice: t=0x03 d=0x000007f4 (s=0x0008 r=0x00)
        resource 0x020c001a com.android.webview:string/abc_shareactionprovider_share_with: t=0x03 d=0x000007ed (s=0x0008 r=0x00)
        resource 0x020c001b com.android.webview:string/abc_shareactionprovider_share_with_application: t=0x03 d=0x000007e8 (s=0x0008 r=0x00)
        resource 0x020c001c com.android.webview:string/abc_toolbar_collapse_description: t=0x03 d=0x000007f5 (s=0x0008 r=0x00)
        resource 0x020c005e com.android.webview:string/search_menu_title: t=0x03 d=0x000007f1 (s=0x0008 r=0x00)
        resource 0x020c005f com.android.webview:string/status_bar_notification_info_overflow: t=0x03 d=0x000007f9 (s=0x0008 r=0x00)

All of these entries are from the compatibility libraries, and none of the android_webview related ones are present :-(

Note that the issue only affects Android UI string resources.
Localized strings accessed from C++ are stored in .pak files as
assets in the APK, and are subject to their own, but similar rules, as well, but are handled properly.

This entry is to better diagnose and fix the issue. We also probably need better regression tests to verify that the content of said APKs is what it should be :-/


[1] https://cs.chromium.org/chromium/src/build/config/locales.gni?rcl=16509560384017dea91f2833c1ffacd3116e1a9c&l=8

[2] https://cs.chromium.org/search/?q=(%22locales%22+OR+%22android_chrome_omitted_locales%22+OR+aapt_locale_whitelist)+AND+file:gn&sq=package:chromium&type=cs

[3] https://chromium-review.googlesource.com/c/chromium/src/+/1288436
 
Components: Mobile>WebView
If this is the same issue that I've discussed with various folks on various bugs before (can look some up if needed but can't easily find them right now it seems), then the problem is not related to compilation at all: it's that last time I checked we *don't translate any of those strings to the extra languages in the first place*. :(

See https://cs.chromium.org/chromium/src/android_webview/java/strings/translations/ for an example, though it also applies to content and other things webview uses.

The native strings are translated through Chrome's normal translation process and get translated into all languages that desktop Chrome supports, which does include the 9 extra locales that WebView supports, which is why those work.

The Java UI strings are translated via a slightly different process which *only* gets performed for the Chrome-on-Anndroid supported locales.

It's possible this isn't the only issue, of course, and that even if they were translated the build may not work correctly, but the translations definitely don't appear to be there.
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 19

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70b843c9e6284b1a95c964a81efbb4f2a67ee89d

commit 70b843c9e6284b1a95c964a81efbb4f2a67ee89d
Author: David 'Digit' Turner <digit@google.com>
Date: Fri Oct 19 16:31:39 2018

build: Add android_debug_resources_temp_dir

Normally, compiling Android resources requires unpacking
and processing resource dependencies into a temporary
directory before sending the result to 'aapt2 compile',
then getting rid of these intermediate files.

To help debug issues related to Android resource compilation,
this CL introduces a new GN variable that can be set in your
args.gn, to force the build to put all such build directories
under a top-level directory, and not removing intermediate
files in it.

This should allow inspecting the intermediate files to better
understand what's going on.

This is needed to debug issues that appeared in my attempt
to change the resources stored in APKs versus bundles, e.g.:
https://chromium-review.googlesource.com/c/chromium/src/+/1270947

BUG=879228,882860,897056
R=agrieve@chromium.org, estevenson@chromium.org,yfriedman@chromium.org

Change-Id: I76c79a893b0bbdb56ba685f4304da929d11c7e05
Reviewed-on: https://chromium-review.googlesource.com/c/1288436
Reviewed-by: agrieve <agrieve@chromium.org>
Commit-Queue: David Turner <digit@chromium.org>
Cr-Commit-Position: refs/heads/master@{#601178}
[modify] https://crrev.com/70b843c9e6284b1a95c964a81efbb4f2a67ee89d/build/android/gyp/compile_resources.py
[modify] https://crrev.com/70b843c9e6284b1a95c964a81efbb4f2a67ee89d/build/android/gyp/util/resource_utils.py

See issue 558569 and issue 650527 (at least) for past discussion about webview's locale support - the summary is that we're not even sure in *which* ways webview's locale support is incomplete right now, but one of the things that's definitely missing is translations of java-side strings into the languages that webview supports but clank does not :(
Cc: heamy@chromium.org
Thanks for the clarifications and additional information. In the end, it looks like that the "feature" of embedding extra strings for 9 "omitted" locales for WebView *never* worked correctly, i.e.:

- Even for SystemWebView.apk, some localized string translations are simply
  missing.

- We don't have a clear idea of which exact localized strings are required
  by the WebView feature.

- We embed *by mistake* 80 kiB of strings localized for these 9 omitted
  locales, strings which come from dependencies (the Android compatibility
  library), though we don't have any idea whether these are really needed.

- The ICU data file is missing collation tables for the 9 missing locales
  anyway (i.e. no proper localized searching/comparison) in all builds.

- Nobody seems to have complained loudly about this, at least we haven't heard
  a lot of application developers complain about it.

I've tried to fix the issue locally, but the end result is nasty and requires more cleanups. Given the points above, I think we should just nuke the feature entirely in order to simplify our build system. I.e.:

  - When building the SystemWebView.apk, embed the strings for the 53 locales
    (this is what is being done), ensuring there are translations for all of
    them (that's where we can provide a simple fix).

  - When building Monochrome.apk, only embed strings for the 44 locales.
    Don't mind about the 9 omitted ones, since this probably changes nothing
    and removes 80 kiB from it.

  - When building Monochrome.aab, embed strings for the 55 locales (easy)
    since language-based splits will take care of saving install size. This
    requires adding some missing translations though.

  - When using 55 locales, instead of 44, ensure the ICU data file contains
    collation tables for all locales (currently, the prebuilt binary file we
    use only supports 44). This essentially requires a new prebuilt data file,
    and a few creative changes to the build system to achieve this.

    (and given that the way GN works, I would advocate modifying apkbuilder.py
     to replace the ICU data file when generating a bundle or apk, instead of
     trying to modify the ICU-related GN targets and everything that depends
     on).

What do you think about this? I think it would be much more workable than what we're trying to do here.

Plan sgtm, but wonder about just leaving Monochrome.apk how it is. E.g.:
* First start shipping Monochrome.aab
* Then, add the 9 omitted languages everywhere (once .aab is shipped, Monochrome.apk will just be for local testing purposes, or maybe we'll even delete it...)

Just don't want to have a window where we're regressing Monochrome's language support.
Project Member

Comment 6 by bugdroid1@chromium.org, Jan 9

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/abea22c4a732eb151c072e2544fc1f5854951027

commit abea22c4a732eb151c072e2544fc1f5854951027
Author: David 'Digit' Turner <digit@google.com>
Date: Wed Jan 09 10:48:08 2019

android: Add dump_apk_resource_strings.py script.

This CL introduces a new simply Python script to parse the
binary Android resources table file inside APKs (resources.arsc)
in order to properly dump the values of all localized strings.

This is needed because 'aapt dump' simply cannot provide the
information we need in a usable way :-(

Note that this script doesn't handle bundles at all. However,
'bundletool.py dump resources --bundle <bundle> --values' can
be used to dump the resources in a usable format.

The output from a typical APK will look like:

  Resource strings count=<number> {
    res_id=0x<number> name=<resource_name> {
      config (default) "Hello world"
      config fr        "Salut le monde"
      config es        "Hola el mundo"
      ...
    }
    res_id=0x<number> name=<resource_name> {
      config (default) "....."
      config ...
    }
    ...
  }

The motivation for this CL is to better understand which strings
exactly end up in the final Chrome APKs. We have had several large
surprises in the past due to the simple fact that compiling Chromium
resources, and localized strings in particular, is a very complex
process.

BUG=897056
R=agrieve@chromium.org,estevenson@chromium.org,torne@chromium.org,pasko@chromium.org,yfriedman@chromium.org

Change-Id: Id3dd464b6571949277bf908a73c7776cfa53619c
Reviewed-on: https://chromium-review.googlesource.com/c/1394306
Commit-Queue: David Turner <digit@chromium.org>
Reviewed-by: agrieve <agrieve@chromium.org>
Reviewed-by: Egor Pasko <pasko@chromium.org>
Cr-Commit-Position: refs/heads/master@{#621108}
[add] https://crrev.com/abea22c4a732eb151c072e2544fc1f5854951027/build/android/dump_apk_resource_strings.py

Project Member

Comment 7 by bugdroid1@chromium.org, Jan 18 (4 days ago)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eaa4e6cf4c68f7bd5833a3120d32e55286014874

commit eaa4e6cf4c68f7bd5833a3120d32e55286014874
Author: David 'Digit' Turner <digit@google.com>
Date: Fri Jan 18 14:25:34 2019

android: Keep Webview resources into base bundle module.

This CL modifies the way the monochrome public_bundle is
built in order to ensure that all Android resources used
by the WebView feature are placed into the base bundle module,
and thus will be available to all installs/users of
Monochrome, even when language-based splits are enabled.

This is required to avoid the problem described by
https://crbug.com/902878 where WebView UI strings
are only partially translated.

To do so, the use of shared_resources_whitelist is used to
build a whitelist of Monochrome resource IDs, taken from the
resource with the same names as the WebView APK ones, to be
passed to bundle's JSON configuration (.BundleConfig) file.

In more details:

- Store a new .build_config entry for each base
  android_app_bundle_module() that uses
  |shared_resources_whitelist|, which stores the
  path to the corresponding input R.txt that will
  act as a whitelist.

  This doesn't require any changes in the bundle
  module definitions (either upstream or downstream).

- Add resource_utils.GenerateStringResourcesWhitelist()
  function and unit-test, which generates a dictionary
  mapping the whitelisted base resources IDs to their
  name.

- Add two new options to create_app_bundle.py:
  --base-module-rtxt-path and --base-whitelist-rtxt-path
  which allow passing the paths of two input R.txt
  files that are used to build the resource ID whitelist.

+ Update .build_config format documentation.

BUG=897056,902878
R=estevenson@chromium.org,torne@chromium.org,benmason@chromium.org,tiborg@chromium.org
TBR=agrieve@chromium.org

Change-Id: I79469730b24f2a0f72fddde4476586ce19ceb919
Reviewed-on: https://chromium-review.googlesource.com/c/1348053
Commit-Queue: David Turner <digit@chromium.org>
Reviewed-by: David Trainor <dtrainor@chromium.org>
Reviewed-by: Richard Coles <torne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#624110}
[modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/create_app_bundle.py
[modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/util/resource_utils.py
[modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/util/resource_utils_test.py
[modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/write_build_config.py
[modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/config/android/internal_rules.gni
[modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/config/android/rules.gni

Sign in to add a comment