Fix Android locale-related resources compilation process for Monochrome and SystemWebView |
||
Issue descriptionI had a very hard time implementing the proper CLs to add extra language strings to Android app bundles (but not regular APKs), during my work for http://crbug.com/882860, only to discover that the way we package locale-related UI string resources does *not* work correctly. More specifically: 1) Chrome doesn't ship all locales on Android in order to save on disk space for regular APKs (see http://crbug.com/369218). This is controlled by the android_chrome_omitted_locales list defined in build/config/locales.gni [1], which currently lists 9 locales. 2) However, UI strings for these omitted locales *must* be included in the system webview feature, which is provided by Chrome, as two separate APKs: SystemWebView.apk (for Android L+M) Monochrome.apk (for Android N+) In practice, this means that: SystemWebView.apk should contain resource strings for all supported locales. Monochrome.apk should not contain resource strings for the omitted locales, unless they are used by the system webview feature. Several places in our build system are impacted by this to implement this. To see them, use code search to look for 'locales', 'aapt_locale_whitelist' and 'android_chrome_omitted_locales' in our GN files) [2] I added a debugging mode to the compile_resources.py script recently [3] and discovered that the APKs we currently generate do *not* have the right Android resources in their resources.arsc file, i.e.: - The UI strings for the 9 omitted locales are *never* included in either SystemWebView.apk or Monochrome.apk - But _other_ UI strings for the same 9 omitted locales are included, and come from the following prebuilt packages that are linked into Chromium: - com_android_support_appcompat_v7_java - com_android_support_mediarouter_v7_java - com_android_support_support_compat_java Our compile_resources.py script is supposed to remove these from the final compiled resources (using the AAPT -c flag), but for some reason, this seems to fail horribly. These "dead strings" are never used and just add 80 kiB to the .apk size needlessly. You can check this using "aapt dump resources out/Release/apks/SystemWebView.apk" for example, and look for "config bn", which currently lists: config bn: resource 0x020c0000 com.android.webview:string/abc_action_bar_home_description: t=0x03 d=0x000007f8 (s=0x0008 r=0x00) resource 0x020c0001 com.android.webview:string/abc_action_bar_up_description: t=0x03 d=0x000007eb (s=0x0008 r=0x00) resource 0x020c0002 com.android.webview:string/abc_action_menu_overflow_description: t=0x03 d=0x000007ea (s=0x0008 r=0x00) resource 0x020c0003 com.android.webview:string/abc_action_mode_done: t=0x03 d=0x000007f7 (s=0x0008 r=0x00) resource 0x020c0004 com.android.webview:string/abc_activity_chooser_view_see_all: t=0x03 d=0x000007f6 (s=0x0008 r=0x00) resource 0x020c0005 com.android.webview:string/abc_activitychooserview_choose_application: t=0x03 d=0x000007ec (s=0x0008 r=0x00) resource 0x020c0006 com.android.webview:string/abc_capital_off: t=0x03 d=0x000007f3 (s=0x0008 r=0x00) resource 0x020c0007 com.android.webview:string/abc_capital_on: t=0x03 d=0x000007f2 (s=0x0008 r=0x00) resource 0x020c0014 com.android.webview:string/abc_search_hint: t=0x03 d=0x000007e9 (s=0x0008 r=0x00) resource 0x020c0015 com.android.webview:string/abc_searchview_description_clear: t=0x03 d=0x000007f0 (s=0x0008 r=0x00) resource 0x020c0016 com.android.webview:string/abc_searchview_description_query: t=0x03 d=0x000007ee (s=0x0008 r=0x00) resource 0x020c0017 com.android.webview:string/abc_searchview_description_search: t=0x03 d=0x000007f1 (s=0x0008 r=0x00) resource 0x020c0018 com.android.webview:string/abc_searchview_description_submit: t=0x03 d=0x000007ef (s=0x0008 r=0x00) resource 0x020c0019 com.android.webview:string/abc_searchview_description_voice: t=0x03 d=0x000007f4 (s=0x0008 r=0x00) resource 0x020c001a com.android.webview:string/abc_shareactionprovider_share_with: t=0x03 d=0x000007ed (s=0x0008 r=0x00) resource 0x020c001b com.android.webview:string/abc_shareactionprovider_share_with_application: t=0x03 d=0x000007e8 (s=0x0008 r=0x00) resource 0x020c001c com.android.webview:string/abc_toolbar_collapse_description: t=0x03 d=0x000007f5 (s=0x0008 r=0x00) resource 0x020c005e com.android.webview:string/search_menu_title: t=0x03 d=0x000007f1 (s=0x0008 r=0x00) resource 0x020c005f com.android.webview:string/status_bar_notification_info_overflow: t=0x03 d=0x000007f9 (s=0x0008 r=0x00) All of these entries are from the compatibility libraries, and none of the android_webview related ones are present :-( Note that the issue only affects Android UI string resources. Localized strings accessed from C++ are stored in .pak files as assets in the APK, and are subject to their own, but similar rules, as well, but are handled properly. This entry is to better diagnose and fix the issue. We also probably need better regression tests to verify that the content of said APKs is what it should be :-/ [1] https://cs.chromium.org/chromium/src/build/config/locales.gni?rcl=16509560384017dea91f2833c1ffacd3116e1a9c&l=8 [2] https://cs.chromium.org/search/?q=(%22locales%22+OR+%22android_chrome_omitted_locales%22+OR+aapt_locale_whitelist)+AND+file:gn&sq=package:chromium&type=cs [3] https://chromium-review.googlesource.com/c/chromium/src/+/1288436
,
Oct 19
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/70b843c9e6284b1a95c964a81efbb4f2a67ee89d commit 70b843c9e6284b1a95c964a81efbb4f2a67ee89d Author: David 'Digit' Turner <digit@google.com> Date: Fri Oct 19 16:31:39 2018 build: Add android_debug_resources_temp_dir Normally, compiling Android resources requires unpacking and processing resource dependencies into a temporary directory before sending the result to 'aapt2 compile', then getting rid of these intermediate files. To help debug issues related to Android resource compilation, this CL introduces a new GN variable that can be set in your args.gn, to force the build to put all such build directories under a top-level directory, and not removing intermediate files in it. This should allow inspecting the intermediate files to better understand what's going on. This is needed to debug issues that appeared in my attempt to change the resources stored in APKs versus bundles, e.g.: https://chromium-review.googlesource.com/c/chromium/src/+/1270947 BUG=879228,882860,897056 R=agrieve@chromium.org, estevenson@chromium.org,yfriedman@chromium.org Change-Id: I76c79a893b0bbdb56ba685f4304da929d11c7e05 Reviewed-on: https://chromium-review.googlesource.com/c/1288436 Reviewed-by: agrieve <agrieve@chromium.org> Commit-Queue: David Turner <digit@chromium.org> Cr-Commit-Position: refs/heads/master@{#601178} [modify] https://crrev.com/70b843c9e6284b1a95c964a81efbb4f2a67ee89d/build/android/gyp/compile_resources.py [modify] https://crrev.com/70b843c9e6284b1a95c964a81efbb4f2a67ee89d/build/android/gyp/util/resource_utils.py
,
Oct 19
See issue 558569 and issue 650527 (at least) for past discussion about webview's locale support - the summary is that we're not even sure in *which* ways webview's locale support is incomplete right now, but one of the things that's definitely missing is translations of java-side strings into the languages that webview supports but clank does not :(
,
Oct 23
Thanks for the clarifications and additional information. In the end, it looks like that the "feature" of embedding extra strings for 9 "omitted" locales for WebView *never* worked correctly, i.e.:
- Even for SystemWebView.apk, some localized string translations are simply
missing.
- We don't have a clear idea of which exact localized strings are required
by the WebView feature.
- We embed *by mistake* 80 kiB of strings localized for these 9 omitted
locales, strings which come from dependencies (the Android compatibility
library), though we don't have any idea whether these are really needed.
- The ICU data file is missing collation tables for the 9 missing locales
anyway (i.e. no proper localized searching/comparison) in all builds.
- Nobody seems to have complained loudly about this, at least we haven't heard
a lot of application developers complain about it.
I've tried to fix the issue locally, but the end result is nasty and requires more cleanups. Given the points above, I think we should just nuke the feature entirely in order to simplify our build system. I.e.:
- When building the SystemWebView.apk, embed the strings for the 53 locales
(this is what is being done), ensuring there are translations for all of
them (that's where we can provide a simple fix).
- When building Monochrome.apk, only embed strings for the 44 locales.
Don't mind about the 9 omitted ones, since this probably changes nothing
and removes 80 kiB from it.
- When building Monochrome.aab, embed strings for the 55 locales (easy)
since language-based splits will take care of saving install size. This
requires adding some missing translations though.
- When using 55 locales, instead of 44, ensure the ICU data file contains
collation tables for all locales (currently, the prebuilt binary file we
use only supports 44). This essentially requires a new prebuilt data file,
and a few creative changes to the build system to achieve this.
(and given that the way GN works, I would advocate modifying apkbuilder.py
to replace the ICU data file when generating a bundle or apk, instead of
trying to modify the ICU-related GN targets and everything that depends
on).
What do you think about this? I think it would be much more workable than what we're trying to do here.
,
Oct 23
Plan sgtm, but wonder about just leaving Monochrome.apk how it is. E.g.: * First start shipping Monochrome.aab * Then, add the 9 omitted languages everywhere (once .aab is shipped, Monochrome.apk will just be for local testing purposes, or maybe we'll even delete it...) Just don't want to have a window where we're regressing Monochrome's language support.
,
Jan 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/abea22c4a732eb151c072e2544fc1f5854951027 commit abea22c4a732eb151c072e2544fc1f5854951027 Author: David 'Digit' Turner <digit@google.com> Date: Wed Jan 09 10:48:08 2019 android: Add dump_apk_resource_strings.py script. This CL introduces a new simply Python script to parse the binary Android resources table file inside APKs (resources.arsc) in order to properly dump the values of all localized strings. This is needed because 'aapt dump' simply cannot provide the information we need in a usable way :-( Note that this script doesn't handle bundles at all. However, 'bundletool.py dump resources --bundle <bundle> --values' can be used to dump the resources in a usable format. The output from a typical APK will look like: Resource strings count=<number> { res_id=0x<number> name=<resource_name> { config (default) "Hello world" config fr "Salut le monde" config es "Hola el mundo" ... } res_id=0x<number> name=<resource_name> { config (default) "....." config ... } ... } The motivation for this CL is to better understand which strings exactly end up in the final Chrome APKs. We have had several large surprises in the past due to the simple fact that compiling Chromium resources, and localized strings in particular, is a very complex process. BUG=897056 R=agrieve@chromium.org,estevenson@chromium.org,torne@chromium.org,pasko@chromium.org,yfriedman@chromium.org Change-Id: Id3dd464b6571949277bf908a73c7776cfa53619c Reviewed-on: https://chromium-review.googlesource.com/c/1394306 Commit-Queue: David Turner <digit@chromium.org> Reviewed-by: agrieve <agrieve@chromium.org> Reviewed-by: Egor Pasko <pasko@chromium.org> Cr-Commit-Position: refs/heads/master@{#621108} [add] https://crrev.com/abea22c4a732eb151c072e2544fc1f5854951027/build/android/dump_apk_resource_strings.py
,
Jan 18
(4 days ago)
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/eaa4e6cf4c68f7bd5833a3120d32e55286014874 commit eaa4e6cf4c68f7bd5833a3120d32e55286014874 Author: David 'Digit' Turner <digit@google.com> Date: Fri Jan 18 14:25:34 2019 android: Keep Webview resources into base bundle module. This CL modifies the way the monochrome public_bundle is built in order to ensure that all Android resources used by the WebView feature are placed into the base bundle module, and thus will be available to all installs/users of Monochrome, even when language-based splits are enabled. This is required to avoid the problem described by https://crbug.com/902878 where WebView UI strings are only partially translated. To do so, the use of shared_resources_whitelist is used to build a whitelist of Monochrome resource IDs, taken from the resource with the same names as the WebView APK ones, to be passed to bundle's JSON configuration (.BundleConfig) file. In more details: - Store a new .build_config entry for each base android_app_bundle_module() that uses |shared_resources_whitelist|, which stores the path to the corresponding input R.txt that will act as a whitelist. This doesn't require any changes in the bundle module definitions (either upstream or downstream). - Add resource_utils.GenerateStringResourcesWhitelist() function and unit-test, which generates a dictionary mapping the whitelisted base resources IDs to their name. - Add two new options to create_app_bundle.py: --base-module-rtxt-path and --base-whitelist-rtxt-path which allow passing the paths of two input R.txt files that are used to build the resource ID whitelist. + Update .build_config format documentation. BUG=897056,902878 R=estevenson@chromium.org,torne@chromium.org,benmason@chromium.org,tiborg@chromium.org TBR=agrieve@chromium.org Change-Id: I79469730b24f2a0f72fddde4476586ce19ceb919 Reviewed-on: https://chromium-review.googlesource.com/c/1348053 Commit-Queue: David Turner <digit@chromium.org> Reviewed-by: David Trainor <dtrainor@chromium.org> Reviewed-by: Richard Coles <torne@chromium.org> Cr-Commit-Position: refs/heads/master@{#624110} [modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/create_app_bundle.py [modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/util/resource_utils.py [modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/util/resource_utils_test.py [modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/android/gyp/write_build_config.py [modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/config/android/internal_rules.gni [modify] https://crrev.com/eaa4e6cf4c68f7bd5833a3120d32e55286014874/build/config/android/rules.gni |
||
►
Sign in to add a comment |
||
Comment 1 by torne@google.com
, Oct 19