"these are now located outside of the main SDK Tools repo" --> do we know if that's a permanent change, or if they simply haven't released an updated version of tools or build-tools w/ these included?

Yes, from a conversation with the bundletool author, this is a permanent change.
Also, bundletool-0.4.0 was released recently.

FWIW, I ran into https://github.com/google/bundletool/issues/11 a lot while using bundletool-0.4.1. They have since fixed that. So, maybe wait for the next release before adding it to Chrome.

Trying to run tools/android/roll/android_deps/fetch_all.sh today without any modification to check that the script works unfortunately fails with the following error:

ERROR at //build/config/android/rules.gni:3198:7: Script returned non-zero exit code.
      exec_script("//build/android/gyp/aar.py",
      ^----------
Current dir: /tmp/dummy_gn_run/
Command: python -- /opt2/work/chromium/src/build/android/gyp/aar.py list ../../opt2/work/chromium/src/third_party/android_deps/repository/android_arch_lifecycle_runtime/runtime-1.0.0.aar --output ../../opt2/work/chromium/src/third_party/android_deps/repository/android_arch_lifecycle_runtime/android_arch_lifecycle_runtime.info
Returned 2.
stderr:

usage: aar.py list [-h] [--output OUTPUT] aar_file
aar.py list: error: argument --output: can't open '../../opt2/work/chromium/src/third_party/android_deps/repository/android_arch_lifecycle_runtime/android_arch_lifecycle_runtime.info': [Errno 13] Permission denied: '../../opt2/work/chromium/src/third_party/android_deps/repository/android_arch_lifecycle_runtime/android_arch_lifecycle_runtime.info'

See //third_party/android_deps/BUILD.gn:149:1: whence it was called.
android_aar_prebuilt("android_arch_lifecycle_runtime_java") {
^------------------------------------------------------------
See //build/config/android/internal_rules.gni:117:23: which caused the file to be included.
            deps += [ "${_target_label}__build_config" ]
                      ^-------------------------------

Looking into it...

Ok, the reason this fails is because the .info files like android_arch_lifecycle_runtime.info are really symlinks to CIPD-managed read-only files.

For example:  ls -l third_party/android_deps/repository/android_arch_lifecycle_runtime/android_arch_lifecycle_runtime.info

lrwxrwxrwx 1 digit primarygroup 73 Apr  9 09:16 third_party/android_deps/repository/android_arch_lifecycle_runtime/android_arch_lifecycle_runtime.info -> ../../../../../.cipd/pkgs/22/_current/android_arch_lifecycle_runtime.info

And the Gradle build step invokes third_party/android_deps/BUILD.gn which actually uses the android_aar_prebuilt GN template [1]. Looking at the source code of said GN template shows that by default, it will simply check the .info path, not try to overwrite it, unless the GN declare_args variable 'update_android_aar_prebuilts' is set to true.

And said variable is set to true in fetch_all.sh [2]

So it looks like the script is broken due to our use of CIPD. Or the documentation needs to be updated with new steps on how to add new dependencies, but I'm leaning on the first hypothesis for now ....


[1] https://cs.chromium.org/chromium/src/build/config/android/rules.gni?type=cs&q=android_aar_prebuilt&sq=package:chromium&g=0&l=3184

[2] https://cs.chromium.org/chromium/src/tools/android/roll/android_deps/fetch_all.sh?q=update_android_aar_prebuilts&sq=package:chromium&l=26&dr=C

Nuke your .cipd directory -- should be in your gclient root -- and then resync. This is a rough edge that we're working on smoothing over in https://bugs.chromium.org/p/chromium/issues/detail?id=794764

er, sorry -- assumed based on your error messages that the symlink didn't exist, in which case you'd want to do that.

I tried nuking my .cipd directory, but this doesn't work, running fetch_all.sh I get a Gradle error early on now:

Execution failed for task ':setUpRepository'.
> Could not copy file '/usr/local/google/home/digit/.gradle/caches/modules-2/files-2.1/com.android.support/preference-leanback-v17/27.0.0/950fc9c45203a83173c356f8a7f0321c424256a1/preference-leanback-v17-27.0.0.aar' to '/opt2/work/chromium/src/third_party/android_deps/repository/com_android_support_preference_leanback_v17/preference-leanback-v17-27.0.0.aar'.

Fortunately, a simple "gclient sync" will recover all files without altering my checkout..

I guess that I will have to package a CIPD zip file, upload it, then reference it in the DEPS file. I doubt that fetch_all.sh can reliably generate the ANDROID_DEPS section within it in its current state. Would you like a patch for this too?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5c8a944c7ef470343be30ede36e475e0a36019be

commit 5c8a944c7ef470343be30ede36e475e0a36019be
Author: David 'Digit' Turner <digit@google.com>
Date: Mon Jun 18 08:49:04 2018

android: New fetch_all.py script to manage Google Maven dependencies.

The fetch_all.sh script which was used to download Chromium dependencies
from Google Maven no longer works since the move to CIPD-managed
packages.

This CL introduces a new script (fetch_all.py) to fix the situation.
Major differences between these script:

* 'fetch_all.py' doesn't modify the workspace by default.

  Instead it uses a temporary build directory and parses
  the build.gradle file from there. This allows checking that
  all changes to this file are valid, and that all new dependencies
  can be properly downloaded.

  It then compares the build directory to the current workspace
  and list any detected differences (i.e. new/updated/deleted
  packages).

* 'fetch_all.py --update-all' can then be used to modify the
  current workspace as well. This updates DEPS, BUILD.gn, and
  the content of third_party/android_deps/repository/, as well
  as prints to stdout a list of commands to upload new and
  updated packages through cipd.

  After the CIPD uploads are performed, one can create a new CL
  and upload it for review.

* 'fetch_all.py --reset-workspace' can be used to reset the
  workspace to a sane state (including cipd-related links),
  in case the result of --update-all isn't satisfying.

  This simply removes ../.cipd followed by
  'gclient sync --reset --nohooks -r src@HEAD'
  while trying to preserve your local build.gradle!

+ This script calls build/android/gyp/aar.py directly to
  regenerate all AAR .info files. This is simpler than
  relying on GN mode with 'update_android_aar_prebuilts=true',
  which no longer works because it tries to write over
  read-only cipd-controlled .info files.

  Maybe this mode should be removed, and instructions on
  how to add an .info file for AAR prebuilts updated to
  simply call 'aar.py --list <aar_path>' instead?

BUG= 845405 
R=agrieve@chromium.org, jduborick@chromium.org, dgn@chromium.org

Change-Id: Id3b14943a0b7094c1a8c1186bff311eaf283b6e3
Reviewed-on: https://chromium-review.googlesource.com/1102319
Commit-Queue: David Turner <digit@chromium.org>
Reviewed-by: agrieve <agrieve@chromium.org>
Cr-Commit-Position: refs/heads/master@{#567953}
[modify] https://crrev.com/5c8a944c7ef470343be30ede36e475e0a36019be/tools/android/roll/android_deps/README.md
[modify] https://crrev.com/5c8a944c7ef470343be30ede36e475e0a36019be/tools/android/roll/android_deps/buildSrc/src/main/groovy/BuildConfigGenerator.groovy
[modify] https://crrev.com/5c8a944c7ef470343be30ede36e475e0a36019be/tools/android/roll/android_deps/buildSrc/src/main/groovy/ChromiumDepGraph.groovy
[add] https://crrev.com/5c8a944c7ef470343be30ede36e475e0a36019be/tools/android/roll/android_deps/fetch_all.py
[delete] https://crrev.com/743262940430fc58261bbcf28658d500a74721b6/tools/android/roll/android_deps/fetch_all.sh

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6

commit 4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6
Author: David 'Digit' Turner <digit@google.com>
Date: Fri Jun 22 11:31:03 2018

android: Add bundletool and appt2 binaries

Android App Bundle support requires using recent versions
of the bundletool and aapt2 programs, which are not part
of the Android SDK copies used by internal and public
Chrome checkouts at the moment.

Also, these tools are still evolving quickly, and their
latest releases are not part of official Android releases
yet.

To solve this, this CL adds two new directories under
third_party/android_build_tools/, in order to store
their corresponding files.

Note that each tool's binary files are distributed
through DEPS as CIPD packages.

For reference, bundletool is officially distributed from
github (see https://github.com/google/bundletool), and
aapt2 from Google Maven.

Note that in the latter case, our script at
tools/android/roll/android_deps/fetch_all.py cannot
be used, because the corresponding .jar file contains
Linux-specific binaries (executable and libraries), and
our Maven-handling Gradle plugin doesn't know how to
deal with these.

BUG= 845405 
R=agrieve@chromium.org,jbudorick@chromium.org,thakis@chromium.org,benmason@chromium.org,chromium-third-party@google.com

Change-Id: Id650acb9eb55f361fe18f6aa0995776aee832eec
Reviewed-on: https://chromium-review.googlesource.com/1110225
Reviewed-by: Grace Kloba <klobag@chromium.org>
Reviewed-by: Robert Sesek <rsesek@chromium.org>
Commit-Queue: David Turner <digit@chromium.org>
Cr-Commit-Position: refs/heads/master@{#569577}
[modify] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/DEPS
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/build/android/gyp/bundletool.py
[modify] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/.gitignore
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/README.chromium
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/aapt2/NOTICE
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/aapt2/OWNERS
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/aapt2/README.chromium
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/aapt2/cipd.yaml
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/bundletool/LICENSE
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/bundletool/OWNERS
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/bundletool/README.chromium
[add] https://crrev.com/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6/third_party/android_build_tools/bundletool/cipd.yaml

The following revision refers to this bug:
  https://webrtc.googlesource.com/src.git/+/72864963fe626e0971d7725b878a49d3784b6f87

commit 72864963fe626e0971d7725b878a49d3784b6f87
Author: Yves Gerey <yvesg@webrtc.org>
Date: Mon Oct 29 10:28:34 2018

Download aap2 and bundletool as part of required dependencies.

In order to fix the roll https://webrtc-review.googlesource.com/c/src/+/108200,
this CL updates webrtc DEPS to be on a par with chromium's CL:
"Android build: Add bundletool and newest aapt2 binaries to chromium src"
https://chromium.googlesource.com/chromium/src.git/+/4ff35dfa790d9d2155fc8367c0ad30e1a0cc1dc6#

Bug:  chromium:845405 
Change-Id: I768b65b34d2a55bbfdf48acb4766b3e9c04de218
Reviewed-on: https://webrtc-review.googlesource.com/c/108280
Commit-Queue: Yves Gerey <yvesg@webrtc.org>
Reviewed-by: Artem Titov <titovartem@webrtc.org>
Reviewed-by: Patrik Höglund <phoglund@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#25409}
[modify] https://crrev.com/72864963fe626e0971d7725b878a49d3784b6f87/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d4c4f2a6c77fd8f2d5ddd9d202121fa16800fe5b

commit d4c4f2a6c77fd8f2d5ddd9d202121fa16800fe5b
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Mon Oct 29 15:21:11 2018

Roll src/third_party/webrtc f3ff14c00b43..72864963fe62 (4 commits)

https://webrtc.googlesource.com/src.git/+log/f3ff14c00b43..72864963fe62


git log f3ff14c00b43..72864963fe62 --date=short --no-merges --format='%ad %ae %s'
2018-10-29 yvesg@webrtc.org Download aap2 and bundletool as part of required dependencies.
2018-10-29 dkirovbroadsoft@gmail.com Modified PressEnterToContinue() to actualy check if Enter is pressed
2018-10-29 nisse@webrtc.org Replace some usage of EventWrapper with rtc::Event.
2018-10-29 Peter) Slatala Add missing assignment in RTCConfiguration.mm


Created with:
  gclient setdep -r src/third_party/webrtc@72864963fe62

The AutoRoll server is located here: https://autoroll.skia.org/r/webrtc-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:linux_chromium_archive_rel_ng;luci.chromium.try:mac_chromium_archive_rel_ng

BUG= chromium:845405 
TBR=webrtc-chromium-sheriffs-robots@google.com

Change-Id: I97c573dd8da9fe866a4d874f2bc2f56f432a4f2e
Reviewed-on: https://chromium-review.googlesource.com/c/1304378
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#603495}
[modify] https://crrev.com/d4c4f2a6c77fd8f2d5ddd9d202121fa16800fe5b/DEPS