Merge ARC symbols upload into Android PFQ. |
||||||||||||||||
Issue descriptionRight now, the ARC symbols upload is somewhat error prone, needs manual updates for every new branch, and isnt' really monitored for failures. I think the most maintainable solution is to get rid of the custom builders, and instead integrate the symbol upload into the Android PFQ (this will also reduce the number of symbols we upload over all). The current system can be found here: repo init -u https://chrome-internal.googlesource.com/android/cbuildbot_symbols_upload It seems reasonable to more the necessary parts into Chromite and add them as new stages for the Android PFQ. We should also be able to get the symbols from the same bucket we pull Android candidate images from, and so won't need the build API support any more.
,
Jul 29 2016
,
Aug 25 2016
This could use an owner. If it's fixed, we can shutdown all of the symbol builders on this waterfall, and I can stop getting special requests to update them every time we branch. Also, looking at the state of tests on the non-symbol builders, we might be able to just shut that waterfall down.
,
Nov 3 2016
Who has owned ARC symbols in the past? I have no context here.
,
Nov 3 2016
I built a special site specific thing a while back without understanding the full context. Maintaining it requires code changes every few weeks, which I'm not good about doing. If we can instead upload symbols from the Android PFQ (which will take some work), then this becomes self maintaining, and errors will be visible, not hidden on a waterfall nobody knows about.
,
Dec 6 2016
Uploading from Android PFQ would be nice for dev/canary builds, but the PFQ doesn't manage release branches, where I think we still need a solution. We do use the cros_mark_android_as_stable script AFAIK so if we put the relevant bits to do this there then this solves all our issues going forward.
,
Dec 6 2016
We manually update on the release branches? I didn't realize that.
,
Dec 7 2016
yeah, but it's part of chromite (and IIUC chrome on release branches is done manually like this as well), so I think if we make the Android PFQ capable of this, the manual pushing would also work.
,
Jan 31 2017
Issue 644948 has been merged into this issue.
,
Feb 2 2017
After talking with dgarrett@ I took a look at this issue. I thought the ideal way would be to install debug symbols from Android binaries to /usr/lib/debug. Then existing DebugSymbols stage will pick up them and upload automatically. Also it might be useful on debugging because symbols will be available on CrOS chroot. However Android dynamic libraries might have packed relocations[1] and dump_syms (and other tools like gdb) may not be able to process them without unpacking them by relocation_packer first. Also, relocation_packer may emit an address offset value which should be adjusted to the output of dump_syms. Therefore it seems a bit difficult to install Android symbols to /usr/lib/debug. However I think DebugSymbols stage is a good place to upload Android debug symbols because symbols from all other binaries (chrome and dynamic libraries) are uploaded there. Adding logic there sounds consistent to me. What do you think? [1]: https://android.googlesource.com/platform/bionic/+/f5e0ba94d911ef2622ecfd3f7fabc4432a4806d3/tools/relocation_packer/README.TXT
,
Feb 2 2017
In one sense, DebugSymbols is a good choice because it's where we currently handle all symbols. On the other hand, there is no need to upload android symbols on every build, since they only change when the container is updated. Hum.... maybe that's unnecessary optimization. Having that step be specific to the Android PFQ would stop them from being uploaded as a build artifact, and DebugSymbols already contains a de-duplication process that should avoid adding excessive additional load to the crash servers. Alright, extra logic in DebugSymbols sounds right to me.
,
Feb 3 2017
Thanks, dgarrett. Then I'll proceed with adding logic to DebugSymbols. FYI, I sent cl/146434886 for review to export symbols.zip.
,
Feb 3 2017
My current plan is as following. If you have any comments, recommendations etc., please let me know! 1. Save symbols.zip in GCS (cl/146434886) 2. In android-container ebuild, extract symbols.zip and install them to /opt/google/containers/android/debug. We set up masks so that they are kept only in chroot and not installed to the devices. Note that I'm proposing to install them in a non-standard location, instead of /usr/lib/debug, because those files might be relocation-packed and can not be processed by standard toolchains. 3. Port gen_breakpad_symbols.py in cbuildbot_symbols_upload to chromite as cros_generate_android_breakpad_symbols.py. It will scan debug symbols under /opt/google/containers/android/debug, convert them to breakpad format and save the results to /usr/lib/debug/breakpad. 4. In DebugSymbols stage of buildbot, run cros_generate_android_breakpad_symbols.
,
Feb 6 2017
Can we keep it as a separate ebuild or only download and install the symbol files when we need to? You're talking about adding another 1-2GB worth of data to sync which isn't great if you're working remotely or in the factory, especially if it's not actually useful or needed. Not to mention how ever long it takes to process during peoples builds. If you're going to add it to every build, can you preprocess during the PFQ run, and save the already processed symbols to use for everyone else's builds.
,
Feb 7 2017
davidriley: Thanks for comments! It makes sense that installing GBs in every builder sounds slow. Then I'll add a new stage (like AndroidDownloadSymbols) to Android PFQ which will run after successful UprevAndroid stage to download symbols.zip from GS. Then Non-PFQ builders do not bother to download and unpack symbols.zip.
,
Feb 7 2017
Well, in addition to the builder, I'm worried about doing it on every developer build. If you add it to an ebuild, then won't everyone need that full set of symbols as part of their build?
,
Feb 8 2017
You are right, I meant every build (not every builder), sorry. By adding a PFQ-only stage we should be able to avoid adding a new ebuild. I'll soon sending CLs to reviews...
,
Feb 8 2017
I uploaded following changes: - manifest.xml updates to check out AOSP prebuilts (for relocation_packer) https://chrome-internal-review.googlesource.com/c/325665/ https://chromium-review.googlesource.com/c/438914/ - Command to generate Android debug symbols https://chromium-review.googlesource.com/c/439035/ - Command to download Android debug symbols archive https://chromium-review.googlesource.com/c/439103/ - cbuildbot changes to process Android debug symbols https://chromium-review.googlesource.com/c/438952/
,
Feb 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/6fca899ff48b67e9594c96f279a94d7358c28bd8 commit 6fca899ff48b67e9594c96f279a94d7358c28bd8 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Feb 13 09:17:53 2017 target-chromium-os-sdk: Add relocation_packer from Android. relocation_packer is a utility to process packed relocations in Android. This command will be used in buildbots to unpack relocations in Android symbol files. Since the archive generated by Gitiles is not deterministic, I mirrored those archives in GS: $ gsutil cp -a public-read \ /var/cache/chromeos-cache/distfiles/host/android-*.tar.gz \ gs://chromeos-localmirror/distfiles/ BUG= chromium:628777 TEST=./update_chroot && relocation_packer --help Change-Id: I2ae91e4b5067caac53ac721fde34fb3663b1fc62 Reviewed-on: https://chromium-review.googlesource.com/440884 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/6fca899ff48b67e9594c96f279a94d7358c28bd8/chromeos-base/android-relocation-packer/Manifest [modify] https://crrev.com/6fca899ff48b67e9594c96f279a94d7358c28bd8/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1.ebuild [add] https://crrev.com/6fca899ff48b67e9594c96f279a94d7358c28bd8/chromeos-base/android-relocation-packer/android-relocation-packer-20170210-r1.ebuild [add] https://crrev.com/6fca899ff48b67e9594c96f279a94d7358c28bd8/chromeos-base/android-relocation-packer/android-relocation-packer-20170210.ebuild [rename] https://crrev.com/6fca899ff48b67e9594c96f279a94d7358c28bd8/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1-r71.ebuild
,
Feb 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/721d5a77476d7c6669066547c908803940a68304 commit 721d5a77476d7c6669066547c908803940a68304 Author: Shuhei Takahashi <nya@chromium.org> Date: Wed Feb 15 18:47:33 2017 Command to generate breakpad symbols for Android. cros_generate_android_breakpad_symbols unpacks an Android symbols archive and processes its contents to generate debug symbols in breakpad format. The code is mostly just copied from: https://chrome-internal.googlesource.com/android/cbuildbot_symbols_upload CQ-DEPEND=CL:440884 BUG= chromium:628777 TEST=cros_generate_android_breakpad_symbols_unittest TEST=cros_generate_android_breakpad_symbols --symbols_file=symbols.zip \ --breakpad_dir=breakpad Change-Id: Ic871a41262fdc5049cdb9a7fd17c1a60a0958fac Reviewed-on: https://chromium-review.googlesource.com/439035 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols_unittest [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/bin/cros_generate_android_breakpad_symbols [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols.py [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols_unittest.py
,
Feb 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/721d5a77476d7c6669066547c908803940a68304 commit 721d5a77476d7c6669066547c908803940a68304 Author: Shuhei Takahashi <nya@chromium.org> Date: Wed Feb 15 18:47:33 2017 Command to generate breakpad symbols for Android. cros_generate_android_breakpad_symbols unpacks an Android symbols archive and processes its contents to generate debug symbols in breakpad format. The code is mostly just copied from: https://chrome-internal.googlesource.com/android/cbuildbot_symbols_upload CQ-DEPEND=CL:440884 BUG= chromium:628777 TEST=cros_generate_android_breakpad_symbols_unittest TEST=cros_generate_android_breakpad_symbols --symbols_file=symbols.zip \ --breakpad_dir=breakpad Change-Id: Ic871a41262fdc5049cdb9a7fd17c1a60a0958fac Reviewed-on: https://chromium-review.googlesource.com/439035 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols_unittest [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/bin/cros_generate_android_breakpad_symbols [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols.py [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols_unittest.py
,
Feb 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/721d5a77476d7c6669066547c908803940a68304 commit 721d5a77476d7c6669066547c908803940a68304 Author: Shuhei Takahashi <nya@chromium.org> Date: Wed Feb 15 18:47:33 2017 Command to generate breakpad symbols for Android. cros_generate_android_breakpad_symbols unpacks an Android symbols archive and processes its contents to generate debug symbols in breakpad format. The code is mostly just copied from: https://chrome-internal.googlesource.com/android/cbuildbot_symbols_upload CQ-DEPEND=CL:440884 BUG= chromium:628777 TEST=cros_generate_android_breakpad_symbols_unittest TEST=cros_generate_android_breakpad_symbols --symbols_file=symbols.zip \ --breakpad_dir=breakpad Change-Id: Ic871a41262fdc5049cdb9a7fd17c1a60a0958fac Reviewed-on: https://chromium-review.googlesource.com/439035 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols_unittest [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/bin/cros_generate_android_breakpad_symbols [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols.py [add] https://crrev.com/721d5a77476d7c6669066547c908803940a68304/scripts/cros_generate_android_breakpad_symbols_unittest.py
,
Feb 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/2507e78f69c0ea9a908be83b86d6996890daa33e commit 2507e78f69c0ea9a908be83b86d6996890daa33e Author: Shuhei Takahashi <nya@chromium.org> Date: Sat Feb 25 18:32:08 2017 Upload Android debug symbols in Android PFQ. A new stage DownloadAndroidDebugSymbols is added just before DebugSymbols stage. It will download Android symbols archive from Google Cloud Storage only when Android uprev is happening (thus in Android PFQ). Downloaded archive is processed by updated DebugSymbols stage to upload to the crash server. CQ-DEPEND=CL:439035 BUG= chromium:628777 TEST=commands_unittest TEST=artifact_stages_unittest TEST=cbuildbot_unittest TEST=cbuildbot --remote veyron_minnie-android-pfq TEST=cbuildbot --remote master-android-pfq Change-Id: I4ff521cd98725ce555d2f4fe0b2592c15d093692 Reviewed-on: https://chromium-review.googlesource.com/438952 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: David Riley <davidriley@chromium.org> [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/cbuildbot/stages/artifact_stages_unittest.py [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/cbuildbot/stages/android_stages.py [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/cbuildbot/builders/simple_builders.py [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/cbuildbot/commands.py [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/lib/constants.py [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/cbuildbot/stages/artifact_stages.py [modify] https://crrev.com/2507e78f69c0ea9a908be83b86d6996890daa33e/cbuildbot/commands_unittest.py
,
Mar 6 2017
The merged phase is running fine recently. Closing. I'll file another bug to drop legacy buildbots.
,
Apr 3 2017
Requesting cherry-pick of this as a dependency of Issue 701307.
,
Apr 4 2017
,
Apr 6 2017
ketakid: I need to cherry-pick these changes as dependencies of Issue 701307. Could you take a look?
,
Apr 7 2017
Approving merge to M57 Chrome OS.
,
Apr 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a72414c518651a359c6b6177d00d9307bab6761b commit a72414c518651a359c6b6177d00d9307bab6761b Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Apr 10 03:14:18 2017 target-chromium-os-sdk: Add relocation_packer from Android. relocation_packer is a utility to process packed relocations in Android. This command will be used in buildbots to unpack relocations in Android symbol files. Since the archive generated by Gitiles is not deterministic, I mirrored those archives in GS: $ gsutil cp -a public-read \ /var/cache/chromeos-cache/distfiles/host/android-*.tar.gz \ gs://chromeos-localmirror/distfiles/ BUG= chromium:628777 TEST=./update_chroot && relocation_packer --help Change-Id: I2ae91e4b5067caac53ac721fde34fb3663b1fc62 Reviewed-on: https://chromium-review.googlesource.com/440884 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> (cherry picked from commit 6fca899ff48b67e9594c96f279a94d7358c28bd8) Reviewed-on: https://chromium-review.googlesource.com/465647 Reviewed-by: Shuhei Takahashi <nya@chromium.org> Commit-Queue: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/a72414c518651a359c6b6177d00d9307bab6761b/chromeos-base/android-relocation-packer/Manifest [modify] https://crrev.com/a72414c518651a359c6b6177d00d9307bab6761b/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1.ebuild [add] https://crrev.com/a72414c518651a359c6b6177d00d9307bab6761b/chromeos-base/android-relocation-packer/android-relocation-packer-20170210-r1.ebuild [add] https://crrev.com/a72414c518651a359c6b6177d00d9307bab6761b/chromeos-base/android-relocation-packer/android-relocation-packer-20170210.ebuild [rename] https://crrev.com/a72414c518651a359c6b6177d00d9307bab6761b/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1-r71.ebuild
,
Apr 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/f944c4949eadb726b3e0f0fd97f480ae11ba9249 commit f944c4949eadb726b3e0f0fd97f480ae11ba9249 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Apr 10 03:14:22 2017 Command to generate breakpad symbols for Android. cros_generate_android_breakpad_symbols unpacks an Android symbols archive and processes its contents to generate debug symbols in breakpad format. The code is mostly just copied from: https://chrome-internal.googlesource.com/android/cbuildbot_symbols_upload CQ-DEPEND=CL:465647 BUG= chromium:628777 TEST=cros_generate_android_breakpad_symbols_unittest TEST=cros_generate_android_breakpad_symbols --symbols_file=symbols.zip \ --breakpad_dir=breakpad Reviewed-on: https://chromium-review.googlesource.com/439035 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> (cherry picked from commit 721d5a77476d7c6669066547c908803940a68304) Change-Id: Ic871a41262fdc5049cdb9a7fd17c1a60a0958fac Reviewed-on: https://chromium-review.googlesource.com/465767 Commit-Queue: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/f944c4949eadb726b3e0f0fd97f480ae11ba9249/scripts/cros_generate_android_breakpad_symbols_unittest [add] https://crrev.com/f944c4949eadb726b3e0f0fd97f480ae11ba9249/bin/cros_generate_android_breakpad_symbols [add] https://crrev.com/f944c4949eadb726b3e0f0fd97f480ae11ba9249/scripts/cros_generate_android_breakpad_symbols.py [add] https://crrev.com/f944c4949eadb726b3e0f0fd97f480ae11ba9249/scripts/cros_generate_android_breakpad_symbols_unittest.py
,
Apr 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/3348fb92bbc94a72a3fda8a5c124fea70ddb6002 commit 3348fb92bbc94a72a3fda8a5c124fea70ddb6002 Author: Shuhei Takahashi <nya@chromium.org> Date: Mon Apr 10 03:14:24 2017 Upload Android debug symbols in Android PFQ. A new stage DownloadAndroidDebugSymbols is added just before DebugSymbols stage. It will download Android symbols archive from Google Cloud Storage only when Android uprev is happening (thus in Android PFQ). Downloaded archive is processed by updated DebugSymbols stage to upload to the crash server. CQ-DEPEND=CL:465767 BUG= chromium:628777 TEST=commands_unittest TEST=artifact_stages_unittest TEST=cbuildbot_unittest TEST=cbuildbot --remote veyron_minnie-android-pfq TEST=cbuildbot --remote master-android-pfq Reviewed-on: https://chromium-review.googlesource.com/438952 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: David Riley <davidriley@chromium.org> (cherry picked from commit 2507e78f69c0ea9a908be83b86d6996890daa33e) Change-Id: I4ff521cd98725ce555d2f4fe0b2592c15d093692 Reviewed-on: https://chromium-review.googlesource.com/465768 Commit-Queue: Shuhei Takahashi <nya@chromium.org> [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/cbuildbot/stages/artifact_stages_unittest.py [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/cbuildbot/stages/android_stages.py [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/cbuildbot/builders/simple_builders.py [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/cbuildbot/commands.py [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/lib/constants.py [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/cbuildbot/stages/artifact_stages.py [modify] https://crrev.com/3348fb92bbc94a72a3fda8a5c124fea70ddb6002/cbuildbot/commands_unittest.py
,
Apr 10 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 14 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||
Comment 1 by aut...@google.com
, Jul 19 2016