cros tryjob --local chromiumos-sdk-tryjob hangs |
||||
Issue descriptioncros tryjob --local chromiumos-sdk-tryjob: <many, many logs> Total: 0 packages, Size of downloads: 0 KiB Done INFO : Elapsed time (run_chroot_version_hooks): 0m0s INFO : Updating chroot INFO : Clearing shadow utils lockfiles under / INFO : Bootstrapping depot_tools INFO : Updating the SDK Starting fast-emerge. Building package virtual/target-sdk world on root Calculating deps... Deps calculated in 0m13.7s Total: 0 packages, Size of downloads: 0 KiB Done Scanning Configuration files... Exiting: Nothing left to do; exiting. :) INFO : Running 'eclean -d packages' to clean up stale binpkgs >>> Building file list for packages cleaning... >>> Your packages directory was already clean. INFO : Elapsed time (update_chroot): 0m34s Now using icedtea-bin-8 as your generation-2 system JVM INFO cros_sdk:make_chroot: Elapsed time (make_chroot.sh): 2m59s cros_sdk:make_chroot: All set up. To enter the chroot, run: $ cros_sdk --enter --chroot=/usr/local/google/home/dgarrett/sand/tryjob/repository/new-sdk-chroot CAUTION: Do *NOT* rm -rf the chroot directory; if there are stale bind mounts you may end up deleting your source tree too. To unmount and delete the chroot cleanly, use: $ cros_sdk --delete --chroot=/usr/local/google/home/dgarrett/sand/tryjob/repository/new-sdk-chroot This appears to hang forever.
,
Apr 26 2018
Do you have the log entries where it ran cros_sdk to create the second chroot? I know it passes some flags that aren't commonly used.
,
Apr 26 2018
Sorry, I didn't same them off. I should have.
,
Apr 27 2018
I see the same thing on my workstation. Here's the last stage before it hangs: BUILD_STEP: SDKTest ************************************************************ ** Start Stage SDKTest - Thu, 26 Apr 2018 21:43:07 -0600 (MDT) ** ** Stage that performs testing an SDK created in a previous stage ************************************************************ Preconditions for the stage successfully met. Beginning to execute stage... 21:43:07: INFO: RunCommand: /work/bmgordon/tryjob/repository/chromite/bin/cros_sdk --chroot new-sdk-chroot --chrome_root /usr/local/google/home/bmgordon/chromiumos/.cache/distfiles/target-master/chrome-src --download --replace --nousepkg --url file:///work/bmgordon/tryjob/repository/built-sdk.tar.xz 'FEATURES=separatedebug' 'CHROME_ORIGIN=LOCAL_SOURCE' -- in /work/bmgordon/tryjob/repository 21:43:09: NOTICE: Deleting chroot. 21:43:18: NOTICE: Mounted /work/bmgordon/tryjob/repository/new-sdk-chroot.img on chroot 21:43:18: NOTICE: Downloading SDK tarball... 21:43:18: NOTICE: Creating chroot. This may take a few minutes... INFO cros_sdk:make_chroot: Unpacking stage3... INFO cros_sdk:make_chroot: Set timezone... INFO cros_sdk:make_chroot: Adding user/group... INFO cros_sdk:make_chroot: Setting up mounts... INFO cros_sdk:make_chroot: Running init_setup()... INFO cros_sdk:make_chroot: Setting up hosts/resolv... INFO cros_sdk:make_chroot: Setting up /etc/make.*... INFO cros_sdk:make_chroot: Running post-inst configuration hacks >>> Regenerating /etc/ld.so.cache... lrwxrwxrwx 1 root root 88 Apr 26 21:45 /etc/make.conf -> /mnt/host/source/src/third_party/chromiumos-overlay/chromeos/config/make.conf.amd64-host lrwxrwxrwx 1 root root 67 Apr 26 21:45 /etc/portage/make.profile -> /usr/local/portage/chromiumos/profiles/default/linux/amd64/10.0/sdk /usr/local/portage/chromiumos/profiles/default/linux/amd64/10.0: total 24 drwxr-xr-x 3 bmgordon 5000 4096 Sep 18 2017 chromeos -rw-r--r-- 1 bmgordon 5000 2 Sep 18 2017 eapi -rw-r--r-- 1 bmgordon 5000 423 Sep 18 2017 make.defaults drwxr-xr-x 2 bmgordon 5000 4096 Sep 18 2017 no-multilib -rw-r--r-- 1 bmgordon 5000 29 Sep 18 2017 parent drwxr-xr-x 2 bmgordon 5000 4096 Sep 18 2017 sdk INFO cros_sdk:make_chroot: Updating python Performing Global Updates (Could take a couple of minutes if you have a lot of binary packages.) >>> Jobs: 0 of 0 complete Load avg: 3.11, 2.17, 1.83 Switching to python2.7 INFO cros_sdk:make_chroot: Updating portage >>> Jobs: 0 of 0 complete Load avg: 2.94, 2.15, 1.83 INFO cros_sdk:make_chroot: Updating ncurses >>> Jobs: 0 of 0 complete Load avg: 2.79, 2.13, 1.82 INFO cros_sdk:make_chroot: Updating host toolchain 21:45:50: INFO: Determining required toolchain updates... 21:45:50: INFO: Nothing to update! 21:45:50: INFO: Nothing to clean! INFO cros_sdk:make_chroot: Running emerge curl sudo gentoolkit ... Starting fast-emerge. Building package pbzip2 dev-libs/openssl net-misc/curl sudo app-portage/gentoolkit on root Calculating deps... Deps calculated in 0m1.9s Total: 0 packages, Size of downloads: 0 KiB Done INFO cros_sdk:make_chroot: Updating Perl modules INFO : Chroot is already initialized to 155 INFO cros_sdk:make_chroot: Updating preinstalled build tools Starting fast-emerge. Building package sys-apps/sandbox >=sys-devel/patch-2.7 sys-devel/automake on root Calculating deps... Deps calculated in 0m2.0s Total: 0 packages, Size of downloads: 0 KiB Done INFO : Elapsed time (run_chroot_version_hooks): 0m0s INFO : Updating chroot INFO : Clearing shadow utils lockfiles under / INFO : Bootstrapping depot_tools INFO : Updating the SDK Starting fast-emerge. Building package virtual/target-sdk world on root Calculating deps... Deps calculated in 0m17.1s Total: 0 packages, Size of downloads: 0 KiB Done Scanning Configuration files... Exiting: Nothing left to do; exiting. :) INFO : Running 'eclean -d packages' to clean up stale binpkgs >>> Building file list for packages cleaning... >>> Your packages directory was already clean. INFO : Elapsed time (update_chroot): 0m37s Now using icedtea-bin-8 as your generation-2 system JVM INFO cros_sdk:make_chroot: Elapsed time (make_chroot.sh): 3m20s cros_sdk:make_chroot: All set up. To enter the chroot, run: $ cros_sdk --enter --chroot=/work/bmgordon/tryjob/repository/new-sdk-chroot CAUTION: Do *NOT* rm -rf the chroot directory; if there are stale bind mounts you may end up deleting your source tree too. To unmount and delete the chroot cleanly, use: $ cros_sdk --delete --chroot=/work/bmgordon/tryjob/repository/new-sdk-chroot
,
Apr 27 2018
Here's the set of processes:
python2,145030,bmgordon /usr/local/google/home/bmgordon/chromiumos/chromite/bin/cros tryjob --local...
└─python2,145033 /usr/local/google/home/bmgordon/chromiumos/chromite/scripts/cbuildbot_launch ...
├─python2,145036 /usr/local/google/home/bmgordon/chromiumos/chromite/scripts/cbuildbot_launch ...
├─python2,145040 /usr/local/google/home/bmgordon/chromiumos/chromite/scripts/cbuildbot_launch ...
└─python2,163534 /work/bmgordon/tryjob/repository/chromite/bin/cbuildbot chromiumos-sdk-tryjob ...
├─python2,163537 /work/bmgordon/tryjob/repository/chromite/bin/cbuildbot ...
├─bash,163544 -c...
├─python2,163555 /work/bmgordon/tryjob/repository/chromite/bin/cbuildbot ...
├─python2,163556 /work/bmgordon/tryjob/repository/chromite/bin/cbuildbot ...
├─python2,163560 /work/bmgordon/tryjob/repository/chromite/bin/cbuildbot ...
└─python2,194416 chromite/bin/cbuildbot chromiumos-sdk-tryjob --no-buildbot-tags --debug ...
├─sudo,17193,root SSH_AGENT_PID=8792 SSH_AUTH_SOCK=/tmp/ssh-juT1nRpsOj8m/agent.8709...
│ └─python2,17197 /work/bmgordon/tryjob/repository/chromite/bin/cros_sdk --chroot ...
│ └─python2,17459 /work/bmgordon/tryjob/repository/chromite/bin/cros_sdk ...
│ └─python2,17462 /work/bmgordon/tryjob/repository/chromite/bin/cros_sdk ...
│ └─bash,22539,bmgordon
├─python2,194418 chromite/bin/cbuildbot chromiumos-sdk-tryjob --no-buildbot-tags ...
├─python2,194419 chromite/bin/cbuildbot chromiumos-sdk-tryjob --no-buildbot-tags ...
└─python2,194423 chromite/bin/cbuildbot chromiumos-sdk-tryjob --no-buildbot-tags ...
That innermost bash is stuck in the T state. I'll look for what it's trying to do.
,
Apr 27 2018
It looks like the problem is that it isn't running any specific command, so cros_sdk enters the chroot. I don't see any recent changes to the SDKTest stage, but there have been some recent changes to cros_sdk. I'll take a look and see if one of those might be the problem.
,
Apr 27 2018
Interesting, especially because the same tryjob runs correctly on a builder.
,
May 8 2018
,
May 24 2018
,
May 29 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/11c1b1c6fa4847225e0f690d0e379cfff3852ed0 commit 11c1b1c6fa4847225e0f690d0e379cfff3852ed0 Author: Benjamin Gordon <bmgordon@chromium.org> Date: Tue May 29 20:37:39 2018 sdk_stages: Don't pass empty command in SDKTest When creating the new-sdk-chroot, SDKTest runs a chroot command that doesn't actually run a command. It relies on the side effect of cros_sdk creating the chroot before running the non-existent command. The empty command results in entering the chroot and running bash. On the builders where stdin isn't attached to a terminal, bash immediately exits and this succeeds. On local builds, this causes the tryjob to hang inside the chroot because stdin is normally attached to a tty. Instead of running no command, run /bin/true to get the same chroot-creation effect without any dependence on stdin's status. BUG= chromium:837330 TEST=cros tryjob --local chromiumos-sdk-tryjob Change-Id: I835920a7bf66437cbab0d49e0162723114e894a8 Reviewed-on: https://chromium-review.googlesource.com/1071877 Commit-Ready: Benjamin Gordon <bmgordon@chromium.org> Tested-by: Benjamin Gordon <bmgordon@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [modify] https://crrev.com/11c1b1c6fa4847225e0f690d0e379cfff3852ed0/cbuildbot/stages/sdk_stages.py
,
May 30 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/61e28293b50996bbc9161f3480af6e7d1e6c4c14 commit 61e28293b50996bbc9161f3480af6e7d1e6c4c14 Author: chromite-chromium-autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Wed May 30 01:23:12 2018 Roll src/third_party/chromite 0bdff2b..11c1b1c (1 commits) https://chromium.googlesource.com/chromiumos/chromite.git/+log/0bdff2b..11c1b1c git log 0bdff2b..11c1b1c --date=short --no-merges --format='%ad %ae %s' 2018-05-22 bmgordon@chromium.org sdk_stages: Don't pass empty command in SDKTest Created with: gclient setdep -r src/third_party/chromite@11c1b1c The AutoRoll server is located here: https://chromite-chromium-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. BUG= chromium:837330 TBR=chrome-os-gardeners@chromium.org Change-Id: I9b3c1e73cc7a4ff8cd86892e76b122c1b14077e7 Reviewed-on: https://chromium-review.googlesource.com/1077014 Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#562687} [modify] https://crrev.com/61e28293b50996bbc9161f3480af6e7d1e6c4c14/DEPS
,
May 31 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by dgarr...@chromium.org
, Apr 26 2018