betty-arc64-paladin times out during testing |
|||
Issue descriptionThe failure is here: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929714750922317488 and the previous successfull run is here: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929733868048387216 If you compare the failing build timeline: https://storage.cloud.google.com/chromeos-image-archive/betty-arc64-paladin/R72-11266.0.0-rc2/timeline-stages.html to the successful one: https://storage.cloud.google.com/chromeos-image-archive/betty-arc64-paladin/R72-11266.0.0-rc1/timeline-stages.html it looks like every stage is just a little bit slower. CommitQueueSync takes an extra 3 minutes, InitSDK takes an extra 2 minutes, BuildPackages takes an extra 9 minutes, etc. So, by the time TastVMTest runs it just doesn't have enough time to complete. In the good run it only took 5 minutes to finish successfully, and in the bad run the global timer expired and we killed it after 1.5 minutes. Given that we are apparently bumping up against the global timer on betty-arc64 for normal runs and that this doesn't seem to be caused by any single stage hanging or having a problem, should we just increase the timeout a bit?
,
Nov 16
Yes, we can increase the timeout but the thing that worries me the most is that a single package, arc-llvm, is taking more than an hour to build holding up everything else. Ben, is there anything that can be done to improve that build time?
,
Nov 16
arc-llvm is a big package and takes a long time to build. I'll check again to see if there's anything more we can turn off, but we haven't changed it since the last time we went through the shrinking exercise in August, so I don't expect any huge gains. I think arc-llvm is sort of an unintended casualty of the revdeps stuff. It ideally shouldn't be getting rebuilt, but it has ~550 packages as indirect reverse dependencies, so it doesn't take much to trigger rebuilding it.
,
Nov 21
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/83196281cee36ed873e7798a7a6b29252c647eb9 commit 83196281cee36ed873e7798a7a6b29252c647eb9 Author: Benjamin Gordon <bmgordon@chromium.org> Date: Wed Nov 21 13:52:55 2018 arc-llvm: Attempt to reduce build times arc-llvm takes so long to rebuild that it can cause timeouts on paladin runs. Ideally it shouldn't be getting rebuilt except when we change it, but the current long chain of portage deps triggers rebuilds relatively often. As a short-term improvement, three changes that will help reduce the arc-llvm build times: * Disable building libLTO, since we don't use it. * Disable building the runtimes, since we don't use any of them. * Patch out installation of static libraries, since dependent packages only use the dynamic library. The combination of these reduces the package size by ~2GB. On an idle workstation, it reduces compile time by 15-20%. The effect on a heavily loaded build machine is harder to determine, but it should help a little. BUG= chromium:906078 TEST=./build_packages --board=grunt Change-Id: Ic93efc65e4c2507c7489182c136af73372600aa8 Reviewed-on: https://chromium-review.googlesource.com/1344373 Commit-Ready: Benjamin Gordon <bmgordon@chromium.org> Tested-by: Benjamin Gordon <bmgordon@chromium.org> Reviewed-by: Daniel Kurtz <djkurtz@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/83196281cee36ed873e7798a7a6b29252c647eb9/sys-devel/arc-llvm/files/arc-llvm-6.0.0-no-static-libraries.patch [modify] https://crrev.com/83196281cee36ed873e7798a7a6b29252c647eb9/sys-devel/arc-llvm/arc-llvm-6.0.0.ebuild [rename] https://crrev.com/83196281cee36ed873e7798a7a6b29252c647eb9/sys-devel/arc-llvm/arc-llvm-6.0.0-r5.ebuild
,
Nov 21
To the extent that this is caused by long arc-llvm compile times, this should be reduced. |
|||
►
Sign in to add a comment |
|||
Comment 1 by jclinton@chromium.org
, Nov 16Labels: OS-Chrome