"package build" step slowly increasing in duration for some builders; now at ~2 hours |
||||||||
Issue descriptionSee https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Linux%20Builder?limit=1000 where cycle time went from ~20min a few weeks ago to now about 2.5 hours. All due to the package build step. Something's probably not getting cleared out from the build dir. Will look into it.
,
Jun 8 2018
All those files total about 71G. Not surprisingly, a new set gets created every time chrome/VERSION changes: https://chromium.googlesource.com/chromium/src/+log/master/chrome/VERSION
,
Jun 9 2018
,
Jun 9 2018
+govind Currently the Linux package files have the version in the filename, but this means they don't get cleaned up when the version increments, leading to huge build dirs like on the Linux Builder. If we change the filename to something deterministic like chromium-browser-stable.deb, would the release team be able to handle this change easily?
,
Jun 9 2018
I rm'ed all chromium-browser-* files in the bot's checkout and it's back down to 20min cycle times: https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Linux%20Builder/102817 So the slow-down is certainly due to those files.
,
Jun 9 2018
If the package_build step was smarter about what .deb / .rpm files to package, that can avoid the slowdown, but not the disk space usage.
,
Jun 9 2018
+ abdulsyed@ & mmoss@, PTAL comment #4. Thank you.
,
Jun 9 2018
The long term right answer is to get rid of the step package_build. We should only send isolated hashes around.
,
Jun 11 2018
Re #4, no, release could not easily handle a change in the package names, and I don't think that's the right fix anyhow. Maybe the build system should have an option (if it doesn't already) to clean up files that it doesn't know about? Accumulated build cruft isn't exclusively a problem of package files (e.g. from file renames, obsolete components, etc.). The packages just highlight the issue because of the frequency of name changes and their size. And on a related note, why are those packages so large? 200MB is like 4X the size of official packages. Are the binaries not being stripped or something?
,
Jun 11 2018
IMHO, I still stand by my c#4 that the package output name should be deterministic, but it seems folks disagree. +dpranke does the build system allow us to do what mmoss is suggesting in c#9?
,
Jun 12 2018
The build system does not currently have that option, no. We should get rid of package_build in favor of isolated hashes as maruel@ says, but that wouldn't solve the underlying problem of unbounded disk growth on the builder. I thought the disk cleanup tool should, though. +friedman, am I misremembering that?
,
Jun 12 2018
Here's the list of all paths cleanup_disk watches: https://chrome-internal.googlesource.com/infra/infra_internal/+/master/go/src/infra_internal/tools/cleanup_disk/cmd/cleanup_disk/main.go#102 I don't think we've ever had that touching build dirs (maybe it used to be covered by the "build.dead" dir that buildbot generates?). We have a step called "clean local files" running in the early-parts of every android-flavored chromium build (eg: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/android-kitkat-arm-rel/6616) https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_android/api.py?rcl=1aaf79598cf2a5b44e98e75003b95ed31f8f09da&l=151 It's old and likely obsolete, but we could generalize that for every chromium build, and tweak it to target these package files. If dpranke/jbud endorse that, I can work on adding it.
,
Jun 12 2018
buildbot used build.dead to manage directories for builders that no longer existed on a bot. One relatively easy answer would be to say that if we're building an installer, the build should be a clobber build, but that might be overly draconian. I'd rather not have an ad-hoc thing that just deleted a few files, so the only other option I see is to do something based on the build graph itself, sort of like an inverted `ninja -t clean` (delete everything *except* the stuff in the build graph). That functionality doesn't currently exist but probably wouldn't be too hard to add to either ninja or GN (both of which could do it).
,
Aug 1
Any updates on this? I suspect this problem is what's killing chromium-internal-arm-builder-rel https://ci.chromium.org/buildbot/internal.client.clank/chromium-internal-arm-builder-rel/?limit=200
,
Aug 1
Not entirely sure it's related. The duration of package_build seems pretty consistent, but it gets stuck at some point and hangs with no stdout for 20min then gets killed. Whatever it is, I clobbered the bot. So that'll prob help. As for this bug, it can mostly be duped into bug 754104 at this point. Though I think I recall dpranke following up on his build graph idea in some other bug...
,
Sep 11
FTR, just ran into the same thing in bug 880932 . Clobbered the bot https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Builder and its package_build step duration went from 2hr to 7min. Releasing this since I'm not actively tracking down & fixing what's generating them. Also lowering pri since this becomes much less disruptive once we've killed the package_build step (bug 754104)
,
Jan 3
,
Jan 3
I ran into this over the holiday break. The .deb and .rpm files built up on the bot since the end of October, and started breaking the bot. Not sure what to do here...
,
Jan 3
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by bpastene@chromium.org
, Jun 8 2018