Split ChromeSDK stage into package / test stages |
||||||||||||||||
Issue descriptionextracted from an e-mail conversation: -- I am looking at this trybot summary: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/3999 the simple chrome stage seems to have succeeded. But I am very suspicious of it: see: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/3999/steps/SimpleChromeWorkflow/logs/stdio This seems to have started at 21:22 and finished by 21:40... and it seems most of the time was building the dependencies, which was done by 21:35.. This looks very short to me. 18 mins to finish the whole simple chrome? I seem to remenber that when the builders were grouped this stage would take around an hour. Also notice that the summary at the end of the log is wrong, it says that the stage took 5 seconds... -- We investigated a little more and found: -- The operative command is: 21:23:00: INFO: RunCommand: sudo 'USE=-cros-debug chrome_internal' 'ROOT=/tmp/tmpXtRkYj/sysroot' 'CROS_SUDO_KEEP_ALIVE=unknown' -- /mnt/host/source/chromite/bin/parallel_emerge '--board=falco' --usepkgonly --noreplace app-arch/bzip2 chromeos-base/chromeos-fonts dev-libs/atk dev-libs/glib dev-libs/nspr '>=dev-libs/nss-3.12.2' dev-libs/libxml2 dev-libs/dbus-glib x11-libs/cairo x11-libs/pango '>=media-libs/alsa-lib-1.0.19' media-libs/fontconfig media-libs/freetype x11-libs/libdrm media-libs/minigbm media-libs/libpng '>=media-sound/adhd-0.0.1-r310' net-misc/wget virtual/opengles sys-apps/pciutils virtual/udev sys-libs/libcap sys-libs/pam sys-libs/zlib x11-libs/libva x11-libs/libxkbcommon x11-misc/xkeyboard-config chromeos-base/gestures chromeos-base/libevdev app-accessibility/brltty chromeos-base/protofiles '>=dev-util/gperf-3.0.3' '>=dev-util/pkgconfig-0.23' dev-vcs/git I see that --usepkgonly flag in there, and I see from the logs that everything that was built was built from prebuilt (see the word binary in front of each package in the dependency tree). So, not sure what it is supposed to be doing, but it's definitely just emerging a bunch of binary prebuilts. -- and: -- Relevant code: cbuildbot/stages/chrome_stages.py:258 Haven't tried to trace in detail, but looks like we are doing some pre-work in parallel on 253, but then doing the real chrome sdk work only if a config flag requests it after 255. Maybe that flag is off for this config? -- and: -- it looks like non-leader boards are not building chrome in the sdk. see below, but the builder output is very confusing because it does show it is trying to build simple chrome and it is spending 20 minutes in it... non-leader "lumpy-release": { "_template": "release", "active_waterfall": "chromeos", "boards": [ "lumpy" ], "buildslave_type": "gce_beefy", "chrome_sdk_build_chrome": false, "critical_for_chrome": true, "important": true, "vm_tests": [] }, leader: "parrot-release": { "_template": "release", "active_waterfall": "chromeos", "boards": [ "parrot" ], "critical_for_chrome": true, "important": true -- If all of this is true, then the non-leader release configs should just not execute the simple chrome stage. It is very confusing to have a simple chrome stage in there that is not doing anything.
,
Jun 1 2016
,
Jun 2 2016
,
Jun 2 2016
On email davidjames@ said: -- Perhaps the confusion here lies from the name of the stage. Originally this stage was called the "ChromeSDK" stage -- the reason why we named it "ChromeSDK" is because that's exactly what it does -- it packages up the Chrome SDK. On grouped builders, typically we package SDKs on all of the boards (so that you can choose any of the boards in the simple chrome workflow) but we only test on the leader board (since building Chrome for N boards is prohibitively expensive). The BuildPackages stage typically works the same way and Chrome is not built for non-leader boards and is instead pulled from a binary package. On builders which are configured as such, the ChromeSDK stage can also build Chrome and follow the Simple Chrome workflow, but that's optional -- it's just a way of testing the ChromeSDK. -- I though the purpose was to test the Simple chrome workflow and I guess it was because of the name of the stage. What David describes makes sense to me. So, maybe the fix is just to change the name of the stage.
,
Jun 2 2016
What name do you suggest?
,
Jun 2 2016
(Pass to me, and I'll do the rename, if you want)
,
Jun 2 2016
I will let you guys decide on the name... ChromeSDK seems better but it does not show that you are sometimes testing the simple chrome workflow... Maybe you should have 2 separate stages:PackageChromeSDK and TestChromeSDK. Second one only executed by leader boards. I am not the best owner for this.. I am no chromite expert. If you really want me to help with this, I will. But it will take me a while before I get time to fix this...
,
Jun 2 2016
I only passed it to you to pick a better name. Splitting the stages seems reasonable, I'll look into that but it'll take a bit longer (and be fairly low priority).
,
Jun 3 2016
,
Jun 7 2016
Thanks all for triaging and taking this on! I like the splitting solution a lot. While I agree this is lower priority than a lot of the other things we are working on, splitting this will avoid a lot of potential gardener confusion.
,
Jun 30 2016
,
Dec 22 2016
,
Feb 1 2017
,
Feb 1 2017
,
Feb 1 2017
,
Feb 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c414446696adcc858ca11869032778cb8a7c5073 commit c414446696adcc858ca11869032778cb8a7c5073 Author: Don Garrett <dgarrett@google.com> Date: Wed Feb 08 04:14:30 2017 SimpleChromeWorkflow: Split into build/test stages. SimpleChromeWorkflow -> SimpleChromeArtifacts TestSimpleChromeWorkflow We prepare artifacts to support Simple Chrome Workflow on many builders, but only run test builds with them on leader boards. Split the artifact/test parts into different stages so that it's easy to understand where tests did and did not run. With this change, we'll see SimpleChromeArtifacts everywhere we used to see SimpleChromeWorkflow, and see an additional stage 'TestSimpleChromeWorkflow' on the builders that do additional testing. BUG= chromium:616293 TEST=run_tests Change-Id: I7329d268c1607d94699032699a40540d90f91113 Reviewed-on: https://chromium-review.googlesource.com/435712 Commit-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/c414446696adcc858ca11869032778cb8a7c5073/cbuildbot/stages/chrome_stages.py [modify] https://crrev.com/c414446696adcc858ca11869032778cb8a7c5073/cbuildbot/stages/chrome_stages_unittest.py [modify] https://crrev.com/c414446696adcc858ca11869032778cb8a7c5073/cbuildbot/builders/simple_builders.py
,
Feb 8 2017
Let me know if there are any problems with the new stages!
,
Feb 8 2017
Fantastic! I looked at a recent PFQ build, and both stages succeeded, horray. https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-generic-chromium-pfq/builds/9690 One quirk: In the SimpleChromeArtifacts stage, it says "duration: 6 mins, 33 secs" However, looking at the 'Build stages timeline' output, SimpleChromeArtifacts appears to extent until the end of TestSimpleChromeWorkflow.
,
Feb 8 2017
The way I implemented it, the test stage is a sub-stage of the artifact stage. We do a lot of that, but it confuses our stats. If we can ever refactor SimpleBuilder class, it'll be easier to get rid of the sub-stages and fix this problem. We're working towards that, but it's a slow background project.
,
Feb 8 2017
OK, that makes sense, it's not very important. The important part is that we should now have the ability to skip the TestSimpleChromeWorkflow stage on the PFQ if we so choose.
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Aug 3 2017
Closing. Please reopen it if its not fixed. Thanks! |
||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||
Comment 1 by llozano@chromium.org
, May 31 2016