New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 632008 link

Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug

Blocked on:
issue 633985
issue 647201
issue 658672
issue 659512
issue 725976



Sign in to add a comment

Move headless build bot to the master waterfall

Project Member Reported by skyos...@chromium.org, Jul 27 2016

Issue description

We'd like to move the headless build bot from the FYI onto the master waterfall. The bot has been green and stable, and moving it onto the master waterfall would allow us to catch regressions more quickly.

Last 200 builds: https://build.chromium.org/p/chromium.fyi/builders/Headless%20Linux%20%28dbg%29?numbuilds=200



 
Cc: jbudorick@chromium.org
Description: Show this description
Blockedon: 633985
Is this still blocked on something? Not trying to rush you or anything -- just wondering if I need to chase someone :)
Blocked on me doing the next steps. :)

We're branching soon, so I'm on health plan mode now. Will get back to this next week.
Sounds good!
Labels: -Pri-3 Pri-2
Status: Started (was: Assigned)
Sorry for me being super slow with this. Taking this now to the top of my queue.
Btw, looks like tests have been failing recently on this bot. Is there any bug attached to that?
Seems that crrev.com/2300703005 is the reason, looking into it.
Thanks both! Having the bot on the main waterfall will help to prevent exactly that kind of breakage :)
Project Member

Comment 11 by bugdroid1@chromium.org, Sep 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/366e76721f37fd25d860c2c3f76d6b43d610abc3

commit 366e76721f37fd25d860c2c3f76d6b43d610abc3
Author: altimin <altimin@chromium.org>
Date: Fri Sep 09 18:05:04 2016

[headless] Call StopLocalDevToolsHttpHandler only when needed.

This fixes test crashes on headless buildbot.

BUG= 632008 

Review-Url: https://codereview.chromium.org/2322333003
Cr-Commit-Position: refs/heads/master@{#417630}

[modify] https://crrev.com/366e76721f37fd25d860c2c3f76d6b43d610abc3/headless/lib/browser/headless_browser_main_parts.cc
[modify] https://crrev.com/366e76721f37fd25d860c2c3f76d6b43d610abc3/headless/lib/browser/headless_browser_main_parts.h

And it's green now!
Project Member

Comment 14 by bugdroid1@chromium.org, Sep 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/4720403d60ca3522c66f873633c4f19b0a463508

commit 4720403d60ca3522c66f873633c4f19b0a463508
Author: perezju <perezju@chromium.org>
Date: Thu Sep 15 10:29:39 2016

Add linux_chromium_headless_dbg trybot builder and slaves

Add slaves and builder configuration for new linux_chromium_headless_dbg
trybot.

New slaves were provided at: crbug.com/633985

BUG= 632008 

Review-Url: https://codereview.chromium.org/2330003002

[modify] https://crrev.com/4720403d60ca3522c66f873633c4f19b0a463508/masters/master.tryserver.chromium.linux/master.cfg
[modify] https://crrev.com/4720403d60ca3522c66f873633c4f19b0a463508/masters/master.tryserver.chromium.linux/slaves.cfg

Project Member

Comment 16 by bugdroid1@chromium.org, Sep 15 2016

Blockedon: 647201
Project Member

Comment 18 by bugdroid1@chromium.org, Sep 15 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/aea4f15000592282cdcfe34533b4a63ce78def53

commit aea4f15000592282cdcfe34533b4a63ce78def53
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 15 20:59:23 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/e1d15cd7795cc1c5465983af8a49669242857107 Add rtc_stats_unittests to the test suite (ehmaldonado@chromium.org)
  https://crrev.com/37294244d9d54d639a083d382da68c50f8fe14e5 Add linux_chromium_headless_dbg trybot (perezju@chromium.org)
  https://crrev.com/f0f4c4a27746dee8a53234f03e452dba556deb55 Reland of Enable the ninja up-to-date check for Android builders (agrieve@chromium.org)
  https://crrev.com/2744734cf3aa1d8a02d0f0b7abf99635e9f2aa48 Revert of Enable the ninja up-to-date check for Android builders (patchset #1 id:1 of https://codereview.chromium.org/2343563003/ ) (agrieve@chromium.org)
  https://crrev.com/1575589a1d8f61fee45ec53ccbf77f073fd38847 recipe_modules/chromite: Use "build_type". (dnj@chromium.org)
  https://crrev.com/5799bab975aa3eda6785f36d9f7879549074b28b Reland of Enable the ninja up-to-date check for Android builders (agrieve@chromium.org)
  https://crrev.com/ecdf065c9d2a6e28d86c8ad333432b497158ee7a chromium.android: Enable swarming on Android arm64 builder (bpastene@chromium.org)
  https://crrev.com/7a71133c89aa2a770fc2326188306cf6a3a6f1e1 Revert of Enable the ninja up-to-date check for Android builders (patchset #1 id:1 of https://codereview.chromium.org/2343953002/ ) (agrieve@chromium.org)
  https://crrev.com/c65424c419105e0720cc0de09fb9e7a36c3d4601 Pass ninja -n in the ninja -d explain step to avoid more work (agrieve@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG=646165, chromium:627996 , 646838 , 632008 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2346723003
Cr-Commit-Position: refs/heads/master@{#418970}

[modify] https://crrev.com/aea4f15000592282cdcfe34533b4a63ce78def53/infra/config/recipes.cfg

Sami, the headless trybot is now available for manual runs in code reviews under "master.tryserver.chromium.linux" on Choose trybots.

I'm also sending out https://codereview.chromium.org/2342813002 for review, so we start running it automatically on 10% of commits.
headless_trybot.png
306 KB View Download
The bot itself is here: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_headless_dbg

Probably a good idea to give it a try :)
Project Member

Comment 22 by bugdroid1@chromium.org, Sep 16 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/7d6811c4c58a9e280bca7bf65a8dfde001bef42a

commit 7d6811c4c58a9e280bca7bf65a8dfde001bef42a
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Sep 16 23:37:07 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/ba9de887ec268453298caa8641de3cc91dbfb818 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/e1d15cd7795cc1c5465983af8a49669242857107 Add rtc_stats_unittests to the test suite (ehmaldonado@chromium.org)
  https://crrev.com/37294244d9d54d639a083d382da68c50f8fe14e5 Add linux_chromium_headless_dbg trybot (perezju@chromium.org)
  https://crrev.com/f0f4c4a27746dee8a53234f03e452dba556deb55 Reland of Enable the ninja up-to-date check for Android builders (agrieve@chromium.org)
  https://crrev.com/2744734cf3aa1d8a02d0f0b7abf99635e9f2aa48 Revert of Enable the ninja up-to-date check for Android builders (patchset #1 id:1 of https://codereview.chromium.org/2343563003/ ) (agrieve@chromium.org)
  https://crrev.com/1575589a1d8f61fee45ec53ccbf77f073fd38847 recipe_modules/chromite: Use "build_type". (dnj@chromium.org)
  https://crrev.com/5799bab975aa3eda6785f36d9f7879549074b28b Reland of Enable the ninja up-to-date check for Android builders (agrieve@chromium.org)
  https://crrev.com/ecdf065c9d2a6e28d86c8ad333432b497158ee7a chromium.android: Enable swarming on Android arm64 builder (bpastene@chromium.org)
  https://crrev.com/7a71133c89aa2a770fc2326188306cf6a3a6f1e1 Revert of Enable the ninja up-to-date check for Android builders (patchset #1 id:1 of https://codereview.chromium.org/2343953002/ ) (agrieve@chromium.org)
  https://crrev.com/c65424c419105e0720cc0de09fb9e7a36c3d4601 Pass ninja -n in the ninja -d explain step to avoid more work (agrieve@chromium.org)
  https://crrev.com/8091fad9d075ecec116e1740360ff955a7654c4a Remove use_isolate from Marshmallow 64 bit Tester (bpastene@chromium.org)
  https://crrev.com/2f0a7c064259dc0e0bf375c2d1771544f5647a1c Changed sequence of package_build step to be before package_build_for_bisect (miimnk@google.com)
  https://crrev.com/194bd0247efb60371dbcae87731f13dce4fa7934 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/d2845f7e87aa167cf7df78003b5768f948803d07 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/eb686c79229ef3cf3b39e6a0807f4c98e3e93cf9 Add BuildBucket manifest scheduling support. (dnj@chromium.org)
  https://crrev.com/537cdb699a70d5d55f3eefb923eae1dc5104a8eb Add goma to wasm waterfall (sbc@chromium.org)
  https://crrev.com/b6d326d6f6f9608bba307109cc4a1c5efc5946bb Run the ninja "up-to-date" check for all compiles, but just as an fyi (agrieve@chromium.org)
  https://crrev.com/07a33586eff8451bbc84366055c3cfa36e7b6fea Roll recipe dependencies (nontrivial). (recipe-roller@chromium.org)
  https://crrev.com/79d8d758281609aa49b9b9af7e18cc187c2ee814 Disable CompilerInfoCache for a while (shinyak@chromium.org)
  https://crrev.com/a8f07009101d46553ea5d4bce1353ecb365dd3a5 Revert "Roll recipe dependencies (nontrivial)." (tandrii@chromium.org)
  https://crrev.com/cbca79967afa4f28df6b0ede337db6138eeb167b Add asan=1 to GYP_DEFINES for Dart asan builds (whesse@google.com)
  https://crrev.com/25b922d8edc1927c3af4bb4a8f7db6434096dca0 WebRTC: Disable iOS API Framework Builder. (ehmaldonado@chromium.org)
  https://crrev.com/769dbabae3b7ff11023dd1b393fcd2a12e165c65 Make sure goma_ctl is running during wasm_llvm build (sbc@chromium.org)
  https://crrev.com/2e89429885f7c2503565cb12d564d0f6f58a42f1 Fix Webkit capitalization for N CTS tests. (mikecase@chromium.org)
  https://crrev.com/e549894cc4f86f3f2107e3686d561ff83331a267 crashpad/continuous: Use bot_update. (dnj@chromium.org)
  https://crrev.com/f74386f7e19ff3f32693d222262d1a3ee79799ef Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
  https://crrev.com/1a9a50a459c9e3c081719d3e835cac213c319136 Roll recipe dependencies (trivial). (recipe-roller@chromium.org)
depot_tools:
  https://crrev.com/f46c20fcee6e6a0a7d75788847632cd4ac18e2e9 codereview.settings: add GIT_NUMBER_FOOTER setting. (tandrii@chromium.org)
  https://crrev.com/5d0a0421ce27046c94177511a05699316ec8097a git_cl: update outdated TODOs. (tandrii@chromium.org)
  https://crrev.com/73449b0bd49eab1e152f419102123d734896da98 Gerrit git cl land: abort if not uploaded. (tandrii@chromium.org)
  https://crrev.com/bf42940536f6c0c123a9e6278c20bc38729b3717 git cl land to refs/pending: remove unused arg. (tandrii@chromium.org)
  https://crrev.com/7475196d4c32d66e1c199bf24945b7ae28255e13 repo: update to v1.12.17-cr1 (vapier@chromium.org)
  https://crrev.com/adcd4b78d5f35da535ccf82a221afb9fa389f631 presubmit_support: Remove a noisy logging.debug() (thakis@chromium.org)
  https://crrev.com/18ca30ca804679ee624a52e73017d234a8c0008f Teach bot_update to remove partially deleted git repos. (vadimsh@chromium.org)
  https://crrev.com/972ac5040176acd90c8a1ce412f75d19f77cc4e8 bot_update: ensure correct depot_tools checkout is used. (tandrii@chromium.org)
  https://crrev.com/15a248123d9032061486cd2d4b3f64369c93a9a5 Revert of bot_update: ensure correct depot_tools checkout is used. (patchset #2 id:20001 of https://codereview.chromium.org/2346973003/ ) (tandrii@chromium.org)
  https://crrev.com/7f245d07b2282f9847072fccddf7162a7e632a2d Bump git-on-windows bleeding edge version to 2.10.0. (vadimsh@chromium.org)
  https://crrev.com/6ac12ffd596e338c43e25dc3889e8ac552c2e885 Make bot_update.py print git version it uses. (vadimsh@chromium.org)
  https://crrev.com/7e16cf303221bbcf81d632924e19ddc888da9c3b owners.py: partial fix for owners-check perf regression (nick@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG=646165,none,646838,632008,webrtc:6372,chromium:647812,642493,647446,chromium:627996,642793,635641,645662,chromium:632203,647046,609225,642759

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2345413002

[modify] https://crrev.com/7d6811c4c58a9e280bca7bf65a8dfde001bef42a/infra/config/recipes.cfg

Cc: phajdan.jr@chromium.org yyanagisawa@chromium.org shinyak@chromium.org
I want to investigate why sometimes the bot takes too long to compile, since this could be a blocker to have it on CQ and the main waterfall.

From issue 633985, Sami noted:

~~~
Here's two traces from the Linux x64 builder and the Headless builder around the same time:

Linux: http://chromium-build-stats.appspot.com/ninja_log/2016/08/03/build48-m1/ninja_log.build48-m1.chrome-bot.20160803-082648.12505.gz/trace.html

Headless: http://chromium-build-stats.appspot.com/ninja_log/2016/08/03/slave16-c1/ninja_log.slave16-c1.chrome-bot.20160803-094644.11126.gz/trace.html

First of all looks like Headless is using -j50 and Linux has -j160 which probably explains some of the difference.

Each object also takes a lot longer to build on the Headless bot. My unsubstantiated theory is that since Headless probably uses unique compiler flags, it doesn't benefit from goma's caching and effectively needs to build every affected file from scratch. Another possibility is that it somehow has less bandwidth to goma. Maybe the machines are different too?
~~~

The number for the -j parameter comes from here: https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/compile.py#340

would it make sense to special-case it for the headless builders too?

+cc'ing some potential reviewers of that change.
Is it possible for you to point out builds in the next time?  It requires me certain processes to find out compiler_proxy.INFO.

Linux: gs://chrome-goma-log/2016/08/03/build48-m1/compiler_proxy.build48-m1.chrome-bot.log.INFO.20160803-081125.12513.gz

compile: # of tasks: 33475
   replies:
     local finish, no goma : 31
     should fallback : 24
     goma success (cache hit) : 32756
     local finish, abort goma : 118
     goma success : 546

Headless: gs://chrome-goma-log/2016/08/03/slave16-c1/compiler_proxy.slave16-c1.chrome-bot.log.INFO.20160803-080342.11134.gz

compile: # of tasks: 0


Headless does not use goma.
D'oh! ... getting ready a CL to fix that.
Cc: eseckler@chromium.org alexclarke@chromium.org
Sami, can you try kicking off some more manual try jobs? Hopefully this time they should work better.
Blockedon: 658672
Blockedon: 659512
Project Member

Comment 35 by bugdroid1@chromium.org, Nov 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eefe833ae27a644a219eb0d4b047ff2e3aee7c0d

commit eefe833ae27a644a219eb0d4b047ff2e3aee7c0d
Author: perezju <perezju@chromium.org>
Date: Wed Nov 09 09:55:01 2016

Turn on linux_chromium_headless_dbg CQ builder at 10%

BUG= 632008 

Review-Url: https://codereview.chromium.org/2342813002
Cr-Commit-Position: refs/heads/master@{#430906}

[modify] https://crrev.com/eefe833ae27a644a219eb0d4b047ff2e3aee7c0d/infra/config/cq.cfg

https://build.chromium.org/p/chromium.fyi/builders/Headless%20Linux%20%28dbg%29?numbuilds=200
is looking quite good after starting the experiment.

Paweł, should we ramp it up now? What's a good number to go up from the current 10%?
Labels: -Pri-2 Pri-1
Ping, any change we could bump the percentage a bit? We're seeing about a couple of build breakages a week when stuff sneaks past the builder.
Project Member

Comment 39 by bugdroid1@chromium.org, Dec 22 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70747375bb3bd1f015faace785dd7a2fa93f0097

commit 70747375bb3bd1f015faace785dd7a2fa93f0097
Author: perezju <perezju@chromium.org>
Date: Thu Dec 22 15:36:53 2016

Set linux_chromium_headless_rel CQ experiment at 100%

BUG= 632008 

Review-Url: https://codereview.chromium.org/2595523002
Cr-Commit-Position: refs/heads/master@{#440426}

[modify] https://crrev.com/70747375bb3bd1f015faace785dd7a2fa93f0097/infra/config/cq.cfg

This bot seems to be chugging along happily (at 100% of cq):

https://build.chromium.org/p/chromium.fyi/builders/Headless%20Linux%20%28dbg%29?numbuilds=200

Could we make it block the cq now?
Cc: dpranke@chromium.org
The headless bot isn't doing what you might expect. args_file in a mixin in mb currently has no effect, though +dpranke has a CL out to fix that. The headless mb mixin relies on that behavior working: https://codesearch.chromium.org/chromium/src/tools/mb/mb_config.pyl?rcl=45196266f164836fca6d41e8f9cfad0eee0b91a6&l=1754

Moving this to CQ blocking should wait until that fix has landed.
#41: Interesting! Has that always been the case? Maybe about a month ago (before we had --headless) these tests should have failed if they weren't truly compiled in headless mode.
I'm not sure how long args_file hasn't worked in a mixin, but it looks like headless has only been using that feature since 2017-02-07: https://chromium.googlesource.com/chromium/src/+/23c2f1952f5845194633c1ccd50e7408576c7465
Oh, didn't realize the mixin wouldn't work when I tried to simplify the configs :O

If that fix from dpranke@ will take a while, I'm happy to revert that patch and reland it after dpranke@'s fix.
Okay, see what Dirk thinks. Agreed that if mixins take some time to get in place, let's revert back to the old config to make sure we're testing the right thing.
*let's see
It should land today, so I'd just leave it.
Hey, any news about the patch in #47?-) Not sure where to look for it.
Woo, excellent, thank you! As far as I can tell there's now nothing else blocking making this bot cq-blocking, right?
The builder is still on the FYI waterfall. We should move it to the main waterfall first. I can post a CL for that later today.
The current FYI builder is a debug builder. The tryserver is a release tryserver.

I'd rather not have a release tryserver in the CQ without also having a release waterfall builder. Can we flip the FYI builder to release, or do you see a lot of debug-only failures and want to have both?
Owner: dpranke@chromium.org
Owner: skyos...@chromium.org
skyostil@ can you take a look at my questions in #c52?
Owner: dpranke@chromium.org
Turning the FYI builder from debug to release sounds fine to me. Breakage so far has mostly been either failing to build or failing some tests.
Labels: OS-Linux
Components: Infra>Client>Chrome
Components: Internals>Headless
Friendly ping -- was there something else blocking making this bot block the CQ?
Nothing other than me having the time to make the changes. I'll post a CL to make the builder non-experimental now, and follow up with one to move the waterfall builder as well.
Blockedon: 725976
Thank you! Not trying to rush anything, just checking in :)
Project Member

Comment 63 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e498ef57c4bca46019a872695dcbba2fbef82e4

commit 1e498ef57c4bca46019a872695dcbba2fbef82e4
Author: dpranke <dpranke@chromium.org>
Date: Wed May 24 18:46:24 2017

Make linux_chromium_headless_rel non-experimental in the CQ.

The bot has been running as a 100% experiment for a long time,
which is just a waste of resources :). This CL makes the bot
a full-fledged member of the CQ, meaning that failures will
block a patch from landing.

The waterfall bot is still on the FYI waterfall (Headless Linux
(dbg)) and we should move it to the main waterfalls (and make it
a release bot to match the CQ config), but I think we can get
away with not doing that before landing this, as long as I get
to it later today.

R=skyostil@chromium.org
BUG= 632008 

Review-Url: https://codereview.chromium.org/2905563003
Cr-Commit-Position: refs/heads/master@{#474374}

[modify] https://crrev.com/1e498ef57c4bca46019a872695dcbba2fbef82e4/infra/config/cq.cfg

Status: Fixed (was: Started)
As far as I can tell we're done here? Thanks for the help everyone!

Sign in to add a comment