The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/439c6033c0ed17450d95aa5e99ead1d72fc03970

commit 439c6033c0ed17450d95aa5e99ead1d72fc03970
Author: Caroline Tice <cmtice@google.com>
Date: Thu Mar 31 20:55:10 2016

[cbuildbot] Fix build_type of toolchain-minor buildbot.

Toolchain minor buildbot is failing in the waterfall, probably due
to not having the right build type.  This CL changes its build
type to 'toolchain' and updates its timeout to match.

BUG= chromium:599619 
TEST=Ran run_tests in chromite/cbuildbot.

Change-Id: Ia3349d34d52f4e38dc836a060fdfbb4c2af3b023
Reviewed-on: https://chromium-review.googlesource.com/336459
Commit-Ready: Caroline Tice <cmtice@chromium.org>
Tested-by: Caroline Tice <cmtice@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/439c6033c0ed17450d95aa5e99ead1d72fc03970/cbuildbot/config_dump.json
[modify] https://crrev.com/439c6033c0ed17450d95aa5e99ead1d72fc03970/cbuildbot/chromeos_config.py

The toolchain-minor error message is


@@@STEP_FAILURE@@@
08:06:46: ERROR: Master build id 788868 has full_version R51-8158.0.0, while slave version is R51-8158.0.0-b3133.
08:06:46: INFO: Running cidb query on pid 40719, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x4fafa10>
08:06:46: INFO: Running cidb query on pid 40719, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x4fafd90>

16:08:04: ERROR: Master build id 754760 has full_version R51-8055.0.0,
while slave version is R51-8055.0.0-b3108.


  The snippet of the code is
         if master_full_version != my_full_version:
          raise failures_lib.MasterSlaveVersionMismatchFailure(
              'Master build id %s has full_version %s, while slave version is '
              '%s.' % (master_id, master_full_version, my_full_version))

 I think it should truncate the '-b3108' when comparing the
full_version, what do you think?

Comment #3 was actually from Yunlian (I copied it from an email...)

Toolchain-minor is *still* failing with the same issue.  Do we need to stop and restart the waterfall?  If so, how do we do that?

See https://uberchromegw.corp.google.com/i/chromeos/builders/internal-toolchain-minor/builds/3135

That slave config is running the wrong flavor of Sync stage. It is running plain vanilla "Sync" which creates a new manifest from scratch (and a unique version number accordingly) rather than MasterSlaveLKGMSync which it should be running.

The fix required is to chromeos_config.py. The way that config is being created is fishy to me. In particular, it inherits from "toolchain-minor" which itself inherits from _cros_sdk, and hence it has "builder_class_name": "sdk_builders.ChrootSdkBuilder". That seems wrong, I'm pretty sure that is the culprit and that that builder class doesn't run MasterSlaveLKGMSync.

Why is toolchain-minor inheriting from _cros_sdk? I think removing that inheritance is the fix.

To answer your questions: This is the way it was originally set up, before we started making our recent buildbot changes.  Would I just remove the _cros_sdk, or would I need to replace it with something else?

Probably just remove it, unless ChroodSdkBuilder is actually providing functionality that you need.

When I remove _cros_sdk from the toolchain-minor configuration, the chromeos_config_unittest fails with the following error:

ERROR: [chromite.cbuildbot.chromeos_config_unittest] CBuildBotTest.testAllBoardsExist
Verifies that all config boards are in _all_boards.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/host/source/chromite/lib/timeout_util.py", line 170, in TimeoutWrapper
    func(*args, **kwargs)
  File "/mnt/host/source/chromite/cbuildbot/chromeos_config_unittest.py", line 587, in testAllBoardsExist
    for board in config['boards']:
TypeError: 'NoneType' object is not iterable

======================================================================
FAIL: [chromite.cbuildbot.chromeos_config_unittest] CBuildBotTest.testBoards
Verify 'boards' is explicitly set for every config.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/host/source/chromite/lib/timeout_util.py", line 170, in TimeoutWrapper
    func(*args, **kwargs)
  File "/mnt/host/source/chromite/cbuildbot/chromeos_config_unittest.py", line 125, in testBoards
    "Config %s doesn't have a list of boards." % build_name)
AssertionError: Config toolchain-minor doesn't have a list of boards.

----------------------------------------------------------------------
Ran 63 tests in 4.493s

FAILED (failures=1, errors=1)


which looks like toolchain-minor doesn't have any board specified, but when I look in the re-generated config_dump.json file, it shows

        "boards": [
            "x86-alex",
            "stumpy",
            "daisy",
            "lakitu"
        ],

for toolchain-minor.

I tried specifying the boards field specifically in chromeos_config.py (for toolchain-minor), but that doesn't seem to work either.  Do you know what I'm missing or doing wrong?

Never mind, I think I've got this figured out now.

See https://chromium-review.googlesource.com/#/c/338030

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/95688e0f5ea0089a091234eebeb15d1fb40f3de1

commit 95688e0f5ea0089a091234eebeb15d1fb40f3de1
Author: Caroline Tice <cmtice@google.com>
Date: Fri Apr 08 21:09:39 2016

[cbuildbot] Fix issue with toolchain-minor build error.

Toolchain-minor buildbot keeps failing during build with
ERROR: Master build id 788868 has full_version R51-8158.0.0, while
slave version is R51-8158.0.0-bxxx.  This CL tries to fix this
issue.

BUG= chromium:599619 
TEST=Ran the run_tests script in cbuildbot directory.

Change-Id: I31e9c8f31c38d6aac3a67463a40ce45d90509c08
Reviewed-on: https://chromium-review.googlesource.com/338030
Commit-Ready: Caroline Tice <cmtice@chromium.org>
Tested-by: Caroline Tice <cmtice@chromium.org>
Reviewed-by: Yunlian Jiang <yunlian@chromium.org>

[modify] https://crrev.com/95688e0f5ea0089a091234eebeb15d1fb40f3de1/cbuildbot/config_dump.json
[modify] https://crrev.com/95688e0f5ea0089a091234eebeb15d1fb40f3de1/cbuildbot/chromeos_config_unittest.py
[modify] https://crrev.com/95688e0f5ea0089a091234eebeb15d1fb40f3de1/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c5c4a5f69aec2080ff9a07882a3def840c6058d5

commit c5c4a5f69aec2080ff9a07882a3def840c6058d5
Author: Caroline Tice <cmtice@google.com>
Date: Mon Apr 11 21:19:12 2016

[cbuildbot] Continuing attempts to fix issue with minor-toolchain.

The buildbot is still failing, in the BuildReexecutionFinished stage
with error messages like:

ERROR: Master build id 795732 has full_version R52-8179.0.0, while slave
version is R52-8179.0.0-b3138

This CL is yet another attempt to fix the problem.

BUG= chromium:599619 
TEST=Ran run_tests in the cbuildbot directory.

Change-Id: Ic4c84c37962de99e39b744f6e49a097c94120f1f
Reviewed-on: https://chromium-review.googlesource.com/338190
Commit-Ready: Caroline Tice <cmtice@chromium.org>
Tested-by: Caroline Tice <cmtice@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/c5c4a5f69aec2080ff9a07882a3def840c6058d5/cbuildbot/config_dump.json
[modify] https://crrev.com/c5c4a5f69aec2080ff9a07882a3def840c6058d5/cbuildbot/waterfall_layout_dump.txt
[modify] https://crrev.com/c5c4a5f69aec2080ff9a07882a3def840c6058d5/cbuildbot/config_lib.py
[modify] https://crrev.com/c5c4a5f69aec2080ff9a07882a3def840c6058d5/cbuildbot/chromeos_config_unittest.py
[modify] https://crrev.com/c5c4a5f69aec2080ff9a07882a3def840c6058d5/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4e46c5a50c2b6e4dd6c580db46269022f9c58870

commit 4e46c5a50c2b6e4dd6c580db46269022f9c58870
Author: Caroline Tice <cmtice@google.com>
Date: Fri Apr 22 15:41:06 2016

[cbuildbot] Fix toolchain buildbots.

Re-arrange internal-minor-toolchain and a grouped buildbot, and
update the HWtests for llvm-toolchain-group and
internal-minor-toolchain.

BUG= chromium:599619 
TEST=Ran unittests; successfully tested with trybot.

Change-Id: Iacaf1c1cd225fcb9c9c8fe97706973df8b2d68b9
Reviewed-on: https://chromium-review.googlesource.com/340247
Commit-Ready: Caroline Tice <cmtice@chromium.org>
Tested-by: Caroline Tice <cmtice@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/4e46c5a50c2b6e4dd6c580db46269022f9c58870/cbuildbot/config_dump.json
[modify] https://crrev.com/4e46c5a50c2b6e4dd6c580db46269022f9c58870/cbuildbot/waterfall_layout_dump.txt
[modify] https://crrev.com/4e46c5a50c2b6e4dd6c580db46269022f9c58870/cbuildbot/config_lib.py
[modify] https://crrev.com/4e46c5a50c2b6e4dd6c580db46269022f9c58870/cbuildbot/chromeos_config_unittest.py
[modify] https://crrev.com/4e46c5a50c2b6e4dd6c580db46269022f9c58870/cbuildbot/chromeos_config.py