It's trying to run a local tryjob on the builder. The command needs updating. I'm running a local test now of an updated command.

Also, looks like I need to pass through a more arguments for branch util.

Further, it would be nice if we could pass git-cache directories to local tryjobs to make the sync MUCH lighter weight.

Sorry I forgot about how the BranchUtilTest is actually run :(

I expected some pain like this, actually, there has been less than I was expecting.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/17f1f5ce3642c06579ce544ef82fecea9cf6e236

commit 17f1f5ce3642c06579ce544ef82fecea9cf6e236
Author: Don Garrett <dgarrett@google.com>
Date: Fri Sep 15 03:14:09 2017

cros tryjob: Add support for build_util options.

Add, and passthrough the build_util tryjob specific options
  --branch-name, --delete-branch, --rename-to,
  --force-create, --skip-remote-push.

Also, start using --no-buildbot-tags for local tryjobs. These tags
aren't normally useful except inside a buildbot wrapper. In addition
we run local tryjobs on builders for the branch util tests, and the
stages from these 'sub' builds shouldn't appear on the waterfall.

BUG= chromium:764999 
TEST=cros tryjob --yes --local  --version 9939.0.0 \
       --skip-remote-push --branch-name=test_branch \
       branch-util

Change-Id: I20ed206a7c483f5ab1b1472d87c9a8ba88223e3f
Reviewed-on: https://chromium-review.googlesource.com/666141
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/17f1f5ce3642c06579ce544ef82fecea9cf6e236/cli/cros/cros_tryjob_unittest.py
[modify] https://crrev.com/17f1f5ce3642c06579ce544ef82fecea9cf6e236/cli/cros/cros_tryjob.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/36266e8b3878d356fc54d32b4188686ba24515b0

commit 36266e8b3878d356fc54d32b4188686ba24515b0
Author: Don Garrett <dgarrett@google.com>
Date: Fri Sep 15 03:14:09 2017

BranchUtilTestStage: Use cros tryjob for testing.

RunBranchUtilTest ran a local tryjob using cbuildbot local. Switch it
to cros tryjob --local.

BUG= chromium:764999 
TEST=run_tests -q

Change-Id: I26bf89ac8a274328139e35442fa2f0911dd3947a
Reviewed-on: https://chromium-review.googlesource.com/666142
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/36266e8b3878d356fc54d32b4188686ba24515b0/cbuildbot/commands.py

The test is fixed, but the logging is not.

cbuildbot_launch needs to support --no-buildbot-tags to fix the logging.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/fbbccec70e97d1679475eb6936c28e04da28002a

commit fbbccec70e97d1679475eb6936c28e04da28002a
Author: Don Garrett <dgarrett@google.com>
Date: Wed Sep 20 22:17:48 2017

cbuildbot_launch: Respect --no-buildbot-tags.

When running a nested build, cbuildbot_launch was still generating
stage tags, despite --no-buildbot-tags being set, which causes the
buildbot output to be confusing. Stop doing that.

BUG= chromium:764999 
TEST=run_tests -q

Change-Id: I0732040611c1c9cdc47d59a4aafc9ce832030c52
Reviewed-on: https://chromium-review.googlesource.com/676183
Tested-by: Don Garrett <dgarrett@chromium.org>
Trybot-Ready: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/fbbccec70e97d1679475eb6936c28e04da28002a/scripts/cbuildbot_launch.py

Should now be fixed, waiting for a build to verify.

The tryjob is running the way I think it should, but it's trying to insert CIDB entries.

These are the startup commands:

08:21:21: INFO: RunCommand: /b/c/cbuild/repository/chromite/bin/cros tryjob --local --yes --skip-remote-push --branch-name test_branch --version 9962.0.0 --buildroot /tmp/cbuildbot-tmpdS5AFk/tmp3ULAli branch-util in /b/c/cbuild/repository
Verifying patches...
08:21:21: CRITICAL: RunCommand: /b/c/cbuild/repository/chromite/scripts/cbuildbot_launch --buildroot /tmp/cbuildbot-tmpdS5AFk/tmp3ULAli --remote-trybot --debug --no-buildbot-tags --skip-remote-push --branch-name test_branch --version 9962.0.0 branch-util in /tmp/cbuildbot-tmpdS5AFk/tmp3ULAli


But... it gets this error:

08:48:24: DEBUG: ending retries with error: <class 'sqlalchemy.exc.IntegrityError'>((IntegrityError) (1062, "Duplicate entry '1-master-release-chromeos-0' for key 'buildbot_generation'") 'INSERT INTO `buildTable` (master_build_id, buildbot_generation, builder_name, waterfall, build_number, build_config, bot_hostname, start_time, important, buildbucket_id) VALUES (%s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP, %s, %s)' (None, 1, 'master-release', 'chromeos', 0, u'branch-util', 'build10-m2.golo.chromium.org', True, None))
08:48:24: ERROR: Error: (IntegrityError) (1062, "Duplicate entry '1-master-release-chromeos-0' for key 'buildbot_generation'") 'INSERT INTO `buildTable` (master_build_id, buildbot_generation, builder_name, waterfall, build_number, build_config, bot_hostname, start_time, important, buildbucket_id) VALUES (%s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP, %s, %s)' (None, 1, 'master-release', 'chromeos', 0, u'branch-util', 'build10-m2.golo.chromium.org', True, None)
 If the buildbucket_id to insert is duplicated to the buildbucket_id of an old build and the old build was canceled because of a waterfall master restart, please ignore this error. Else, the error needs more investigation. More context:  crbug.com/679974  and  crbug.com/685889 

STEP_FAILURE
08:48:24: ERROR: <class 'sqlalchemy.exc.IntegrityError'>: (IntegrityError) (1062, "Duplicate entry '1-master-release-chromeos-0' for key 'buildbot_generation'") 'INSERT INTO `buildTable` (master_build_id, buildbot_generation, builder_name, waterfall, build_number, build_config, bot_hostname, start_time, important, buildbucket_id) VALUES (%s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP, %s, %s)' (None, 1, 'master-release', 'chromeos', 0, u'branch-util', 'build10-m2.golo.chromium.org', True, None)
Traceback (most recent call last):
  File "/tmp/cbuildbot-tmpdS5AFk/tmp3ULAli/repository/chromite/lib/failures_lib.py", line 228, in wrapped_functor
    return functor(*args, **kwargs)
  File "/tmp/cbuildbot-tmpdS5AFk/tmp3ULAli/repository/chromite/cbuildbot/stages/report_stages.py", line 284, in PerformStage
    raise e
IntegrityError: (IntegrityError) (1062, "Duplicate entry '1-master-release-chromeos-0' for key 'buildbot_generation'") 'INSERT INTO `buildTable` (master_build_id, buildbot_generation, builder_name, waterfall, build_number, build_config, bot_hostname, start_time, important, buildbucket_id) VALUES (%s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP, %s, %s)' (None, 1, 'master-release', 'chromeos', 0, u'branch-util', 'build10-m2.golo.chromium.org', True, None)


I would have thought that with both --remote-try and and explicit --debug, it would NOT write to CIDB.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/06f4ada706cd879e364c52b490347b78235b8d33

commit 06f4ada706cd879e364c52b490347b78235b8d33
Author: Don Garrett <dgarrett@google.com>
Date: Tue Sep 26 20:36:17 2017

cros tryjob: Do not set --remote-tryjob for local tryjobs.

I was incorrectly using the --remote-tryjob when running local
tryjobs. This caused the tryjob to behave as if it were on a builder,
when it shouldn't have.

Other code was detecting that it wasn't really a builder and blocking
the CIDB connection, UNLESS it was run on an actual builder, such as
for the branch util test.

BUG= chromium:764999 
TEST=run_tests + running a local tryjob.

Change-Id: I39af97be075d7f43ca68499251d46c9ca3caf6d4
Reviewed-on: https://chromium-review.googlesource.com/680019
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/06f4ada706cd879e364c52b490347b78235b8d33/scripts/cbuildbot.py
[modify] https://crrev.com/06f4ada706cd879e364c52b490347b78235b8d33/cli/cros/cros_tryjob_unittest.py
[modify] https://crrev.com/06f4ada706cd879e364c52b490347b78235b8d33/cli/cros/cros_tryjob.py

Once again, believed fixed. Waiting for the next canary run.