CL:769658 is out for review, but might take a while since pprabhu@ is out.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/9f3fb697850adedf1ab91e4b605a9acfe142c565

commit 9f3fb697850adedf1ab91e4b605a9acfe142c565
Author: Don Garrett <dgarrett@google.com>
Date: Thu Nov 16 00:07:36 2017

branch-util: Fix branch-util-tryjob for testing.

branch-util has lots of special handling, which was broken for
branch-util-tryjob. Add the new config to the special case handling.

BUG= chromium:784985 
TEST=cros tryjob --cbuildbot branch-util-tryjob --branch-name test_branch --version 10127.0.0

Change-Id: Id60bfbf54148ede3d7af2da732c630df78c7c088
Reviewed-on: https://chromium-review.googlesource.com/769658
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/9f3fb697850adedf1ab91e4b605a9acfe142c565/scripts/cbuildbot.py
[modify] https://crrev.com/9f3fb697850adedf1ab91e4b605a9acfe142c565/cbuildbot/stages/branch_stages_unittest.py
[modify] https://crrev.com/9f3fb697850adedf1ab91e4b605a9acfe142c565/cbuildbot/commands.py
[modify] https://crrev.com/9f3fb697850adedf1ab91e4b605a9acfe142c565/cbuildbot/stages/branch_stages.py

Should be fixed.

This command now works:

cros tryjob --local branch-util-tryjob --branch-name test_branch --version 10127.0.0

Verified.

I used the following command but it doesn't work?

cros tryjob branch-util-tryjob --local --branch-name firmware-fizz-10138.B --version 10138.0.0

The command failed with

************************************************************
** FAIL ManifestVersionedSync (0:04:17) with InfrastructureFailure
************************************************************

Stack trace,

<class 'chromite.cbuildbot.repository.SrcCheckOutException'>: return code: 1; command: repo --time sync --force-sync -n
cmd=['repo', '--time', 'sync', '--force-sync', '-n'], cwd=/usr/local/google/home/yueherngl/tryjob/repository
Traceback (most recent call last):
  File "/usr/local/google/home/yueherngl/tryjob/repository/chromite/lib/failures_lib.py", line 229, in wrapped_functor
    return functor(*args, **kwargs)
  File "/usr/local/google/home/yueherngl/tryjob/repository/chromite/cbuildbot/stages/sync_stages.py", line 779, in PerformStage
    self.ManifestCheckout(new_manifest)
  File "/usr/local/google/home/yueherngl/tryjob/repository/chromite/cbuildbot/stages/sync_stages.py", line 412, in ManifestCheckout
    self.repo.Sync(next_manifest)
  File "/usr/local/google/home/yueherngl/tryjob/repository/chromite/cbuildbot/repository.py", line 684, in Sync
    raise SrcCheckOutException(err_msg)
SrcCheckOutException: return code: 1; command: repo --time sync --force-sync -n

Thanks.

The actual error is on the kernel repo in the experimental checkout:
...
Ferror: Cannot fetch chromiumos/third_party/kernel (GitError: chromiumos/third_party/kernel update-ref: fatal: 31b04e605374a58b37caede7000e95b5fee27cb
7^0: not a valid SHA1
)
Exception in thread Thread-27:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/subcmds/sync.py", line 276, in _FetchProjectList
    success = self._FetchHelper(opt, project, *args, **kwargs)
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/subcmds/sync.py", line 321, in _FetchHelper
    cache_dir=opt.cache_dir)
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/project.py", line 1350, in Sync_NetworkHalf
    self._InitMRef()
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/project.py", line 2443, in _InitMRef
    self._InitAnyMRef(R_M + self.manifest.branch)
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/project.py", line 2455, in _InitAnyMRef
    self.bare_git.UpdateRef(ref, dst, message=msg, detach=True)
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/project.py", line 2752, in UpdateRef
    self.update_ref(*cmdv)
  File "/usr/local/ssd960gb/tmp/tryjob/repository/.repo/repo/project.py", line 2826, in runner
    (self._project.name, name, p.stderr))
GitError: chromiumos/third_party/kernel update-ref: fatal: 31b04e605374a58b37caede7000e95b5fee27cb7^0: not a valid SHA1



error: Exited sync due to fetch errors
real    0m6.926s
...

The hash is in the manifest https://chrome-internal.googlesource.com/chromeos/manifest-versions/+/master/buildspecs/64/10138.0.0.xml

Yet it does not exist today

...
bhthompson@ragnarok:/usr/local/ssd960gb/chromiumos-1/src/third_party/kernel/experimental$ git show 31b04e605374a58b37caede7000e95b5fee27cb7
fatal: bad object 31b04e605374a58b37caede7000e95b5fee27cb7
...

So I think this may be a matter of losing history, the way we treat the experimental branch may be a problem, we probably want to ensure any hashes that exist stay existing, so we probably want to just change what experimental points at as opposed to going over it with --force?

Ah, this is annoying. We can not ensure that hashes on kernel-experimental stay around. That would defeat the purpose of the branch. If checking out a specific SHA on it can not be disabled, we'll have to remove it completely, and I'll have to find some other means to test pre-published branches next time around.

adding deputies as dgarrett@ is out

Switching to newer version 10138.0.0 (thanks Bernie for the advise), dry-run completed successfully with,

cros tryjob branch-util-tryjob --local --branch-name=firmware-fizz-10139.B --version=10139.0.0

However, to actually create the branch, neither

cros tryjob branch-util-tryjob --local --production --branch-name=firmware-fizz-10139.B --version=10139.0.0

nor

cros tryjob branch-util --local --production --branch-name=firmware-fizz-10139.B --version=10139.0.0

worked.

The former finished without error (but the branch wasn't created/pushed) while the latter error out with message,

06:14:41: ERROR: Refusing to run non-tryjob config as a tryjob.
Please "repo sync && cros tryjob --list branch-util" for alternatives.
See go/cros-explicit-tryjob-build-configs-psa.

Thanks.

That tryjob is very special and breaks our normal rules for local tryjobs in a variety of ways. I keep forgetting it when reasoning about how tryjobs are supposed to work.

I'm going to investigate creating the "cros branch-util" command. If it's straight forward enough, I'll create it instead of adding more hacks to keep that tryjob running.

dgarrett to look again if there's a simple hack to restore the testing

In the current iteraction, this doesn't affect testing.

However, branch-util MUCH run as a local production tryjob. Local production tryjobs are disabled by design.

The quick hack is to create an exception for this build config.

The "right" fix is to create the "cros branch-util" command, and remove this tryjob. I started that process, but it's a much bigger refactor than I was hoping and isn't something I can finish in an afternoon.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8a5dd427130d19a37a309570f68191bd708ce12a

commit 8a5dd427130d19a37a309570f68191bd708ce12a
Author: Don Garrett <dgarrett@google.com>
Date: Tue Dec 12 21:52:10 2017

branch-util: Fix branch-util-tryjob for testing.

branch-util has lots of special handling, which was broken for
branch-util-tryjob. Add the new config to the special case handling.

BUG= chromium:784985 
TEST=cros tryjob --cbuildbot branch-util-tryjob --branch-name test_branch --version 10127.0.0

Change-Id: I718437dc85d5bead0530f0eef3b0bc1239cec7e6
Reviewed-on: https://chromium-review.googlesource.com/815675
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/8a5dd427130d19a37a309570f68191bd708ce12a/scripts/cbuildbot.py
[modify] https://crrev.com/8a5dd427130d19a37a309570f68191bd708ce12a/cbuildbot/stages/branch_stages_unittest.py
[modify] https://crrev.com/8a5dd427130d19a37a309570f68191bd708ce12a/cbuildbot/commands.py
[modify] https://crrev.com/8a5dd427130d19a37a309570f68191bd708ce12a/cbuildbot/stages/branch_stages.py

I took at stab at creating "cros branch-util", but it was a bigger refactor than I could do quickly.

Instead, I've uploaded CL:826489 to fix the tryjob. Hopefully, I can take a longer stab at that refactor early next year.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0c6e0276b59fde63e793b87d009380e26458d34f

commit 0c6e0276b59fde63e793b87d009380e26458d34f
Author: Don Garrett <dgarrett@google.com>
Date: Thu Dec 14 10:02:43 2017

cbuildbot: Hack to allow branch-util again.

Add the necessary hacks to allow the following command to run.
  cros tryjob --local --production branch-util <branch args>

If we ever replace the branch-util tryjob with a local command, we
should remove these hacks, and either remove the ability to run
--local --production, or force --debug when doing it.

BUG= chromium:784985 
TEST=run_tests
     Difficult to run local test tryjob until committed.

Change-Id: Ie8cdd19db122a36d26ab75745bb62f69c2564bf8
Reviewed-on: https://chromium-review.googlesource.com/826489
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/0c6e0276b59fde63e793b87d009380e26458d34f/scripts/cbuildbot.py
[modify] https://crrev.com/0c6e0276b59fde63e793b87d009380e26458d34f/cli/cros/cros_tryjob_unittest.py
[modify] https://crrev.com/0c6e0276b59fde63e793b87d009380e26458d34f/cbuildbot/cbuildbot_unittest.py
[modify] https://crrev.com/0c6e0276b59fde63e793b87d009380e26458d34f/cli/cros/cros_tryjob.py

Thanks Don. It would probably another few weeks before we need create another branch. We will try it out then.