+stevenjb who owns this builder? Don't think it is chromeos infra, it doesn't run cbuildbot.

Nobody seems to know much about this builder. My notes say "Run cros_best_revision. Why?" :) 

cros_best_revision is part of chromite, +vapier@

I have no clue who owns chrome-bot@ or chromeos-commit-bot@. There are lots of references to chrome-bot in chromite.

dgarrett@ made the last non debugging changes to cros_best_revision.py, any thoughts?

akeshet/dgarret/phobbs would prob be best to handle/investigate this

There was another builder with this issue.

We update Puppet to populate the email address in .gitconfig recently, and it seems it's wrong on some builders.

In general, the address in .gitconfig should be in sync with the GoB credentials installed on the builder.

Can anyone explain why some builders use the credentials for "chrome-bot@chromium.org" and some use "chromeos-commit-bot@chromium.org"?

Either way, the .gitconfig change I requested a while back was overly simplistic.

if repo already synced before the gitconfig change, the bad setting might be encoded in the checkout

In that case, a clobber might fix it?

Forced clobber in progress.

Didn't help. I think we need to fix the .gitconfig.

Unfortunately this is still happening. Would it be possible to fix the credentials?

Is this chrome-infra puppet or chromeos puppet?

Chrome Infra.

A while back, I asked for a chance to .gitconfig to add username/email address. My request was in error, since the username/email need to match the GoB credentials on that builder.

Can you give me the hostnames of a working and a broken machine?

Bumping this up to a P0. This determines the version of chromeos used for SimpleChrome which is now a month out of date.

Here are the broken builders:
https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM?numbuilds=200

The slavename for the most recent build (that failed) is vm32-m2. Not sure how that maps to the hostname.

I'm not sure where to find the hostname for a working machine.

What username do you want it to be?

Right now it's:
[user]
        name = chrome-bot
        email = chrome-bot@chromium.org

What do you want there?

remote: ERROR:  In commit 089b3dfdc40518a0e62dc55de9a6cc281d4ca7cd        
remote: ERROR:  committer email address chrome-bot@chromium.org        
remote: ERROR:  does not match your user account.        
remote: ERROR:        
remote: ERROR:  The following addresses are currently registered:        
remote: ERROR:    chromeos-commit-bot@chromium.org  

Is that what you want?  Do I need to set name = chromeos-commit-bot too?

Can you set this on a host manually and run a job to confirm?

About to start a manual test on this builder.

Forcing a build now with a manually applied fix. If this works, I'll undo it, and we'll put the real fix in puppet.

Manual testing failed.... because I typo'd the email address. Sigh.

After manually updating the .gitconfig on that builder to contain the following section, the build turned green.

[user]
        name = chromeos-commit-bot
        email = chromeos-commit-bot@chromium.org

Yes, looking good. Does this need any more work?

Yes, the manual fix was temporary and is undone. Talking to Elliot to work out the way to fix this problem via puppet (and on all builders that might be affected).

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/9982721796069c5de62c28536fcbb5488864c36d

commit 9982721796069c5de62c28536fcbb5488864c36d
Author: Elliott Friedman <friedman@google.com>
Date: Fri Jun 17 01:02:55 2016

Only the Chrome LKGM bot vm32-m2 fixed with this change.  If others break, please re-open.

Re-opening: We seem to have migrated the bots and are facing a similar issue now.

The changes in issue 689830 coincide directly with the Chrome LKGM builder failures we've had every night for the past month. It appears the new(?) slaves aren't configured properly or are not authenticated properly.

Builder: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM?numbuilds=200
Example stdio: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM/builds/1584/steps/cros_best_revision/logs/stdio

00:02:43: INFO: RunCommand: git cl land -f --bypass-hooks -m 'Automated Commit: Committing new LKGM version 9384.0.0 for chromeos.' in /b/build/slave/chromelkgm/build/src
remote: 
remote: Processing changes: refs: 1        
remote: Processing changes: refs: 1, done            
remote: 
remote: ERROR:  In commit fbe3d62b0fce0b938209cdedc9f72bc9bd34debf        
remote: ERROR:  committer email address chromeos-commit-bot@chromium.org        
remote: ERROR:  does not match your user account.        
remote: ERROR:        
remote: ERROR:  The following addresses are currently registered:        
remote: ERROR:    3su6n15k.default@developer.gserviceaccount.com        
remote: ERROR:        
remote: ERROR:  To register an email address, please visit:        
remote: ERROR:  https://chromium-review.googlesource.com/#/settings/contact        
remote: 
remote: 
error: failed to push some refs to 'https://chromium.googlesource.com/a/chromium/src.git'

I've run cros_best_revision.py locally, updating chromeos/CHROMEOS_LKGM to 9391.0.0, since the manifest version is a month behind. (It's the same result as the bot would have had, except that my account is listed as the commit's author.)

https://chromium.googlesource.com/chromium/src/+/07e9a78420c274fd31d0a7f08d4bc719337f4271

Thanks for doing the manual update Michael, good call, that takes the pressure off. Can you paste the command here for future reference and in case fixing the git authentication doesn't go smoothly? Thanks!

To manually run cros_best_revision.py to update chromeos/CHROMEOS_LKGM:

Run the following from the build root (the one containing src and .gclient).

cros chrome-sdk --board=samus # or whichever
cros_best_revision --dryrun
cd src
git show HEAD # confirm the commit makes sense
git cl land --bypass-hooks -m "$(git log lkgm-roll -n 1 --pretty='format:%s')"

The -m for git cl land is because the message is rejected otherwise due to its length.

If cros_best_revision fails with a python error, install the relevant package (optionally using a virtualenv)

If git cl land fails because upstream has new changes, then run:
    git fetch origin master && git rebase
and try the git cl land command again. Repeat as necessary.

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

+xixuan@ - Can you make sure this has the right owner? We need to get this sorted so that Simple Chrome does not continue to fall behind.

I am running cros_best_revision locally now so that we get an update.

I think friedman is the right owner since he made a puppet a change for this.

@dgarrett, can you verify that?

Manually updated Simple Chrome version to 9403.0.0

I'm a little lost here... what do you want?  Do you want the same gitconfig to be written to hosts?  Which masters do you want if that's the case?  See https://chromereviews.googleplex.com/449347013/diff/20001/puppetm/opt/puppet/conf/nodes.yaml

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Please reassign this if it is still something you want to complete.

dgarret@ or dpranke@, can you give friedman@ some more guidance?

Lowering to P-1 since we are working around this for now, but manually updating simple chrome is a PITA and is not maintainable.

The builder in question has started passing now. Does that mean this problem is fixed?

I'm guessing this is fixed, as best I can tell from the problem description. Please reopen and point to a problem if I'm wrong :).

Does anyone understand WHY this builder is fixed?

Unfortunately, it looks like the logs from the recent batch of failures (comment 26&c) seem to be gone. Build #1600 seems to be the oldest I see a log for, and I don't see that error in that log, so hard to say.

We could change this to WontFix (could not reproduce) if that made any difference :).

friedman@ is re-investigating this, encountered a similar error locally.

We tried re-creating src dir but that didnt work.

~/.gitconfig is good:
user]
        name = chromeos-commit-bot
        email = chromeos-commit-bot@chromium.org

Can someone on Infra take a look WRT git cl land auth issues possibly as mentioned in #26 above?

Also, it's worth mentioning this bot moved from vm32-m2 to cros-beefy429-c2 around Feb 14th, which was the last known good commit.

vm32-m2: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM/builds/1548

cros-beefy429-c2: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM/builds/1549

In particular, that builder is configured the way ChromeOS builders are usually configured, and ChromeOS builders normally use "git push" not "git cl land", which might have a different authentication flow.

Yes, and the script it's running uses chromite, which seems to mostly be used for Chrome OS rather than Chrome repos (but should still work).

Good catch about the vm builder. It wouldn't have had a GCE service account.

The error message in #26 mentions the GCE service account.

I'm confused. The most recent builds are passing, and the most recent failures do not appear to have anything to do with the git cl error mentioned in #26:

  chromite.scripts.cros_best_revision.LKGMNotFound: No valid LKGM found. Scores are too low.

but even that's a week old now. What's the problem that needs triaging here?

The problem is indeed the git cl failure in #26, but the git cl step hasn't happened since February due to issue 717020. We can reproduce the git cl failure by manually invoking cros_best_revision on the bot.

Temporarily assigning to friedman to post the output we get today when manually running cros_best_revision on the bot. It's similar to #26 but using a different username.

(The "No valid LKGM found. Scores are too low." failure above is actually WAI. The LKGM builder depends on the canary builders, so if they've been red, there's nothing this bot can do.)

The bot is reaching (and failing) the `git cl land` step now that issue 717020 was fixed. The email address "registered" according to `git cl land` is "3su6n15k.default@developer.gserviceaccount.com" but the bot uses "chromeos-commit-bot@chromium.org".

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM/builds/1627/steps/cros_best_revision/logs/stdio



00:06:33: WARNING: Command failed with retriable error.
return code: 1; command: git cl land -f --bypass-hooks -m 'Automated Commit: Committing new LKGM version 9517.0.0 for chromeos.'
From https://chromium.googlesource.com/a/chromium/src
 * [new branch]                master     -> git-cl-cherry-pick
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
remote: 
remote: ERROR:  In commit 863177a50064d44dd6024cc3708688b01ea72f58        
remote: ERROR:  committer email address chromeos-commit-bot@chromium.org        
remote: ERROR:  does not match your user account.        
remote: ERROR:        
remote: ERROR:  The following addresses are currently registered:        
remote: ERROR:    3su6n15k.default@developer.gserviceaccount.com        
remote: ERROR:        
remote: ERROR:  To register an email address, please visit:        
remote: ERROR:  https://chromium-review.googlesource.com/#/settings/contact        
remote: 
remote: 
error: failed to push some refs to 'https://chromium.googlesource.com/a/chromium/src.git'
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
remote: 
remote: ERROR:  In commit 905470624faee1b0079cfc7a6f8547d7589c2226        
remote: ERROR:  committer email address chromeos-commit-bot@chromium.org        
remote: ERROR:  does not match your user account.        
remote: ERROR:        
remote: ERROR:  The following addresses are currently registered:        
remote: ERROR:    3su6n15k.default@developer.gserviceaccount.com        
remote: ERROR:        
remote: ERROR:  To register an email address, please visit:        
remote: ERROR:  https://chromium-review.googlesource.com/#/settings/contact        
remote: 
remote: 
error: failed to push some refs to 'https://chromium.googlesource.com/a/chromium/src.git'
remote: 
remote: Processing changes: refs: 1
remote: Processing changes: refs: 1, done            
remote: 
remote: ERROR:  In commit d830ddd584ae076b525bd95a3161ba9a4730f7c1        
remote: ERROR:  committer email address chromeos-commit-bot@chromium.org        
remote: ERROR:  does not match your user account.        
remote: ERROR:        
remote: ERROR:  The following addresses are currently registered:        
remote: ERROR:    3su6n15k.default@developer.gserviceaccount.com        
remote: ERROR:        
remote: ERROR:  To register an email address, please visit:        
remote: ERROR:  https://chromium-review.googlesource.com/#/settings/contact        
remote: 
remote: 
error: failed to push some refs to 'https://chromium.googlesource.com/a/chromium/src.git'

Using 50% similarity for rename/copy detection. Override with --similarity.
Description:
Automated Commit: Committing new LKGM version 9517.0.0 for chromeos.
Attempt 1 of 3
Fetching origin/refs/heads/master...
Cherry-picking commit on top of latest refs/heads/master
To https://chromium.googlesource.com/a/chromium/src.git
!	HEAD:refs/heads/master	[remote rejected] (invalid committer)
Done

Attempt 2 of 3
Fetching origin/refs/heads/master...
Cherry-picking commit on top of latest refs/heads/master
To https://chromium.googlesource.com/a/chromium/src.git
!	HEAD:refs/heads/master	[remote rejected] (invalid committer)
Done

Attempt 3 of 3
Fetching origin/refs/heads/master...
Cherry-picking commit on top of latest refs/heads/master
To https://chromium.googlesource.com/a/chromium/src.git
!	HEAD:refs/heads/master	[remote rejected] (invalid committer)
Done

Failed to push. If this persists, please file a bug.

Same issues for me as well on the bot.

This is still happening on the bot with the error in comment #53.

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM/builds/1634

remote: ERROR:  In commit 311093f2536742c4d6da507cdf9627d4525048fa        
remote: ERROR:  committer email address chromeos-commit-bot@chromium.org        
remote: ERROR:  does not match your user account.        

Can someone clarify if this is a configuration issue for the machine (and assign to someone on infra) vs. a bug in the script (and assign to someone on chrome team)?

This is not a labs issue.  The correct credentials are on the machine.  This is an issue with the software stack.  I don't know why it is trying to use the GCE credential instead of the one it is supposed to use.  This was triaged out of labs queue, please don't drop it back there.  It only gives a false sense of hope.  Labs is helpless here outside of being able to re-run the job.

Putting in ChromeOS queue, since this is a CrOS process.

This error occurs when the credentials that git is using to upload (i.e. the value in .gitcookies) do not match the configured user email address (i.e. `git config user.email`). The .gitcookies value is tied to some service account which is registered with gerrit as '3su6n15k.default@developer.gserviceaccount.com'.  Some possibilities:
  * change your `git config user.email` to '3su6n15k.default@developer.gserviceaccount.com'
  * have Labs change the deployed git credentials on that bot to 'chromeos-commit-bot@chromium.org'. I'm not sure that Labs manages that credential though.
  * there might be a way to link 'chromeos-commit-bot@chromium.org' to the '3su6n15k.default@developer.gserviceaccount.com' account, but I'm not 100% certain how to do so; that would require a bug to be filed with the googlesource.com team in b/

That's all I've got off the top of my head.

git cl land, uses gerrit_util to land the cl
gerrit_util.py uses this for authentication
class Authenticator(object):
  """Base authenticator class for authenticator implementations to subclass."""

  def get_auth_header(self, host):
    raise NotImplementedError()

  @staticmethod
  def get():
    """Returns: (Authenticator) The identified Authenticator to use.

    Probes the local system and its environment and identifies the
    Authenticator instance to use.
    """
    if GceAuthenticator.is_gce():
      return GceAuthenticator()
    return CookiesAuthenticator()

So unless SKIP_GCE_AUTH_FOR_GIT is set

 def is_gce(cls):
    if os.getenv('SKIP_GCE_AUTH_FOR_GIT'):
      return False

is_gce() will return most likely true and use google compute engine authentication.

I believe the solution is to add export SKIP_GCE_AUTH_FOR_GIT=1 or something like that to chromeos-bot .profile

I just re-ran it on cros-beefy429-c2 with:
SKIP_GCE_AUTH_FOR_GIT=1 /b/build/slave/chromelkgm/build/chromite/bin/cros_best_revision
and
export SKIP_GCE_AUTH_FOR_GIT=1; /b/build/slave/chromelkgm/build/chromite/bin/cros_best_revision

both have the same error as https://bugs.chromium.org/p/chromium/issues/detail?id=618430#c53.

dbehr@ "git cl land" uses Gerrit REST API to land the CL **if and only iff** the project is configured to use Gerrit in codereview.settings (or if you use git cl land --gerrit), but even in a case "git cl land" would expect the CL to be uploaded first with "git cl upload", which uses git protocol, and so you are still bound by this problem.

Now, I don't know what action you expect from git admin. What iannucci@ wrote above seems like the way to go for me.

It looks like depot_tools in chromiumos manifest is very outdated and it doesn't have this change https://chromium-review.googlesource.com/c/411386/ to skip GCE authentication. So before we can use this variable we need to update our depot_tools.

I just put these CLs to update the pin up for review.

https://chromium-review.googlesource.com/c/503533/
https://chrome-internal-review.googlesource.com/c/374370/

Abandoned in favor of the ones already in-flight.

https://chromium-review.googlesource.com/c/503572

Can someone own this bug?

dbehr, please feel free to reassign as needed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/manifest/+/6558107e56da33bbd8f236ebd742164a381e4f38

commit 6558107e56da33bbd8f236ebd742164a381e4f38
Author: Dominik Behr <dbehr@chromium.org>
Date: Sat May 20 10:55:10 2017

depot_tools: update hash to ToT

We need a change in gerrit_util.py which allows our commit bot to skip GCE
authentication and use .netrc instead. So update depot_tools to ToT so
it includes this change.

CQ-DEPEND=CL:*374269
BUG= chromium:618430 
TEST=pre cq passes

Change-Id: I445e3e3894462b86f84e1c9be2b8066970f15d29
Signed-off-by: Dominik Behr <dbehr@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/503572
Tested-by: Dominik Behr <dbehr@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/6558107e56da33bbd8f236ebd742164a381e4f38/full.xml

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/manifest/+/6558107e56da33bbd8f236ebd742164a381e4f38

commit 6558107e56da33bbd8f236ebd742164a381e4f38
Author: Dominik Behr <dbehr@chromium.org>
Date: Sat May 20 10:55:10 2017

depot_tools: update hash to ToT

We need a change in gerrit_util.py which allows our commit bot to skip GCE
authentication and use .netrc instead. So update depot_tools to ToT so
it includes this change.

CQ-DEPEND=CL:*374269
BUG= chromium:618430 
TEST=pre cq passes

Change-Id: I445e3e3894462b86f84e1c9be2b8066970f15d29
Signed-off-by: Dominik Behr <dbehr@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/503572
Tested-by: Dominik Behr <dbehr@google.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/6558107e56da33bbd8f236ebd742164a381e4f38/full.xml

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/manifest-internal/+/d0e1cccd6b2fbb2a0504762e1ece724747ed7b1a

commit d0e1cccd6b2fbb2a0504762e1ece724747ed7b1a
Author: Dominik Behr <dbehr@google.com>
Date: Sat May 20 10:55:09 2017

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/90985844796a8d6392e60e1d253841fe6e340cf5

commit 90985844796a8d6392e60e1d253841fe6e340cf5
Author: Dominik Behr <dbehr@google.com>
Date: Thu May 25 21:30:51 2017

FWIW, the change in comment #72 alone didn't seem to fix the LKGM builder, see https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/Chrome%20LKGM/builds/1650/steps/cros_best_revision/logs/stdio which does include the env var SKIP_GCE_AUTH_FOR_GIT=1 in the attempt to uprev to 9589.0.0, but it still fails with the same error as comment #26.

I did update the LKGM version manually to 9588.0.0 yesterday in https://chromium.googlesource.com/chromium/src/+/9cec3742b1357cfe8a2db8f7c7df2f17de1c7641 since it was about two weeks out-of-date, but was hoping the automatic process would turn green.

Hi Dominik! I see the LKGM build is still red. Is there something else we need to do? Feel free to assign to me as this weeks gardener if needed.

ChromeOS Infra P1 Bugscrub.

P1 Bugs in this component should be important enough to get weekly status updates.

Is this already fixed?  -> Fixed
Is this no longer relevant? -> Archived or WontFix
Is this not a P1, based on go/chromeos-infra-bug-slo rubric? -> lower priority.
Is this a Feature Request rather than a bug? Type -> Feature
Is this missing important information or scope needed to decide how to proceed? -> Ask question on bug, possibly reassign.
Does this bug have the wrong owner? -> reassign.

Bugs that remain in this state next week will be downgraded to P2.

Doesn't look like the current issue is an Infra>Git>Admin issue:

00:00:47: ERROR: Failed to download CIDB creds from gs.
Can try obtaining your credentials at go/cros-cidb-admin and manually passing it in with --cred-dir.
cros_best_revision: Unhandled exception:
Traceback (most recent call last):
  File "/b/rr/tmpkjWrOZ/w/chromite/bin/cros_best_revision", line 168, in <module>
    DoMain()
  File "/b/rr/tmpkjWrOZ/w/chromite/bin/cros_best_revision", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/rr/tmpkjWrOZ/w/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/rr/tmpkjWrOZ/w/chromite/scripts/cros_best_revision.py", line 293, in main
    cidb_creds = GetCidbCreds(args.cred_dir)
  File "/b/rr/tmpkjWrOZ/w/chromite/scripts/cros_best_revision.py", line 280, in GetCidbCreds
    cidb_creds = cros_cidbcreds.CheckAndGetCIDBCreds()
  File "/b/rr/tmpkjWrOZ/w/chromite/cli/cros/cros_cidbcreds.py", line 33, in CheckAndGetCIDBCreds
    with locking.FileLock(cidb_dir_lock).write_lock():
  File "/b/rr/tmpkjWrOZ/w/chromite/lib/locking.py", line 162, in write_lock
    self._enforce_lock(fcntl.LOCK_EX, message)
  File "/b/rr/tmpkjWrOZ/w/chromite/lib/locking.py", line 85, in _enforce_lock
    self.locking_mechanism(self.fd, flags|fcntl.LOCK_NB)
  File "/b/rr/tmpkjWrOZ/w/chromite/lib/locking.py", line 71, in fd
    self._fd = self._GetFd()
  File "/b/rr/tmpkjWrOZ/w/chromite/lib/locking.py", line 251, in _GetFd
    fd = os.open(self.path, os.W_OK|os.O_CREAT|cloexec, 0o666)
OSError: [Errno 2] No such file or directory: '/b/rr/tmpkjWrOZ/w/.cros_cache/cidb_creds.lock'

Merging into  issue 732579  - We are going to turn down this builder and move the CHROMEOS_LKGM update step to the PFQ master.