FWIW, over the PFQ runs, all three of the failures had passed at least once on a prior run (maybe we should change the PFQ logic to retry only failed builds?). This particular build of Chrome is highly likely to be safe.

I'm pretty sure the uprevchrome tool is busted:

(cr) ((c16e45e...)) achuith@achuithz620 ~/trunk/src/scripts $ cros uprevchrome --pfq-build 2915321 -- bug chromium:881559  --cred-dir=/mnt/host/source/.cache/cidb_creds --draft --debug
16:42:42: DEBUG: Cache dir lookup.
16:42:42: DEBUG: Configured cache_dir to '/mnt/host/source/.cache'
16:42:42: INFO: Running cidb query on pid 150223, repr(query) starts with 'SHOW DATABASES'
16:42:43: WARNING: /mnt/host/source/chromite/lib/cidb.py:236: Warning: 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
  cur.execute("SET SESSION sql_mode='STRICT_ALL_TABLES'")

16:42:43: WARNING: /mnt/host/source/chromite/lib/cidb.py:236: Warning: Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.
  cur.execute("SET SESSION sql_mode='STRICT_ALL_TABLES'")

16:42:43: INFO: Created cidb engine readonly@104.197.188.23 for pid 150223
16:42:43: INFO: Running cidb query on pid 150223, repr(query) starts with 'SHOW TABLES'
16:42:44: INFO: Running cidb query on pid 150223, repr(query) starts with 'SELECT MAX(schemaVersion) from schemaVersionTable'
16:42:44: INFO: Created a SchemaVersionedMySQLConnection, sqlalchemy version 0.9.2
16:42:45: INFO: Running cidb query on pid 150223, repr(query) starts with <sqlalchemy.sql.selectable.Select at 0x7f137fc66d10; Select object>
16:42:45: INFO: Running cidb query on pid 150223, repr(query) starts with 'SELECT id, build_config, start_time, finish_time, status, waterfall, build_number, builder_name, pl
Setting up working directory...
16:42:45: INFO: RunCommand: git clone https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay /tmp/uprevchrome_O8ERHu/pub_overlay --reference /mnt/host/source/src/third_party/chromiumos-overlay --branch master --single-branch in /tmp/uprevchrome_O8ERHu/pub_overlay
16:44:29: INFO: RunCommand: git clone https://chrome-internal.googlesource.com/chromeos/overlays/chromeos-partner-overlay /tmp/uprevchrome_O8ERHu/priv_overlay --reference /mnt/host/source/src/private-overlays/chromeos-partner-overlay --branch master --single-branch in /tmp/uprevchrome_O8ERHu/priv_overlay
Preparing CLs...
16:44:42: INFO: Checking remote refs.
16:44:44: INFO: git fetch origin refs/pfq/staging_pfq_branch_2915321:staging_pfq_branch_2915321_21b43a43a69ce13cbb659a227274fdc8
16:44:47: INFO: git checkout staging_pfq_branch_2915321_21b43a43a69ce13cbb659a227274fdc8
16:44:47: INFO: git fetch origin refs/pfq/staging_pfq_branch_2915321:staging_pfq_branch_2915321_21b43a43a69ce13cbb659a227274fdc8
16:44:48: INFO: git checkout staging_pfq_branch_2915321_21b43a43a69ce13cbb659a227274fdc8
16:44:49: INFO: git pull --rebase origin master
16:44:51: INFO: Upload CLs to Gerrit.
Counting objects: 17, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (17/17), done.
Writing objects: 100% (17/17), 18.53 KiB | 3.71 MiB/s, done.
Total 17 (delta 10), reused 0 (delta 0)
remote: Resolving deltas: 100% (10/10)
remote: It seems you're using an old version of Git (git/2.16.4,gzip(gfe)). Please update the client following the instructions in go/old-git-client
remote: Processing changes: refs: 1, done    
To https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay
 ! [remote rejected]           HEAD -> refs/drafts/master (private changes are disabled)
error: failed to push some refs to 'https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay'
16:44:58: ERROR: cros uprevchrome failed before completing.
cros: Unhandled exception:
Traceback (most recent call last):
  File "/mnt/host/source/chromite/bin/cros", line 169, in <module>
    DoMain()
  File "/mnt/host/source/chromite/bin/cros", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/mnt/host/source/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/mnt/host/source/chromite/scripts/cros.py", line 66, in main
    code = _RunSubCommand(subcommand)
  File "/mnt/host/source/chromite/scripts/cros.py", line 51, in _RunSubCommand
    return subcommand.Run()
  File "/mnt/host/source/chromite/cli/cros/cros_uprevchrome.py", line 373, in Run
    self.UprevChrome(work_dir, self.options.pfq_build, build_number)
  File "/mnt/host/source/chromite/cli/cros/cros_uprevchrome.py", line 339, in UprevChrome
    debug_level=logging.NOTICE, reviewers=reviewers)
  File "/mnt/host/source/chromite/lib/git.py", line 1221, in UploadCL
    return GitPush(git_repo, local_branch, remote_ref, **kwargs)
  File "/mnt/host/source/chromite/lib/git.py", line 1248, in GitPush
    **kwargs)
  File "/mnt/host/source/chromite/lib/git.py", line 762, in RunGit
    return cros_build_lib.RunCommand(['git'] + cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 647, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: git push https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay HEAD:refs/drafts/master
cmd=['git', 'push', 'https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay', 'HEAD:refs/drafts/master'], cwd=/tmp/uprevchrome_O8ERHu/pub_overlay

Looks like the author nxia no longer works on ChromeOS. pprabhu@ - who currently owns this tool?

I wonder if disabling private CLs in Gerrit is causing this failure:
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-dev/PdcVEpbiIug

This should be assigned to somebody from the build / CI teams.

Likely lannm@ as he's been looking at build stuff this week.

Aviv / I no longer take care of chromite tools ( at least in a timely manner ;) )

I've manually prepared CLs based on uprevchrome's output:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1211435
https://chrome-internal-review.googlesource.com/c/chromeos/overlays/chromeos-partner-overlay/+/674629

Need a reviewer.

I'll file a separate bug for uprevchrome's failure to upload CLs. Disabling private CLs seems to be a red herring.

mmm i think the tool is WAI.  you passed --draft, and private/draft have been disabled on our repos (as noted in comment #4).  so don't pass --draft and there's no problem.

if you're going to merge a CL, then you really shouldn't be using --draft/private either.  all merged CLs better be public before they're merged.

Thanks Mike, dropping --draft worked. I'm looking at a YAQS entry for the uprev tool that recommends it.

Should we remove the --draft option from the tool since this isn't supported any longer?

The CLs are here:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1212463
https://chrome-internal-review.googlesource.com/c/chromeos/overlays/chromeos-partner-overlay/+/674689

in general, i don't think we should go through and burn down the --draft/--private options everywhere because some GoB instances disabled them.  remember, it's a runtime server option, not the Gerrit protocol changing.

unrelated to the GoB settings, i think it makes sense to drop --draft from the chrome pin tool.  as mentioned, we never want to merge private CLs, and these CLs aren't secret (they're mechanically reverting code already committed).

i suspect the flag dates back to before we had private/wip status, and draft (largely) functioned as wip does now.  then gerrit converted all "draft" usage to "private".

Ok, I'll update the YAQS directives to avoid --draft (along with other updates for cidb credentials).

We had a natural PFQ uprev, so hallelujah!

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8936096562268633472