New issue
Advanced search Search tips

Issue 810943 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

LKGR finder fails for WebRTC due to getting a commit hash from Chromium's builds

Project Member Reported by oprypin@chromium.org, Feb 9 2018

Issue description

Last succeeding build
https://build.chromium.org/p/client.webrtc.fyi/builders/WebRTC%20lkgr%20finder/builds/33902
First failing build
https://build.chromium.org/p/client.webrtc.fyi/builders/WebRTC%20lkgr%20finder/builds/33903

2018-02-08 16:38:00,356 |    DEBUG | infra.services.lkgr_finder.lkgr_lib | Collating builder Linux Builder (dbg)
2018-02-08 16:38:00,359 |    DEBUG | infra.libs.git | Running `git show -s --format=%B f05cf6a09c983e3e7fc054d8199d82663f55733a` with {'cwd': '/b/c/b/WebRTC_lkgr_finder/infra/infra/services/lkgr_finder/workdir/webrtc'}
Traceback (most recent call last):
  File "/b/c/b/WebRTC_lkgr_finder/infra/run.py", line 29, in <module>
    sys.exit(run_helper.run_py_main(sys.argv[1:], RUNPY_PATH, ENV_PATH, 'infra'))
  File "/b/c/b/WebRTC_lkgr_finder/infra/bootstrap/run_helper.py", line 123, in run_py_main
    runpy.run_module(args[0], run_name='__main__', alter_sys=True)
  File "/usr/lib/python2.7/runpy.py", line 176, in run_module
    fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/services/lkgr_finder/__main__.py", line 321, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/services/lkgr_finder/__main__.py", line 207, in main
    builds, lkgr_builders, repo)
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/services/lkgr_finder/lkgr_lib.py", line 377, in CollateRevisionHistory
    builder_history, keyfunc=lambda x: x[0])
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/services/lkgr_finder/lkgr_lib.py", line 117, in sort
    self._cache(*map(keyfunc, revisions))
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/services/lkgr_finder/lkgr_lib.py", line 91, in _cache
    positions = self._git.number(*unknown_revs)
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/libs/git.py", line 86, in number
    out = self(*cmd)
  File "/b/c/b/WebRTC_lkgr_finder/infra/infra/libs/git.py", line 51, in __call__
    cmd, stderr=subprocess.STDOUT, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['git', 'show', '-s', '--format=%B', 'f05cf6a09c983e3e7fc054d8199d82663f55733a']' returned non-zero exit status 128
step returned non-zero exit code: 1


Note that all commit hashes referenced in that log are WebRTC commit hashes, however somehow Chromium's commit hash sneaked in here (the one that causes the error): crrev.com/f05cf6a09c983e3e7fc054d8199d82663f55733a

ehmaldonado@ hypothesized that it's somehow pulling information for chromium.linux's "Linux Builder (dbg)" instead of chromium.webrtc.fyi's "Linux Builder (dbg)"
 
Oh, and this blocks rolling WebRTC into Chromium (though of course we could manually update LKGR or manually roll)
Owner: hinoka@chromium.org
Status: Assigned (was: Untriaged)
Timestamp is suspiciously close to when we flipped chromium.linux/Linux Builder (dbg).  Will investigate.
Cc: no...@chromium.org
Indeed, when querying for client.webrtc.fyi/Linux Builder (dbg), the first build that comes up is this one:

https://ci.chromium.org/p/chromium/builds/b8955028560300319664

And it's even labeled as a chromium.webrtc.fyi bot:
>>> j['Master']
u'chromium.webrtc.fyi'

I think there's a bug in the emulation API somewhere.

Comment 4 Deleted

It's probably not related but I accidentally set the repo for the chromium.webrtc.fyi console to chromium/src initially and fixed it later.

https://chromium.googlesource.com/chromium/src/+/6cf641746ca44de1ce3dd4313f3aefe765f01292
Project Member

Comment 7 by bugdroid1@chromium.org, Feb 9 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/56f0e72122f87b03674c27510f05083493ef959f

commit 56f0e72122f87b03674c27510f05083493ef959f
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Feb 09 23:41:17 2018

Talked to nodir, looks like it's a config error.  chromium.webrtc.fyi was mistakenly mapped to luci.chromium.ci (implying that the chromium team care about it) instead of luci.webrtc.ci (implying that the webrtc team cares about it).

The fix is to remap chromium.webrtc.fyi to luci.webrtc.ci.

We understand that this means the "Linux Builder (dbg)" would show up in luci.webrtc.ci, and the name might be confusing.  If it is, you can request a new bucket or just rename it to something else ("Linux Chromium Builder (dbg)"?)

Once the config fix is propagated, we'll reset memcache and this should be fixed.

Comment 9 Deleted

Thanks.
I'm leaving for now.

You can check if the fix works by just looking at https://build.chromium.org/p/client.webrtc.fyi/builders/WebRTC%20lkgr%20finder
and then the issue can be closed.
Project Member

Comment 11 by bugdroid1@chromium.org, Feb 10 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/56fcd5cfc4e76777642b5f1445302e143326b4f0

commit 56fcd5cfc4e76777642b5f1445302e143326b4f0
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Feb 10 00:27:50 2018

[milo] log and swallow 403 from buildbucket

If a LUCI bucket is not configured, API emulation currently returns 404, even
if there are Buildbot builds. Change this behavior: if buildbucket returned 403
for a build search, treat it as "no builds".

Bug:  810943 
Change-Id: I73f79dd87303d03691000361d0b96df354b500b9
Reviewed-on: https://chromium-review.googlesource.com/912318
Reviewed-by: Ryan Tseng <hinoka@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/56fcd5cfc4e76777642b5f1445302e143326b4f0/milo/buildsource/buildbot/buildstore/query.go

Deployed and it's no longer returning Not Found.  Lets see if that cycles green.
Status: Fixed (was: Assigned)
It's green now.  Thanks nodir@ for the quick fixes.

Comment 14 by efoo@chromium.org, Feb 28 2018

Labels: LUCI-Clients
Labels: LUCI-Mig-Client LUCI-Clients-Manual

Sign in to add a comment