New issue
Advanced search Search tips

Issue 855169 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: ----



Sign in to add a comment

TKO jobs are overwritten each other.

Project Member Reported by xixuan@chromium.org, Jun 21 2018

Issue description

E.g. https://uberchromegw.corp.google.com/i/chromeos/builders/nyan_blaze-paladin/builds/180

The child tests in tko are only 5:

MySQL [chromeos_autotest_db]> select t.test, j.tag, t.started_time from tko_tests as t, tko_jobs as j where t.started_time > '2018-06-21 08:50:00' and t.finished_time < '2018-06-21 10:10:00' and t.test != 'CLIENT_JOB.0' and t.test != 'SERVER_JOB' and t.job_idx=j.job_idx; 
+------------------------------+-----------------------------------------------------------------+---------------------+
| test                         | tag                                                             | started_time        |
+------------------------------+-----------------------------------------------------------------+---------------------+
| provision                    | chromeos4-row7-rack8-host7/provision_chromeos4-row7-rack8-host7 | 2018-06-21 08:54:11 |
| login_OwnershipApi           | chromeos4-row7-rack8-host5/autoserv_test                        | 2018-06-21 09:03:06 |
| graphics_Gbm                 | chromeos4-row7-rack7-host9/autoserv_test                        | 2018-06-21 09:10:32 |
| login_GuestAndActualSession  | chromeos4-row7-rack8-host9/autoserv_test                        | 2018-06-21 09:11:49 |
| login_RetrieveActiveSessions | chromeos4-row7-rack7-host7/autoserv_test                        | 2018-06-21 09:11:58 |
| login_OwnershipApi           | chromeos4-row7-rack7-host21/autoserv_test                       | 2018-06-21 10:03:44 |
+------------------------------+-----------------------------------------------------------------+---------------------+

My guess, https://chrome-swarming.appspot.com/task?id=3e3bdde3aa4ad210&refresh=10&show_raw=1 is not in tko, it's overwritten by https://chrome-swarming.appspot.com/task?id=3e3bde7172a69410&refresh=10&show_raw=1.


 
I see one reference to skylab_task_id in the tko/parser (as an alternative to afe_job_id), but it doesn't look like it actually gets used.  Was that intended to be added as a new column in the TKO DB?

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/be695382c7c5eb7a15ba388c1d104789b99f4084/tko/parse.py#349

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/be695382c7c5eb7a15ba388c1d104789b99f4084/tko/db.py#445
I'm not sure if it is causing this bug yet, but we will eventually need to fix this problem: the build/suite/board information in the TKO is parsed directly from an Autotest format job label.

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/be695382c7c5eb7a15ba388c1d104789b99f4084/tko/parse.py#357

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/be695382c7c5eb7a15ba388c1d104789b99f4084/server/site_utils.py#626
Oh, I think older jobs are being deleted by this code

https://chromium.googlesource.com/chromiumos/third_party/autotest/+/be695382c7c5eb7a15ba388c1d104789b99f4084/tko/parse.py#343

It matches by tko_jobs.tag, is that unique per job or per bot?  swarming-3e2e66eaabd64011/chromeos4-row7-rack7-host21
Cc: -ayatane@chromium.org pprabhu@chromium.org
Owner: ayatane@chromium.org
Status: Started (was: Untriaged)
Oh yeah, it is using chromeos4-row7-rack7-host7/autoserv_test as the unique jobname...
Oh, TKO parse uses the parts of the path to determine the job name

So when we added the autoserv_test directory, that pushed the swarming task id off the job name...
Project Member

Comment 6 by bugdroid1@chromium.org, Jun 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/infra/lucifer/+/b503d9036292e6306a76865b6051bb3b9b99d432

commit b503d9036292e6306a76865b6051bb3b9b99d432
Author: Allen Li <ayatane@google.com>
Date: Fri Jun 22 19:44:24 2018

Use increased level for tko/parse for Skylab

tko/parse uses an explicit level to chop off parts of the results
directory path to use as a unique job name.  The path for Skylab got
one deeper to accommodate provisioning and resulted in the job name
being non-unique.

BUG= chromium:855169 
TEST=None

Change-Id: Ia67b7f7ac73b5591fbaaf6272a10b7d13419fad3
Reviewed-on: https://chromium-review.googlesource.com/1111202
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/b503d9036292e6306a76865b6051bb3b9b99d432/src/lucifer/cmd/lucifer_run_job/autotest.go
[modify] https://crrev.com/b503d9036292e6306a76865b6051bb3b9b99d432/src/lucifer/cmd/lucifer_run_job/flags.go
[modify] https://crrev.com/b503d9036292e6306a76865b6051bb3b9b99d432/src/lucifer/autotest/autotest.go
[modify] https://crrev.com/b503d9036292e6306a76865b6051bb3b9b99d432/src/lucifer/cmd/lucifer_run_job/main.go

Comment 7 by xixuan@chromium.org, Jun 29 2018

Status: Verified (was: Started)

Sign in to add a comment