Separate out result streams from AFE vs skylab_swarming tasks |
|||||
Issue descriptionThere are multiple result pipelines from the autotest tests: - TKO - To Bigquery (ci_results_archiver) - To Goldeneye - Sponge This bug tracks support for autotest jobs running in either AFE or skylab-swarming such that downstream results consumers are not impacted.
,
Apr 17 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/0c8e2e29dea9ad1096071bde628b40558b7134ab commit 0c8e2e29dea9ad1096071bde628b40558b7134ab Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 17 20:44:39 2018
,
Apr 19 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7776945890c678a2089cbc9ebbcf8aaa3ece66ed commit 7776945890c678a2089cbc9ebbcf8aaa3ece66ed Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Apr 19 01:27:01 2018 autotest: Stop reporting parsing time to metadata db. Nobody looks at metadata db anymore. We have better metrics and trace pipelines now. BUG= chromium:833988 TEST=skylab-drone task. Change-Id: I2bac9f9676e0a53867f2c291504496ff8bf3bfd5 Reviewed-on: https://chromium-review.googlesource.com/1015887 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/7776945890c678a2089cbc9ebbcf8aaa3ece66ed/tko/parse.py
,
Apr 19 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/11130e1f65e0240710cdb0b2a16ba37fec0aa0b5 commit 11130e1f65e0240710cdb0b2a16ba37fec0aa0b5 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Apr 19 01:27:02 2018 autotest: Inline get_afe_job_id_and_hostname All external callers of this function have been deleted. BUG= chromium:833988 TEST=None Change-Id: Ifbea117e68d38f4db00fbd0bb1a9f105cd76a5de Reviewed-on: https://chromium-review.googlesource.com/1015888 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/11130e1f65e0240710cdb0b2a16ba37fec0aa0b5/tko/utils.py
,
Apr 19 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ebd6c24303ce610671547ec3189c3e830143fce9 commit ebd6c24303ce610671547ec3189c3e830143fce9 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Apr 19 01:27:02 2018 autotest: Default afe_job_id to None when missing In the TKO database, tko_jobs.afe_job_id is a nullable integer column. Defaulting to '' for the column is wrong. BUG= chromium:833988 TEST=Insert a tko entry from skylab_swarming job (where afe_job_id is None) Change-Id: I1b29f3c577acaec1a0db11a0d05f6a86c186d926 Reviewed-on: https://chromium-review.googlesource.com/1015889 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/ebd6c24303ce610671547ec3189c3e830143fce9/tko/utils.py
,
Apr 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/83b0f488e55cbc065f458d0aa5459c24e44a329f commit 83b0f488e55cbc065f458d0aa5459c24e44a329f Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Apr 20 18:53:24 2018 autotest: Add table tko_task_references This is a normalized replacement for (tko_jobs.afe_job_id, tko_jobs.afe_parent_job_id) columns. Now that jobs/tasks can run on either autotest infrastructure (with afe_* references) as well as skylab, this new table allows tko_jobs to separately reference either of these task scheduling infrastructures' task id. BUG= chromium:833988 TEST=manual. Change-Id: Ib8c634a636612508c1034fcb4250f446cbcc91e4 Reviewed-on: https://chromium-review.googlesource.com/1020450 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/83b0f488e55cbc065f458d0aa5459c24e44a329f/frontend/migrations/128_add_tko_task_references.py
,
Apr 20 2018
Migrated staging lab:
chromeos-test@chromeos-staging-master2:/usr/local/autotest$ database/migrate.py sync --debug
Migration starting for database chromeos_autotest_db
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
Applying migration 128_add_tko_task_references up
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
Executing CREATE TABLE tko_task_references (
id integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
reference_type enum('skylab', 'afe') NOT NULL,
tko_job_idx int(10) unsigned NOT NULL,
task_id varchar(20) DEFAULT NULL,
parent_task_id varchar(20) DEFAULT NULL,
CONSTRAINT tko_task_references_ibfk_1 FOREIGN KEY (tko_job_idx) REFERENCES tko_jobs (job_idx) ON DELETE CASCADE,
KEY reference_type_id (reference_type, id)
) ENGINE=InnoDB, ()
Executing UPDATE migrate_info SET version=%s, (128,)
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
At version 128
Migration complete
Now wait for a successful staging test before migrating prod tko (and also the new V2 tko?)
,
Apr 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3b7b3a3f23836358f24f6ac7f488fc11703ae752 commit 3b7b3a3f23836358f24f6ac7f488fc11703ae752 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Apr 20 19:09:20 2018 autotest: Create new indices on tko_task_references. BUG= chromium:833988 TEST=manual migration. Change-Id: Ib612f0b9b068d0720e5be715ab226aaf14d49e65 Reviewed-on: https://chromium-review.googlesource.com/1022450 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/3b7b3a3f23836358f24f6ac7f488fc11703ae752/frontend/migrations/129_create_indices_tko_task_references.py
,
Apr 20 2018
chromeos-test@chromeos-staging-master2:/usr/local/autotest$ database/migrate.py sync --debug Migration starting for database chromeos_autotest_db Executing SELECT * FROM migrate_info, () Executing SELECT * FROM migrate_info, () Applying migration 129_create_indices_tko_task_references up Executing SELECT * FROM migrate_info, () Executing SELECT * FROM migrate_info, () Executing CREATE INDEX reference_type_task_id ON tko_task_references (reference_type, task_id), () Executing CREATE INDEX reference_type_parent_task_id ON tko_task_references (reference_type, parent_task_id), () Executing UPDATE migrate_info SET version=%s, (129,) Executing SELECT * FROM migrate_info, () Executing SELECT * FROM migrate_info, () At version 129 Migration complete
,
Apr 23 2018
staging lab has been happy with the migration (why wouldn't it?): https://viceroy.corp.google.com/chromeos/lab_staging#_VG_f2SrYffS Migrating prod TKO db now.
,
Apr 23 2018
TKO CloudSQL instance's schema hasn't been migrated in a while. MySQL [chromeos_autotest_db]> select * from migrate_info \G *************************** 1. row *************************** version: 119 The latest added by my CLs above is 128
,
Apr 23 2018
I can't apply migrations 120-127 tko the CloudSQL instance for TKO because we've manually dropped the afe_* tables from this database. I've verified that all those are afe_* table updates. I'm going to hack my way forward by setting the information schema to 127 and then running the migration from that point forward (what I need is 128, 129). This is the "right thing to do" given the current situation -- since there are no afe_* tables, skip updating them.
,
Apr 23 2018
prod TKO instance migrated after the hack in #12.
pprabhu@pprabhu:files$ ./database/migrate.py sync --debug | tee /tmp/tko_sql_migrations.log
Migration starting for database chromeos_autotest_db
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
Applying migration 128_add_tko_task_references up
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
Executing CREATE TABLE tko_task_references (
id integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
reference_type enum('skylab', 'afe') NOT NULL,
tko_job_idx int(10) unsigned NOT NULL,
task_id varchar(20) DEFAULT NULL,
parent_task_id varchar(20) DEFAULT NULL,
CONSTRAINT tko_task_references_ibfk_1 FOREIGN KEY (tko_job_idx) REFERENCES tko_jobs (job_idx) ON DELETE CASCADE,
KEY reference_type_id (reference_type, id)
) ENGINE=InnoDB, ()
Executing UPDATE migrate_info SET version=%s, (128,)
Applying migration 129_create_indices_tko_task_references up
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
Executing CREATE INDEX reference_type_task_id ON tko_task_references (reference_type, task_id), ()
Executing CREATE INDEX reference_type_parent_task_id ON tko_task_references (reference_type, parent_task_id), ()
Executing UPDATE migrate_info SET version=%s, (129,)
Executing SELECT * FROM migrate_info, ()
Executing SELECT * FROM migrate_info, ()
At version 129
Migration complete
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/432648c3033d2b53d1c803dc188c0ccefdd788ec commit 432648c3033d2b53d1c803dc188c0ccefdd788ec Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:16 2018 autotest: Mark internal tko/db functions private. BUG= chromium:833988 TEST=None Change-Id: I373626ff1f1f567f0608edab7cac4e74f6675329 Reviewed-on: https://chromium-review.googlesource.com/1018539 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/432648c3033d2b53d1c803dc188c0ccefdd788ec/tko/db.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ec96d6eaf493d2f5982e501e4e46425eeb3f5ccc commit ec96d6eaf493d2f5982e501e4e46425eeb3f5ccc Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:17 2018 autotest: Refactor out tko/db.insert_or_update_machine. BUG= chromium:833988 TEST=None Change-Id: I8cb75e8c000f2ff708392ade262dd26c57652513 Reviewed-on: https://chromium-review.googlesource.com/1018580 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/ec96d6eaf493d2f5982e501e4e46425eeb3f5ccc/tko/db.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/34103fc52d24f81f6bc3d4554fbfc3915e3feccd commit 34103fc52d24f81f6bc3d4554fbfc3915e3feccd Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:17 2018 autotest: Hoist tko_jobs related updates out of insert_job BUG= chromium:833988 TEST=None Change-Id: I63cc1e279d9ad47457c940b330a8a6141f634b3e Reviewed-on: https://chromium-review.googlesource.com/1018581 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/34103fc52d24f81f6bc3d4554fbfc3915e3feccd/tko/db.py [modify] https://crrev.com/34103fc52d24f81f6bc3d4554fbfc3915e3feccd/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6d4d8b6556a8d292d8f443d220c619107866c638 commit 6d4d8b6556a8d292d8f443d220c619107866c638 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:18 2018 autotest: Stop returning duplicate data from tko/db.insert_job BUG= chromium:833988 TEST=None Change-Id: Iab07cb3cd1b5a42c316af8ea1e526e5dce232c62 Reviewed-on: https://chromium-review.googlesource.com/1018582 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/6d4d8b6556a8d292d8f443d220c619107866c638/tko/db.py [modify] https://crrev.com/6d4d8b6556a8d292d8f443d220c619107866c638/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8957a3404f5452c16ce648f98abf4aeed3cc9a4f commit 8957a3404f5452c16ce648f98abf4aeed3cc9a4f Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:19 2018 autotest: Refactor out _write_job_to_db BUG= chromium:833988 TEST=None Change-Id: Ifc7188cd6b28beb7a5af41d1db5fa50ddc43f604 Reviewed-on: https://chromium-review.googlesource.com/1018583 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/8957a3404f5452c16ce648f98abf4aeed3cc9a4f/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e59b08bafba90ff2151cb69116ad092d2f95486d commit e59b08bafba90ff2151cb69116ad092d2f95486d Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:20 2018 autotest: Delete code trying to use a dead site_export function BUG= chromium:833988 TEST=None Change-Id: I69a8e0b19fea7bf7b10fc774b497506d90445692 Reviewed-on: https://chromium-review.googlesource.com/1018584 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/e59b08bafba90ff2151cb69116ad092d2f95486d/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/42a2bb4fbc4c52a147f56a71041769d45d504634 commit 42a2bb4fbc4c52a147f56a71041769d45d504634 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:20 2018 autotest: Extract _find_status_log_path BUG= chromium:833988 TEST=None Change-Id: I664aea28fe178f317998b09c76a91817b72e0de5 Reviewed-on: https://chromium-review.googlesource.com/1018585 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/42a2bb4fbc4c52a147f56a71041769d45d504634/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e06c49bf950ffe699518e1a6f9c0d8e2aa975a11 commit e06c49bf950ffe699518e1a6f9c0d8e2aa975a11 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 00:51:21 2018 autotest: Extract _parse_status_log BUG= chromium:833988 TEST=None Change-Id: I2294017636fe23c467bca047f59f8405acb1b8ff Reviewed-on: https://chromium-review.googlesource.com/1018586 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/e06c49bf950ffe699518e1a6f9c0d8e2aa975a11/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/edac1eeb962ff7a719404cbd1b5dcb861f718678 commit edac1eeb962ff7a719404cbd1b5dcb861f718678 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 03:51:16 2018 auotest: Extract _match_existing_tests BUG= chromium:833988 TEST=None Change-Id: Id60d722823229d2508cc926fbb39c72642fe2c26 Reviewed-on: https://chromium-review.googlesource.com/1018587 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/edac1eeb962ff7a719404cbd1b5dcb861f718678/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/bb69f7d904872dec0530ef9f1702835f4d06bbe2 commit bb69f7d904872dec0530ef9f1702835f4d06bbe2 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 18:11:27 2018 autotest: Also ignore TypeError when discovering own job_id for a suite CL:1015889 changed the default value returned for a job_id to None. dynamic_suite calls this to find out its own job_id. Haven't seen this fail in prod, but have some (flaky?) failures on moblab. BUG= chromium:833988 TEST=None Change-Id: I5d62b9723e247a002515bcc8c3232b3ffe40152b Reviewed-on: https://chromium-review.googlesource.com/1024591 Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Trybot-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/bb69f7d904872dec0530ef9f1702835f4d06bbe2/server/cros/dynamic_suite/dynamic_suite.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/30dee86472c72a86855464dd6460853c837aa6a9 commit 30dee86472c72a86855464dd6460853c837aa6a9 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 21:53:21 2018 autotest: Hoist afe_{parent,}_job_id update out of tko/db.py BUG= chromium:833988 TEST=None Change-Id: I6109ea873320770738707823f35d51855430ddab Reviewed-on: https://chromium-review.googlesource.com/1018720 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/30dee86472c72a86855464dd6460853c837aa6a9/tko/db.py [modify] https://crrev.com/30dee86472c72a86855464dd6460853c837aa6a9/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fc06c24f3aa20edb053b81f7aad0e1ebd723a832 commit fc06c24f3aa20edb053b81f7aad0e1ebd723a832 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 21:53:22 2018 autotest: Initialize tko.models.job.index attribute BUG= chromium:833988 TEST=None Change-Id: I63cb1a8530dbfb3be0229ba65ad3b0982cf3e22a Reviewed-on: https://chromium-review.googlesource.com/1018722 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/fc06c24f3aa20edb053b81f7aad0e1ebd723a832/tko/models.py [modify] https://crrev.com/fc06c24f3aa20edb053b81f7aad0e1ebd723a832/tko/db.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f5030d36e82ef630e4a142f2aa35a0fc498570d1 commit f5030d36e82ef630e4a142f2aa35a0fc498570d1 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 21:53:22 2018 autotest: Start populating tko_task_references. For autotest jobs, tko_jobs.afe_job_id and tko_jobs.afe_parent_job_id is now duplicated in tko_task_references table. Users should be migrated to use the tko_task_references table instead of the columns in tko_jobs table. BUG= chromium:833988 TEST=None Change-Id: Id0a4bbb8c84cee3dbbbdaadfec3aa1226e58dae6 Reviewed-on: https://chromium-review.googlesource.com/1020459 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/f5030d36e82ef630e4a142f2aa35a0fc498570d1/tko/models.py [modify] https://crrev.com/f5030d36e82ef630e4a142f2aa35a0fc498570d1/tko/db.py [modify] https://crrev.com/f5030d36e82ef630e4a142f2aa35a0fc498570d1/tko/parse.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/17905886254b3a4a7dd28d2b4e144ccbbc309315 commit 17905886254b3a4a7dd28d2b4e144ccbbc309315 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 23:38:29 2018 autotest: Insert skylab tasks into tko_skylab_tasks table BUG= chromium:833988 TEST=None Change-Id: I37efa217827612d36984b0e25d3b9837e7c32bcb Reviewed-on: https://chromium-review.googlesource.com/1018723 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/17905886254b3a4a7dd28d2b4e144ccbbc309315/tko/db.py [modify] https://crrev.com/17905886254b3a4a7dd28d2b4e144ccbbc309315/tko/parse.py [modify] https://crrev.com/17905886254b3a4a7dd28d2b4e144ccbbc309315/tko/utils.py
,
Apr 24 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c2a8a6a2fd117609112221eae02f87dcd29855cb commit c2a8a6a2fd117609112221eae02f87dcd29855cb Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Tue Apr 24 23:38:29 2018 autotest: Remove duplicate tko.models.job.index attribute Both job.job_idx and job.index were being used to store the same information (tko_jobs.job_idx column). We now consistently use job.job_idx BUG= chromium:833988 TEST=None Change-Id: I2bdf1b41dc5c55db0f4f59386b8407952c1db087 Reviewed-on: https://chromium-review.googlesource.com/1020500 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/c2a8a6a2fd117609112221eae02f87dcd29855cb/tko/models.py [modify] https://crrev.com/c2a8a6a2fd117609112221eae02f87dcd29855cb/tko/db.py [modify] https://crrev.com/c2a8a6a2fd117609112221eae02f87dcd29855cb/tko/parse.py
,
Apr 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e0b70c6b8b6c3ac8c47b947f6b36063c821058c8 commit e0b70c6b8b6c3ac8c47b947f6b36063c821058c8 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Apr 26 06:00:09 2018 autotest: Deprecate autoserv argument --parse_job Nobody has used this argument for over 8 years. This argument adds a dependence from autoserv to tko DB, which is wrong. BUG= chromium:833988 TEST=moblab-vm-pre-cq Change-Id: Ib3c263bbda986b0620666398e7363f96064ab8bf Reviewed-on: https://chromium-review.googlesource.com/1022333 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/e0b70c6b8b6c3ac8c47b947f6b36063c821058c8/server/server_job.py [modify] https://crrev.com/e0b70c6b8b6c3ac8c47b947f6b36063c821058c8/server/autoserv_parser.py [modify] https://crrev.com/e0b70c6b8b6c3ac8c47b947f6b36063c821058c8/server/autoserv
,
Apr 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6f1d4786165096ee3a23f551cb49497479e22267 commit 6f1d4786165096ee3a23f551cb49497479e22267 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Apr 26 06:00:10 2018 autotest: Drop support for continuous parsing of results from server_job This feature has not been used for over 8 years and adds a dependency from autoserv to TKO DB, which is bad. Kill it. BUG= chromium:833988 TEST=Test on skylab-drone; moblab-pre-cq Change-Id: I4379a4bbf21d2ecf6af7da24913047c76052e7b2 Reviewed-on: https://chromium-review.googlesource.com/1022334 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/6f1d4786165096ee3a23f551cb49497479e22267/server/server_job.py [modify] https://crrev.com/6f1d4786165096ee3a23f551cb49497479e22267/server/autoserv [modify] https://crrev.com/6f1d4786165096ee3a23f551cb49497479e22267/global_config.ini
,
Apr 26 2018
Verified that staging lab TKO is getting the new tko_task_references populated correctly. I'll have to backfill data for existing tko_jobs so downstream users can cleanly migrate to using that. https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1030957 I've also backfilled the staging lab TKO table using this script.
,
Apr 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/13b26ccb4a1bb4fa0beb383e5a38c1996f31e110 commit 13b26ccb4a1bb4fa0beb383e5a38c1996f31e110 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Apr 26 23:06:03 2018 contrib: Add a script to backfill tko_task_references table. BUG= chromium:833988 TEST=Used script to backfill staging lab TKO. Change-Id: Iaa55affbf6087be4a60976b163f9ffec7c062340 Reviewed-on: https://chromium-review.googlesource.com/1030957 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [add] https://crrev.com/13b26ccb4a1bb4fa0beb383e5a38c1996f31e110/contrib/__init__.py [add] https://crrev.com/13b26ccb4a1bb4fa0beb383e5a38c1996f31e110/contrib/backfill_tko_task_references.py
,
Apr 26 2018
I'm backfilling prod TKO's tko_task_references table now.
,
Apr 27 2018
Tracking bug to update bigquery pipeline + stainless: b/78659566 Tracking bug to update goldeneye: b/78660465
,
Apr 27 2018
Backfilling prod TKO is done. It added ~11 GB to storage usage. (2.9% increase) Marking this bug fixed as the TKO instance bits are done and tracking bugs for downstream users have been created.
,
Apr 27 2018
TKO slave had trouble with the backfilling operation: issue 837682
,
Apr 30 2018
New TKO ro slave is all caught up.
,
May 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/21aff987ec2a3a9488ba941723929cbf3bae57ee commit 21aff987ec2a3a9488ba941723929cbf3bae57ee Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu May 10 05:33:10 2018 Patch BigQuery table schema before appending data. This allows appending data to existing tables with new columns in the schema. Our bigquery client library is too old to support the ALLOW_FIELD_ADDITION option, so we explicitly patch the table schema before loading data. BUG= chromium:833988 TEST=On the dev instance, upload partial data with old schema, then load data with extra column. Change-Id: Ib60e3a340769f8bbaafe837be562de2568d4ba30 [modify] https://crrev.com/21aff987ec2a3a9488ba941723929cbf3bae57ee/ci_results_archiver/utils/bigquery_wrapper.py [modify] https://crrev.com/21aff987ec2a3a9488ba941723929cbf3bae57ee/ci_results_archiver/utils/bigquery_wrapper_unittest.py
,
May 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88 commit a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu May 10 05:33:16 2018 Populate task references from tko_task_references table. tko_task_references is a new table in TKO database that replaces the tko_jobs.afe_job_id and tko_jobs.afe_parent_job_id columns. This column starts joining the new table information into tko jobs bigquery table. A separate one off job will be used to backfill the existing rows. BUG= chromium:833988 TEST=unittests, manual --one-shot update to the dev bigquery tables. Change-Id: I0c365a4704407bc4b5cf55fa9b9588f2c6053ca6 [modify] https://crrev.com/a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88/ci_results_archiver/importers/tko_job_importer_unittest.py [modify] https://crrev.com/a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88/ci_results_archiver/utils/test/fake_tko_connection.py [modify] https://crrev.com/a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88/ci_results_archiver/importers/tko_job_importer.py [modify] https://crrev.com/a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88/ci_results_archiver/utils/tko_connection.py [modify] https://crrev.com/a7d0f1ccd76e05cf0b65090a8f3a62f5bd7b1c88/ci_results_archiver/table_specs.py
,
May 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/ci_results_archiver/+/365f91f48ce305682853c01cc9f63ff7a310732b commit 365f91f48ce305682853c01cc9f63ff7a310732b Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu May 10 05:33:16 2018 Improve debugging messages. BUG= chromium:833988 TEST=None Change-Id: I12fd5754459865c62715e536410688c6e4ea8414 [modify] https://crrev.com/365f91f48ce305682853c01cc9f63ff7a310732b/ci_results_archiver/utils/mysql_wrapper_unittest.py [modify] https://crrev.com/365f91f48ce305682853c01cc9f63ff7a310732b/ci_results_archiver/utils/mysql_wrapper.py [modify] https://crrev.com/365f91f48ce305682853c01cc9f63ff7a310732b/ci_results_archiver/importers/tko_job_importer.py |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by pprabhu@chromium.org
, Apr 17 2018