make gs_offloader work for skylab tasks |
||||||
Issue descriptiongs_offloader isn't offloading any skylab tasks right now: pprabhu@pprabhu:chromiumos$ gsutil ls gs://chromeos-autotest-results/swarming-* | head CommandException: One or more URLs matched no objects. FixIt.
,
May 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b3418184e8aa48e3e080ad4bbc60424e61e4a515 commit b3418184e8aa48e3e080ad4bbc60424e61e4a515 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 19:50:43 2018 [autotest] Persist autotest job_id as str in lxc.ContainerId This prepares the ContainerId to accept skylab ids, which are hexadecimal. BUG= chromium:842343 TEST=TBD Change-Id: Idf6b0449d90ac271f73579b4692ac0a32e8c07d8 Reviewed-on: https://chromium-review.googlesource.com/1056366 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/b3418184e8aa48e3e080ad4bbc60424e61e4a515/site_utils/lxc/container.py
,
May 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/9a631086d69352d3db2c0f012761a9f94b08d5fa commit 9a631086d69352d3db2c0f012761a9f94b08d5fa Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 19:50:43 2018 [autotest] Excise job_overhead and metadata_reporter Funny story: metadata_reporter is never start()ed, and even if it were, metadata_reporter._run() doesn't send anything to elasticsearch at all, and even if it did, there is no elasticsearch instance running anymore. Dead code. BUG= chromium:842343 TEST=None Change-Id: Ia9e3b57e879ee38ff42e99a71ed0f432be79872b Reviewed-on: https://chromium-review.googlesource.com/1056407 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/9a631086d69352d3db2c0f012761a9f94b08d5fa/server/autoserv [modify] https://crrev.com/9a631086d69352d3db2c0f012761a9f94b08d5fa/site_utils/run_suite.py [delete] https://crrev.com/b3418184e8aa48e3e080ad4bbc60424e61e4a515/site_utils/job_overhead.py [modify] https://crrev.com/9a631086d69352d3db2c0f012761a9f94b08d5fa/tko/parse.py [modify] https://crrev.com/9a631086d69352d3db2c0f012761a9f94b08d5fa/scheduler/host_scheduler.py [delete] https://crrev.com/b3418184e8aa48e3e080ad4bbc60424e61e4a515/site_utils/metadata_reporter.py
,
May 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5ad4d650124bac70e919a188564a9b9a30c18e8c commit 5ad4d650124bac70e919a188564a9b9a30c18e8c Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 19:50:44 2018 [autotest] Explicitly cast job_id to int in sequences. This prepares the way for job_directories.get_job_id_or_task_id() to return opaque str ids BUG= chromium:842343 TEST=None Change-Id: If899534674eb84739894d00c2b090274e5d46a51 Reviewed-on: https://chromium-review.googlesource.com/1056408 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/5ad4d650124bac70e919a188564a9b9a30c18e8c/server/sequence.py
,
May 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/28a4651f2f33c694f2d8bfd1540a5e46a1b8ad1e commit 28a4651f2f33c694f2d8bfd1540a5e46a1b8ad1e Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 19:50:45 2018 [autotest] Change job_directores.get_job_id_or_task_id return to str All callers have been updated to expect str instead of int. This allows us to report back autotest job or skylab task ids as opaque str. BUG= chromium:842343 TEST=None Change-Id: I83314286e5f3003e5726a425719f1782419082de Reviewed-on: https://chromium-review.googlesource.com/1056409 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/28a4651f2f33c694f2d8bfd1540a5e46a1b8ad1e/site_utils/job_directories.py [modify] https://crrev.com/28a4651f2f33c694f2d8bfd1540a5e46a1b8ad1e/site_utils/gs_offloader_unittest.py
,
May 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/fc35a20c45c8f9c76ed75044917c091125452edb commit fc35a20c45c8f9c76ed75044917c091125452edb Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed May 23 23:37:44 2018 skylab_swaming_work: Seal results directory when task completes A special timestamp file is now dropped in the results directory when skylab_swarming_worker finishes processing the task. gs_offloader will use this timestamp file to determine when the results folder is available to be offloaded. BUG= chromium:842343 TEST=Run task on skylab-drone. Change-Id: I3b3aab47bad063ae83cb674d6d7210d46266e614 Reviewed-on: https://chromium-review.googlesource.com/1070000 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/fc35a20c45c8f9c76ed75044917c091125452edb/src/lucifer/cmd/skylab_swarming_worker/main.go
,
Jun 26 2018
,
Jun 26 2018
Allen taking over to finish this before logdog logging.
,
Jun 26 2018
,
Jun 26 2018
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1069992 (2 CL stack) should make gs_offloader start processing skylab task result directories. The offload should still fail because there are no AFE jobs associated with these tasks.
,
Jun 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/infra/lucifer/+/0f28631831e73f3f437a5f244c08e209250884b5 commit 0f28631831e73f3f437a5f244c08e209250884b5 Author: Allen Li <ayatane@google.com> Date: Wed Jun 27 02:55:02 2018 skylab_swarming_worker: Seal root results dir e.g., seal .../results/swarming-3e4391423c3a4311/ instead of .../results/swarming-3e4391423c3a4311/chromeos6-row2-rack7-host19/ BUG= chromium:842343 TEST=None Change-Id: I7086ba2420f11d7e2e6d55243f6d3fb1c1d2ea5b Reviewed-on: https://chromium-review.googlesource.com/1116001 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/0f28631831e73f3f437a5f244c08e209250884b5/src/lucifer/cmd/skylab_swarming_worker/main.go
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a4557034451e0c211f2fde96b60d7c5c286c9c93 commit a4557034451e0c211f2fde96b60d7c5c286c9c93 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Jun 28 05:07:08 2018 [autotest] job_directories: Internalize some constants BUG= chromium:842343 TEST=unittests Change-Id: Ifcdb91d731d1ed9fb4b490e18c082b556ad80305 Reviewed-on: https://chromium-review.googlesource.com/1069990 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/a4557034451e0c211f2fde96b60d7c5c286c9c93/site_utils/job_directories.py [modify] https://crrev.com/a4557034451e0c211f2fde96b60d7c5c286c9c93/site_utils/gs_offloader_unittest.py
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f76a85c448a948f716a3dadee7f1b125bc0f16ea commit f76a85c448a948f716a3dadee7f1b125bc0f16ea Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Jun 28 05:07:09 2018 [autotest] job_directores: Recognize swarming task directories BUG= chromium:842343 TEST=None Change-Id: Ie800bbe10916dfb19c29484b1348f34a1551fc44 Reviewed-on: https://chromium-review.googlesource.com/1069992 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/f76a85c448a948f716a3dadee7f1b125bc0f16ea/site_utils/job_directories.py
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7402f09a2f6ac704a57ace9a662b562d69a49003 commit 7402f09a2f6ac704a57ace9a662b562d69a49003 Author: Allen Li <ayatane@chromium.org> Date: Thu Jun 28 23:47:36 2018 autotest: Add gs_offloader support for swarming BUG= chromium:842343 TEST=None CQ-DEPEND=CL:1116001 Change-Id: I205f1a4695332d4ee12fa22605c3fe66ea077065 Reviewed-on: https://chromium-review.googlesource.com/1115971 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/7402f09a2f6ac704a57ace9a662b562d69a49003/site_utils/gs_offloader_unittest.py [modify] https://crrev.com/7402f09a2f6ac704a57ace9a662b562d69a49003/site_utils/job_directories.py [modify] https://crrev.com/7402f09a2f6ac704a57ace9a662b562d69a49003/site_utils/gs_offloader.py
,
Jun 30 2018
,
Jul 2
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/41dbe2309a9b0066dba70215bde0d9265e0b6cea commit 41dbe2309a9b0066dba70215bde0d9265e0b6cea Author: Allen Li <ayatane@chromium.org> Date: Mon Jul 02 22:32:49 2018 autotest: Remove trailing slash for swarming offloads Im pretty sure this is causing double/nested uploads. https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/swarming-3e62b1753d9d3511/ BUG= chromium:842343 TEST=None Change-Id: Ia0fa99007dbe74e51e150c4a62b0d8836a552217 Reviewed-on: https://chromium-review.googlesource.com/1121543 Commit-Ready: Allen Li <ayatane@chromium.org> Tested-by: Allen Li <ayatane@chromium.org> Reviewed-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/41dbe2309a9b0066dba70215bde0d9265e0b6cea/site_utils/job_directories.py
,
Jul 16
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by bugdroid1@chromium.org
, May 23 2018