Standardizing artifacts in JSON test results format for chromium testing |
|||||||||||||||||
Issue descriptionWe are planning on adding artifacts to the JSON test results to pass debugging info (screenshots, stack traces, logs, etc) back to sheriff-o-matic for surfacing to users. Artifacts will be collected during tests, uploaded to cloud storage during the swarming collect process, and then passed through to SOM in the test results. Although we are primarily doing this to our solve log consumption problem when perf moves to one buildbot step per isolate, we believe it has value to other use cases in perf so we are standardizing the artifact format so it can be re-used. See the design doc for the exact contract, but there will be a set of artifacts per run and each artifact in the set corresponds to a piece of debugging information that the test decides to store. https://docs.google.com/document/d/1SqBbeiRRemt-LS0SUbOODDY-xZ6kkczrK8w9WGoivW0/edit?ts=59d3b8e8#heading=h.nzni6eo73p4s
,
Oct 6 2017
,
Oct 6 2017
,
Oct 6 2017
,
Oct 6 2017
,
Oct 12 2017
Stephen will drive this project in Q4.
,
Oct 20 2017
,
Oct 20 2017
,
Nov 10 2017
,
Nov 14 2017
,
Nov 15 2017
An update; https://chromium-review.googlesource.com/c/chromium/src/+/729212 landed (and I forgot to set the bug id on that :( ), which adds the documentation. I'm currently working on https://chromium-review.googlesource.com/c/chromium/src/+/749624/10, which is an upload script to upload artifacts to cloud storage. Once this lands, two more things need to happen: 1. Test runners need to output artifacts. https://chromium-review.googlesource.com/c/catapult/+/708048 is a CL for catapult I've been making 2. We need to plug this into the recipes somehow. This is the most TBD about all the work to do.
,
Nov 15 2017
,
Dec 2 2017
,
Dec 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3fc597bc88491b6dfb19bcfc8dc2a42f02d16b3e commit 3fc597bc88491b6dfb19bcfc8dc2a42f02d16b3e Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Dec 06 06:45:14 2017 Add new test result artifact upload script This script is intended to be run on buildbots, after they've collected the results of a swarming execution. The script takes a single test results json file, and uploads any artifacts in the file to cloud storage, to a public bucket (support for private buckets will be added later). Bug: 772208 Change-Id: Ideca5912d2c8f64a62fafe21f6537f59250c62f3 Reviewed-on: https://chromium-review.googlesource.com/749624 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Cr-Commit-Position: refs/heads/master@{#522029} [modify] https://crrev.com/3fc597bc88491b6dfb19bcfc8dc2a42f02d16b3e/build/android/pylib/utils/google_storage_helper.py [add] https://crrev.com/3fc597bc88491b6dfb19bcfc8dc2a42f02d16b3e/testing/buildbot/scripts/upload_test_result_artifacts.py [add] https://crrev.com/3fc597bc88491b6dfb19bcfc8dc2a42f02d16b3e/testing/buildbot/scripts/upload_test_result_artifacts_unittest.py
,
Dec 13 2017
,
Jan 8 2018
This is stalled on eyaich@'s work.
,
Apr 3 2018
Originally the plan was to upload these on the recipe side, most likely in collect_step: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/resources/collect_task.py?q=collect_task&sq=package:chromium&l=121 For now we have decided that since perf is the only current user of artifacts we are going to rely on the generator of the json test results format to upload these to cloud storage instead of plugging it into recipes. Especially with the shamwow work going on we think this is the best design for the end state.
,
Apr 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/23d04ac91acd96d02be00dca03310a22daf8baeb commit 23d04ac91acd96d02be00dca03310a22daf8baeb Author: Emily Hanley <eyaich@google.com> Date: Wed Apr 04 13:38:21 2018 Updating artifact paths to cloud storage paths. Bug: chromium:772208 Change-Id: Iab0c4f8a4cb9420a9926ebc66a164a5d338abdee Reviewed-on: https://chromium-review.googlesource.com/992864 Commit-Queue: Emily Hanley <eyaich@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> [modify] https://crrev.com/23d04ac91acd96d02be00dca03310a22daf8baeb/telemetry/telemetry/internal/results/page_test_results.py [modify] https://crrev.com/23d04ac91acd96d02be00dca03310a22daf8baeb/telemetry/telemetry/internal/results/page_test_results_unittest.py
,
Apr 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f7bc768ebe208f5daf5541bbe5bdc0d8c3e437a5 commit f7bc768ebe208f5daf5541bbe5bdc0d8c3e437a5 Author: catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Wed Apr 04 15:02:05 2018 Roll src/third_party/catapult/ e13394ffe..23d04ac91 (1 commit) https://chromium.googlesource.com/catapult.git/+log/e13394ffef36..23d04ac91acd $ git log e13394ffe..23d04ac91 --date=short --no-merges --format='%ad %ae %s' 2018-04-04 eyaich Updating artifact paths to cloud storage paths. Created with: roll-dep src/third_party/catapult BUG= chromium:772208 The AutoRoll server is located here: https://catapult-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. TBR=sullivan@chromium.org Change-Id: I1c69b0313e48b5156a93b78e3e562cf4ecddbeec Reviewed-on: https://chromium-review.googlesource.com/995579 Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#548058} [modify] https://crrev.com/f7bc768ebe208f5daf5541bbe5bdc0d8c3e437a5/DEPS
,
Jan 7
|
|||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||
Comment 1 by nedngu...@google.com
, Oct 6 2017Components: Speed>Benchmarks>Waterfall