New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 690167 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Last visit > 30 days ago
Closed: May 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

teach export_to_gcloud to assume parent build ID

Project Member Reported by shuqianz@chromium.org, Feb 8 2017

Issue description

Teach export_to_gcloud to assume parent build ID, then it can upload the suite timeline report and any other metric filea to data store under the given build. Implementation detail:
1. Teach run_suite.py to accept a build ID arg, and it can write to database.
2. Teach tko/parse.py to get the build ID from database, and pass to export_to_gcloud to upload metric file to datastore
3. teach builder to pass the build ID to run_suite.py
 
Yep, this is the right track. 2 points to add:
 - No actual modification to export_to_gcloud. It already knows how to accept a parent key argument.
 - I'd call the argument to run_suite something more specific that "build ID". It might not be a build id in some cases. I'd call it "datastore_parent_key" or something like that.
Project Member

Comment 2 by bugdroid1@chromium.org, Feb 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/da1118db50fa006d1377a344262425126ddd9105

commit da1118db50fa006d1377a344262425126ddd9105
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Sat Feb 25 18:32:09 2017

[autotest] Teach afe rpcs to accept job_keyvals arg to pass general job keyvals

Add a job_keyval variable to afe rpcs, to pass a general dictionary of
keyvals to suite job. This dict of keyval can then be used by tko/parse
for any suite result operations or analysis.

BUG= chromium:690167 
TEST=unittest and test on a testing server.

Change-Id: I8a348fee601257345b71ff0a573b18d0f2b9550d
Reviewed-on: https://chromium-review.googlesource.com/441151
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Shuqian Zhao <shuqianz@chromium.org>

[modify] https://crrev.com/da1118db50fa006d1377a344262425126ddd9105/server/cros/dynamic_suite/suite.py
[modify] https://crrev.com/da1118db50fa006d1377a344262425126ddd9105/server/cros/dynamic_suite/suite_unittest.py
[modify] https://crrev.com/da1118db50fa006d1377a344262425126ddd9105/server/cros/dynamic_suite/dynamic_suite.py
[modify] https://crrev.com/da1118db50fa006d1377a344262425126ddd9105/frontend/afe/rpc_interface.py

Project Member

Comment 3 by bugdroid1@chromium.org, Mar 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/843ae5c72986bae1978bce7a91616539240faa03

commit 843ae5c72986bae1978bce7a91616539240faa03
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Sat Mar 04 01:11:33 2017

[autotest] teach run_suite.py to accept job_keyvals arg

job_keyvals is a dict of job keyvals to be inject into suite control
file, which can be written into the keyval file later. Then this dict
can be read by tko/parse to do any suite result analysis.

BUG= chromium:690167 
TEST=unittest and test on a testing server

Change-Id: I0bf5c3e517b9904b2644ea715d2c8194306d0005
Reviewed-on: https://chromium-review.googlesource.com/441247
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/843ae5c72986bae1978bce7a91616539240faa03/site_utils/run_suite.py

Project Member

Comment 4 by bugdroid1@chromium.org, Mar 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/637d22c759498c5c34c20f93f9f22a66363dc819

commit 637d22c759498c5c34c20f93f9f22a66363dc819
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Fri Mar 10 06:18:52 2017

[autotest] teach run_suite to convert string cmd line input to dict

Previously, we use json.loads to parse the string input and convert to
dictionary. However, it has many restrictions, like not supporting
tuple.

BUG= chromium:690167 
TEST=unittest and test in a testing server.

Change-Id: Ic31c863b36ff844adf7b7a5c189f51f3a39e236c
Reviewed-on: https://chromium-review.googlesource.com/450939
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/637d22c759498c5c34c20f93f9f22a66363dc819/site_utils/run_suite.py

Project Member

Comment 5 by bugdroid1@chromium.org, Mar 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/6fe3590f45a8384f3dd195c09b94c6e06b004ac8

commit 6fe3590f45a8384f3dd195c09b94c6e06b004ac8
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Tue Mar 14 22:06:02 2017

cbuildbot: Pass build_id to HWtest stage used as datastore_parent_key

Pass build_id to HWtest stage to be used as datastore_parent_key to
upload any file to gcloud datastore under a given build.

BUG= chromium:690167 
TEST=unittest and trybot

Change-Id: I3b27e1b1a1690f41069e703bedac780487ca5faf
Reviewed-on: https://chromium-review.googlesource.com/440410
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/6fe3590f45a8384f3dd195c09b94c6e06b004ac8/lib/constants.py
[modify] https://crrev.com/6fe3590f45a8384f3dd195c09b94c6e06b004ac8/cbuildbot/commands.py
[modify] https://crrev.com/6fe3590f45a8384f3dd195c09b94c6e06b004ac8/cbuildbot/stages/test_stages.py

Project Member

Comment 6 by bugdroid1@chromium.org, Mar 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a42bba1ecf7ddba729c37fe99bb8e9953bd099b6

commit a42bba1ecf7ddba729c37fe99bb8e9953bd099b6
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Sat Mar 25 02:38:21 2017

[autotest] teach tko parse to read datastore_parent_key from keyval file

Teach tko parse to read the datastore_parent_key from result keyval
file, and pass it to export_to_gcloud to upload suite timeline report to
gcloud under a given build.

BUG= chromium:690167 
TEST=Test on a testing server.

Change-Id: I5b82b7f2c433d7a0c8718ca61ba1f7c05957367d
Reviewed-on: https://chromium-review.googlesource.com/440408
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/a42bba1ecf7ddba729c37fe99bb8e9953bd099b6/tko/parse.py

Project Member

Comment 7 by bugdroid1@chromium.org, Mar 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a42bba1ecf7ddba729c37fe99bb8e9953bd099b6

commit a42bba1ecf7ddba729c37fe99bb8e9953bd099b6
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Sat Mar 25 02:38:21 2017

[autotest] teach tko parse to read datastore_parent_key from keyval file

Teach tko parse to read the datastore_parent_key from result keyval
file, and pass it to export_to_gcloud to upload suite timeline report to
gcloud under a given build.

BUG= chromium:690167 
TEST=Test on a testing server.

Change-Id: I5b82b7f2c433d7a0c8718ca61ba1f7c05957367d
Reviewed-on: https://chromium-review.googlesource.com/440408
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/a42bba1ecf7ddba729c37fe99bb8e9953bd099b6/tko/parse.py

Project Member

Comment 8 by bugdroid1@chromium.org, Mar 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/fe2e8017fa5fe87b48cb1cb5d6f67d973407b9ce

commit fe2e8017fa5fe87b48cb1cb5d6f67d973407b9ce
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Thu Mar 30 21:03:45 2017

cbuildbot: correct indention error for job_keyvals passed to run_suite

BUG= chromium:690167 
TEST=trybot

Change-Id: I418883024556abd488f2aade1a9f1b10808bab6c
Reviewed-on: https://chromium-review.googlesource.com/462379
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: David Riley <davidriley@chromium.org>

[modify] https://crrev.com/fe2e8017fa5fe87b48cb1cb5d6f67d973407b9ce/cbuildbot/commands.py

Project Member

Comment 9 by bugdroid1@chromium.org, Mar 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4ff74737faee35acf61f3aa90fdf5d034863a0be

commit 4ff74737faee35acf61f3aa90fdf5d034863a0be
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Fri Mar 31 04:21:58 2017

tko: fix the suite_timeline string format error

The datastore_parent_key read from keyval file is already a string,
which can be passed directly to export_to_gcloud script. No need to
parse it.

BUG= chromium:690167 
TEST=locally

Change-Id: I2e7931e3a1d1b1cb63988a798a93dd51f1ceaa99
Reviewed-on: https://chromium-review.googlesource.com/464088
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: David Riley <davidriley@chromium.org>

[modify] https://crrev.com/4ff74737faee35acf61f3aa90fdf5d034863a0be/tko/parse.py

Status: Fixed (was: Untriaged)
Labels: VerifyIn-61

Comment 12 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Sign in to add a comment