flow to get from build page to provision/repair logs is hopelessly complex |
|||||||
Issue descriptionCurrent flow I follow. (example build: https://luci-milo.appspot.com/buildbot/chromeos/kip-paladin/845) 1) Read the stdio for the hwtest stage, to determine which test failed to complete (this is not surfaced to waterfall). answer: video_ChromeCameraMJpegHWDecodeUsed 2) Back out of there, and follow link to suite answer: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=103483768 3) Click "next" on the child test list to scroll to next page. Click on video_ChromeCameraMJpegHWDecodeUsed which admits to having failed. answer: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=103483861 4) ctrl+click on Debug Logs link to load that in background tab. In the meantime, click on host so that can go to host page (ugh, can't even select the host name for copy/paste from this page, because this uses an onMouseUp handler) answer: chromeos4-row1-rack12-host9 5) Run dut-status against that host, to fish out provision and repair logs. answers: http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row1-rack12-host9/523162-provision/ http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos4-row1-rack12-host9/523245-repair/ 6) Examine those two logs to determine cause of provision failure. There must be some shortcuts that we can build to this.
,
Mar 1 2017
I wrote some scripts which can assist with that. There's one script which goes from build id -> list of suite job ids (via CIDB/milo), and then another script which takes a suite job and lists the failing provision jobs followed by the next job. There's quick and dirty draft versions up at: https://chromium-review.googlesource.com/c/447964/ https://chromium-review.googlesource.com/c/447865/ It's two scripts because I don't know if you can call autotest stuff from chromite, and you can't call into CIDB from autotest. There's another one which uses Don's classifier to take the logs and do some classification on it as well.
,
Mar 7 2017
what's the next step needed on this? Who's the best owner?
,
Mar 8 2017
davidriley / dgarret's script might be ready to share, and perhaps even integrate in to builder? jrbarnette provision rework should help address this
,
Mar 8 2017
It's up in varying stages of code reviews. There's a bit of complexity since it requires changes from both dgarrett and I across chromite and autotest. I don't think we can integrate to a builder at this time because there isn't access to AFE/TKO if I understand correctly.
,
Mar 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/681d4a498543a0b6eb07037cdb548a493cec8266 commit 681d4a498543a0b6eb07037cdb548a493cec8266 Author: David Riley <davidriley@chromium.org> Date: Tue Mar 14 22:06:00 2017 chromite: Scripts to lookup suite ids of builds from Milo. BUG=chromium:696762 TEST=find_build_suites --build_config master-release --num_builds 1 Change-Id: Ia437ae06c1c647881bfa163ade2e3e7c13f27e9d Reviewed-on: https://chromium-review.googlesource.com/447964 Commit-Ready: David Riley <davidriley@chromium.org> Tested-by: David Riley <davidriley@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [add] https://crrev.com/681d4a498543a0b6eb07037cdb548a493cec8266/scripts/find_build_suites [add] https://crrev.com/681d4a498543a0b6eb07037cdb548a493cec8266/scripts/find_build_suites.py
,
Mar 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/69d15e025bee28f349f2b53574d12390bef6b7d2 commit 69d15e025bee28f349f2b53574d12390bef6b7d2 Author: David Riley <davidriley@chromium.org> Date: Tue Mar 14 22:06:01 2017 classify_special_task: Parse JSON files. Add option to specify JSON file that lists the logs to be parsed. BUG=chromium:696762 TEST=autotest/contrib/find_suite_tasks.py --output tasks.json; classify_special_task --input_json tasks.json --find_suite_tasks TEST=classify_special_task_unittest Change-Id: Ice713e2f3be740d5f8581b036cf2e68814983426 Reviewed-on: https://chromium-review.googlesource.com/453277 Commit-Ready: David Riley <davidriley@chromium.org> Tested-by: David Riley <davidriley@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [add] https://crrev.com/69d15e025bee28f349f2b53574d12390bef6b7d2/scripts/classify_special_task_unittest [modify] https://crrev.com/69d15e025bee28f349f2b53574d12390bef6b7d2/scripts/classify_special_task.py [add] https://crrev.com/69d15e025bee28f349f2b53574d12390bef6b7d2/scripts/classify_special_task_unittest.py
,
Mar 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5939cdea914ccf0902f795cb096800f3ffc5535c commit 5939cdea914ccf0902f795cb096800f3ffc5535c Author: David Riley <davidriley@chromium.org> Date: Tue Mar 21 02:30:19 2017 [autotest] Script to find provision special tasks that fail. BUG=chromium:696762 TEST=find_suite_tasks 103972765 Change-Id: I6ea5ebc82408b4ee97ccfbf20c2902e7fe975b75 Reviewed-on: https://chromium-review.googlesource.com/447865 Commit-Ready: David Riley <davidriley@chromium.org> Tested-by: David Riley <davidriley@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [add] https://crrev.com/5939cdea914ccf0902f795cb096800f3ffc5535c/contrib/find_suite_tasks.py
,
Apr 4 2017
More work needed on this?
,
Apr 4 2017
Still active. I've added documentation on the current available tooling at: https://yaqs.googleplex.com/eng/q/5277276940009472?team_name=infra-product-taskforce&is_done=false&is_muted=false
,
Apr 18 2017
,
Mar 14 2018
This bug is Untriaged and very old. Because it has an owner, the status will be set to assigned to avoid closing a bug someone is using. If this bug still needs triage, change it back to Untriaged.
,
Jun 8 2018
,
Jan 11
I was looking to clean up CIDB stuff and noticed the chromite/scripts/find_build_suites.py file. Is anybody using the file now?
,
Jan 11
,
Jan 15
There's some analysis that can be done with that script that likely cannot be easily accomplished any other way, since it builds together information from a lot of different sources. That being said, it's likely somewhat bit rotted, and I'm not working on infra related things any more so I'm not sure if anyone will ever pick it up. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by dgarr...@chromium.org
, Feb 28 2017