Sheriff-o-matic request: link to flakiness dashboard for non-layout tests |
|||
Issue descriptionSheriff-o-matic links to the flakiness dashboard ( https://test-results.appspot.com/dashboards/flakiness_dashboard.html ) for Blink layout tests, which is really useful for seeing test history. I can't find any similar links for other test failures, though. For example, today I spent time trying to figure out some browser_test timeouts, then realized I could just go to https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=MediaRouterElementsBrowserTest.MediaRouterContainerFilter and look to see if the timeouts were flaky versus consistently recent. It would be nice to have these sort of links surfaced automatically. This may be broader than just sheriff-o-matic; I dunno if the buildbot output/parsing would have to change to surface more detail of specific tests to make this possible. And I have no proposal for the precise UI to expose this. Just that it's useful functionality when sheriffing, that's currently pretty hidden for non-layout tests.
,
Oct 20 2016
In this case, the relevant test runs are appearing on the dashboard, but they weren't called out with links in sheriff-o-matic. It simply said that browser_tests were failing on Win debug, with no test names. A relevant log: https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20(dbg)(1)/builds/53861
,
Oct 20 2016
Do we get test results uploaded from swarming?
,
Oct 20 2016
(Note for example that as I write this, sheriff-o-matic is listing four different failures for Chromium, but only the webkit_tests one has a dashboard link.)
,
Oct 20 2016
Yeah, I saw it live as well. We should; it's strange, because it says that it's uploading test results for tests run on swarming. https://uberchromegw.corp.google.com/i/chromium.linux/builders/Linux%20Tests/builds/47898/steps/Upload%20to%20test-results%20%5Bbrowser_tests%20on%20Ubuntu-12.04%5D/logs/stdio I have meetings and I'm a sheriff today, so I don't know how soon I'll get to look at this :( I'll try to by end of day.
,
Oct 20 2016
,
Oct 27 2016
Ok, I found some of the problem. The title of the test is "browser_tests on Windows-7-SP1", but in alerts-dispatcher, we see the test name as "browser_tests". Not sure exactly how to fix this, but I'll look around some.
,
Nov 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/fe2cc6c15a39c25425c274f966f2cf2f07051be7 commit fe2cc6c15a39c25425c274f966f2cf2f07051be7 Author: Stephen Martinis <martiniss@chromium.org> Date: Fri Oct 28 03:20:32 2016 AD: Fix for swarming tests BUG= 657955 Change-Id: If74e49f336b68b555e2406be543585681b2cb550 Reviewed-on: https://chromium-review.googlesource.com/404648 Reviewed-by: Sean McCullough <seanmccullough@chromium.org> Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/fe2cc6c15a39c25425c274f966f2cf2f07051be7/go/src/infra/monitoring/analyzer/step/test_step.go [modify] https://crrev.com/fe2cc6c15a39c25425c274f966f2cf2f07051be7/go/src/infra/monitoring/analyzer/step/test_step_test.go
,
Nov 11 2016
This is live in production sheriff-o-matic. Yay! |
|||
►
Sign in to add a comment |
|||
Comment 1 by martiniss@chromium.org
, Oct 20 2016Status: Assigned (was: Untriaged)
In general, if the test uploads data to test results, we will show you the data. So, the problem is the test you care isn't uploading data somehow. For the UI, I attached what we currently do for links to the test results dashboard. The test itself ("virtual/stable/webexposed/global-interface-listing.html") links to the page on test results for it. Is that an ok UI? Were you thinking about something else? Do you happen to have a link to the buildbot log of the failed step you were trying to investigate? So I can dig into why that didn't end up showing up on Sheriff-o-matic.51.5 KB
51.5 KB View Download