New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 603982 link

Starred by 4 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature



Sign in to add a comment

Add easy test_expectations updates to Sheriff-o-matic

Project Member Reported by jpar...@google.com, Apr 15 2016

Issue description

Motivation: Sheriffing is still too hard and time consuming.  One common task is disabling failing tests.

This is a feature request to make it super easy to update a failing LayoutTest right from the SOM UI, ideally with a button right next to the failing test "disable".  Note that this will just be for LayoutTests, since they use test_expectations, which will be easier to pipe into this workflow (other test suites have more complicated ways of disabling tests).

Ojan, please fill in any details I missed.
 
I was just talking to wkorman about this exact feature yesterday. So it is on my radar.
Project Member

Comment 2 by sheriffbot@chromium.org, Jun 3 2016

Labels: Hotlist-Google

Comment 3 by aga...@chromium.org, Nov 22 2016

Status: Available (was: Untriaged)
This is a fantastic idea and totally worth doing. Triaging into Available.
Components: Infra>Sheriffing>SheriffOMatic
Labels: -Sheriff-O-Matic Milestone-Workflow

Comment 5 by ojan@chromium.org, Mar 7 2017

Cc: -ojan@chromium.org
Cc: -seanmccullough@chromium.org
Owner: seanmccullough@chromium.org
Status: Started (was: Available)
Project Member

Comment 7 by bugdroid1@chromium.org, Apr 20 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d17b5f7ece1da54718e2e7feb691422ec7216857

commit d17b5f7ece1da54718e2e7feb691422ec7216857
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Apr 20 20:28:29 2017

[som] update deps to pull in my fixes to go-gerrit client lib

https://github.com/andygrunwald/go-gerrit/pull/30
https://github.com/andygrunwald/go-gerrit/pull/31
https://github.com/andygrunwald/go-gerrit/pull/32

Bug:603982

Change-Id: I55f7b9b5ca84768752d2442e80928cd677a590b6
Reviewed-on: https://chromium-review.googlesource.com/483032
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/d17b5f7ece1da54718e2e7feb691422ec7216857/go/deps.lock

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 20 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d17b5f7ece1da54718e2e7feb691422ec7216857

commit d17b5f7ece1da54718e2e7feb691422ec7216857
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Apr 20 20:28:29 2017

[som] update deps to pull in my fixes to go-gerrit client lib

https://github.com/andygrunwald/go-gerrit/pull/30
https://github.com/andygrunwald/go-gerrit/pull/31
https://github.com/andygrunwald/go-gerrit/pull/32

Bug:603982

Change-Id: I55f7b9b5ca84768752d2442e80928cd677a590b6
Reviewed-on: https://chromium-review.googlesource.com/483032
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/d17b5f7ece1da54718e2e7feb691422ec7216857/go/deps.lock

Project Member

Comment 11 by bugdroid1@chromium.org, Apr 21 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/9f3e105eeea3b689c930342437bea9b1a84061e8

commit 9f3e105eeea3b689c930342437bea9b1a84061e8
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Fri Apr 21 18:34:46 2017

[som] Add basic geritt support for creating CLs

Bug:603982

Change-Id: I6116bdef16dae6192c73aac0a92809bee8bc9c52
Reviewed-on: https://chromium-review.googlesource.com/483969
Reviewed-by: Aaron Gable <agable@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[add] https://crrev.com/9f3e105eeea3b689c930342437bea9b1a84061e8/go/src/infra/appengine/sheriff-o-matic/som/gerrit_test.go
[add] https://crrev.com/9f3e105eeea3b689c930342437bea9b1a84061e8/go/src/infra/appengine/sheriff-o-matic/som/gerrit.go

Project Member

Comment 15 by bugdroid1@chromium.org, May 18 2017

Project Member

Comment 16 by bugdroid1@chromium.org, May 19 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c35667cdc90baae88509800d3b1b9900993b05cd

commit c35667cdc90baae88509800d3b1b9900993b05cd
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Fri May 19 22:48:14 2017

[som] Frontend for layout test expectations list.

vaadin-grid was recommended as a 'data table' element on polymer-discuss@:
https://groups.google.com/a/google.com/forum/?utm_source=digest&utm_medium=email#!topic/polymerdiscuss/C62bYsfkJhs

Bug: 603982
Change-Id: Iabdaf5fae452db9c8176be9a647d8c7eba4538da
Reviewed-on: https://chromium-review.googlesource.com/509912
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[add] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-test-expectations/som-test-expectations.html
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/test/index.html
[add] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/test/som-test-expectations-test.html
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-app/som-app.html
[add] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-app/som-app.js
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/bower.json

Project Member

Comment 17 by bugdroid1@chromium.org, May 19 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c35667cdc90baae88509800d3b1b9900993b05cd

commit c35667cdc90baae88509800d3b1b9900993b05cd
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Fri May 19 22:48:14 2017

[som] Frontend for layout test expectations list.

vaadin-grid was recommended as a 'data table' element on polymer-discuss@:
https://groups.google.com/a/google.com/forum/?utm_source=digest&utm_medium=email#!topic/polymerdiscuss/C62bYsfkJhs

Bug: 603982
Change-Id: Iabdaf5fae452db9c8176be9a647d8c7eba4538da
Reviewed-on: https://chromium-review.googlesource.com/509912
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[add] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-test-expectations/som-test-expectations.html
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/test/index.html
[add] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/test/som-test-expectations-test.html
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-app/som-app.html
[add] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/elements/som-app/som-app.js
[modify] https://crrev.com/c35667cdc90baae88509800d3b1b9900993b05cd/go/src/infra/appengine/sheriff-o-matic/bower.json

Project Member

Comment 18 by bugdroid1@chromium.org, Jun 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/911a8e5b6b9c057bfff3b109af356a911f932615

commit 911a8e5b6b9c057bfff3b109af356a911f932615
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Jun 08 17:58:00 2017

[som] Add LayoutTest expectation editing form.

This is just the front-end, no server-side action to actuall make the
changes yet.

Bug:603982
Change-Id: I6abd574d26e2fc86b31908944f9ffc13c908f537
Reviewed-on: https://chromium-review.googlesource.com/528058
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>

[modify] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/test/index.html
[add] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/test/som-test-expectation-form-test.html
[modify] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/Makefile
[add] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-edit-expectation-form.js
[modify] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/test/som-test-expectations-test.html
[modify] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.html
[modify] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-alert-item/som-alert-item.js
[add] https://crrev.com/911a8e5b6b9c057bfff3b109af356a911f932615/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-edit-expectation-form.html

Project Member

Comment 20 by bugdroid1@chromium.org, Jun 16 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d23930af43996fa483e4b65d55149ec128f57b64

commit d23930af43996fa483e4b65d55149ec128f57b64
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Fri Jun 16 22:26:16 2017

[som] Add server-side handler for updating layout test expectations.

Bug: 603982
Change-Id: I6d516174ab8890aaf705f7bc8611220930b273c7
Reviewed-on: https://chromium-review.googlesource.com/538039
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>

[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/libs/testexpectations/expectations.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/appengine/sheriff-o-matic/frontend/main.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/appengine/sheriff-o-matic/som/layout_tests.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/libs/testexpectations/parser_test.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/appengine/sheriff-o-matic/som/layout_tests_test.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/appengine/sheriff-o-matic/som/gerrit.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/appengine/sheriff-o-matic/som/gerrit_test.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/libs/testexpectations/parser.go
[modify] https://crrev.com/d23930af43996fa483e4b65d55149ec128f57b64/go/src/infra/libs/testexpectations/expectations_test.go

Project Member

Comment 21 by bugdroid1@chromium.org, Jun 21 2017

Project Member

Comment 22 by bugdroid1@chromium.org, Jun 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e854b3687ef0da225ccd633123bc4cd834197a2b

commit e854b3687ef0da225ccd633123bc4cd834197a2b
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Jun 22 22:09:49 2017

[som] Move CL-generating logic to async worker queue.

This change makes it possible for the UI to show
status of the pending change CL async worker task. 

Bug: 603982
Change-Id: I23670e30aa988f4b9a3dfda8d76badd76654654a
Reviewed-on: https://chromium-review.googlesource.com/543912
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/e854b3687ef0da225ccd633123bc4cd834197a2b/go/src/infra/appengine/sheriff-o-matic/som/layout_tests_test.go
[modify] https://crrev.com/e854b3687ef0da225ccd633123bc4cd834197a2b/go/src/infra/appengine/sheriff-o-matic/frontend/main.go
[modify] https://crrev.com/e854b3687ef0da225ccd633123bc4cd834197a2b/go/src/infra/appengine/sheriff-o-matic/som/layout_tests.go

Project Member

Comment 23 by bugdroid1@chromium.org, Jun 28 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a731f0a7fb4adbb895922b67282139251d422a1a

commit a731f0a7fb4adbb895922b67282139251d422a1a
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Jun 28 18:04:15 2017

[som] Add status message, polling for async CL worker to expectations UI

Bug: 603982
Change-Id: I36d9a3b4837468290af76224806bfa11b41c78af
Reviewed-on: https://chromium-review.googlesource.com/550602
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/test/index.html
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/som/layout_tests.go
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-edit-expectation-form.js
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/bower.json
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.html
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/test/som-test-expectations-test.html
[modify] https://crrev.com/a731f0a7fb4adbb895922b67282139251d422a1a/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-edit-expectation-form.html

Project Member

Comment 25 by bugdroid1@chromium.org, Aug 9 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/074638fc3349692a57aab99a5dd467cdb26d6de8

commit 074638fc3349692a57aab99a5dd467cdb26d6de8
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Aug 09 19:22:53 2017

[som] Add test expecation logic to match statements with {test, config}

Bug: 603982,622359
Change-Id: I567a760fb80f7290bf9c632a154434150cbdfb2e
Reviewed-on: https://chromium-review.googlesource.com/606457
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/074638fc3349692a57aab99a5dd467cdb26d6de8/go/src/infra/libs/testexpectations/expectations.go
[modify] https://crrev.com/074638fc3349692a57aab99a5dd467cdb26d6de8/go/src/infra/libs/testexpectations/expectations_test.go

Cc: rnep...@chromium.org
I will likely be the one that integrates this with work Telemetry, so CCing myself to keep up to date with the progress. 
Project Member

Comment 28 by bugdroid1@chromium.org, Nov 9 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/010f098723fc831693943dbfac7c082aeb1082b4

commit 010f098723fc831693943dbfac7c082aeb1082b4
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Nov 09 21:04:27 2017

[som] TA/DA: allow adding expectations for tests that don't already
have them in any existing expectations files.

Also:
- adds parameters for the expectation edit page so we can link to
"add these modifiers and expectations to this test".
- adds a countdown timer to update the UI to show seconds remaining
until next retry.

Bug: 622359,603982
Change-Id: I4b2bf9478204809638734947e0925a90326e56c6
Reviewed-on: https://chromium-review.googlesource.com/761198
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/testexpectations/expectations_test.go
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/testexpectations/expectations.go
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/handler/layout_tests.go
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/frontend/queue.yaml
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/analyzer/step/test_step.go

Project Member

Comment 29 by bugdroid1@chromium.org, Nov 30 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/006253c6a06ccd9dc961881cc11a1de76d01aa2b

commit 006253c6a06ccd9dc961881cc11a1de76d01aa2b
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Nov 30 20:33:59 2017

[som, TA/DA] Add link to test expectation editor.

This required some extra Behavior work to pre-fill the appropriate
"modifiers" for each builder.

This one is hard to demo since you have to catch a very specific kind of
alert (layout test failures) that isn't always active.

It looks like this: https://screenshot.googleplex.com/F911Y2DC3Ak
And clicking on it will pre-fill some checkboxes in the form for you: https://screenshot.googleplex.com/HARbuyQHVhr

Or try it live here: https://12975-4521a54-tainted-seanmcculloug-dot-sheriff-o-matic-staging.appspot.com/chromium
REPLY

Bug: 603982
Change-Id: I03756cc5bd826ae8c66e7dec6e18c4b99f7f5959
Reviewed-on: https://chromium-review.googlesource.com/795013
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/006253c6a06ccd9dc961881cc11a1de76d01aa2b/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-extension-build-failure/som-extension-build-failure.html
[modify] https://crrev.com/006253c6a06ccd9dc961881cc11a1de76d01aa2b/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-extension-build-failure/som-extension-build-failure.js
[modify] https://crrev.com/006253c6a06ccd9dc961881cc11a1de76d01aa2b/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-edit-expectation-form.html
[add] https://crrev.com/006253c6a06ccd9dc961881cc11a1de76d01aa2b/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-layout-test-builder-configs-behavior.html
[modify] https://crrev.com/006253c6a06ccd9dc961881cc11a1de76d01aa2b/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-edit-expectation-form.js

Sign in to add a comment