This ticket is also about automating this process. Specifically, for Chromium that would involve moving towards and expectation-based structure like in WebKit or using clang magic to find test in the source and change the code.

Emma, how do automated suggested fix feature works in Critique/Tricorder? Is there some existing framework for that or do developers each create their own heuristics?

 Issue 109904  has been merged into this issue.

Update: created a doc to document how to disable a test: https://docs.google.com/document/d/1NBLvUr0CNDn15jG5gloSxD7M1CTM9WwjlDwgmaQXZik.

We currently have higher priority issues (e.g. issue 615468, issue 576274), therefore lowering the priority. Also writing automation is non-trivial as it requires semantic understanding of the code. This will probably be built on top of a clang-based analysis platform that will probably be developed as part of Tricium (http://go/tricium). Since this will not happen soon, I am going to remove myself from owners.

Nothing internal here.

Public version of the doc from #7: https://docs.google.com/document/d/1QpxgJBwGp37dUSUQa2FM1w-84EKu703kKXTI5nL0dng.

 Issue 680615  has been merged into this issue.

Ahh, thanks for the merge! I'm glad to know work has already been done on this issue. 

CCing Emma to ask if Tricium has changed enough since the previous update on the bug to make sense as something to use to identify tests in code and edit them. 

And if not, should we wait or seek out other alternatives for developing a tool to edit tests. 

To clarify, Tricimum is code analysis tool for uploaded code reviews and is not actually designed for doing arbitrary clang-matcher queries. I was just hoping that we can build on parts of the pipeline used by Tricium, once it is working.

Looks like I missed the initial Tricium question on this bug. or had an offline discussion with sergiyb.

For fixes suggested by robot comments, there is a format for that. The format to be used internally in Tricium is added here: https://cs.chromium.org/chromium/infra/go/src/infra/tricium/api/v1/data.proto?rcl=0e994cf750e5727b02b74304e97e9819e01fdf30&l=81
The included information is very close to that in Gerrit robot comments, which you now can read more about here: https://gerrit-review.googlesource.com/Documentation/config-robot-comments.html

It sounds like the data format for expressing fixes could match the need here. However, I'm not sure this is an obvious match for Tricium which is aimed at scenarios where issues in code result in comments with possible suggestions. 

That said, the main difference seems to be the triggering. The triggering in this context seems to be that a sheriff detects a problem and wants to disable a test. At that point, you would like a fix disabling the selected test to be generated. In Tricium, the application of a suggested fix will happen in Gerrit (resulting in a new patch set). 

I think major part of Tricium that we can re-use is the Clank analysis pipeline. I am not sure how it will work, but would be nice if we could trigger some endpoint with the following parameters:

 - set of ClangMatcher rules and matching strings. If any matches - replace the
   found code with the string. 
 - set of ClangMatcher rules blacklisting automatic changes in special cases,
   e.g. when the test name is actually defined elsewhere (https://codereview.chromium.org/2692003002).

and get a CL link in response (with CQ bit clicked).

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/074638fc3349692a57aab99a5dd467cdb26d6de8

commit 074638fc3349692a57aab99a5dd467cdb26d6de8
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Aug 09 19:22:53 2017

[som] Add test expecation logic to match statements with {test, config}

Bug: 603982,622359
Change-Id: I567a760fb80f7290bf9c632a154434150cbdfb2e
Reviewed-on: https://chromium-review.googlesource.com/606457
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/074638fc3349692a57aab99a5dd467cdb26d6de8/go/src/infra/libs/testexpectations/expectations.go
[modify] https://crrev.com/074638fc3349692a57aab99a5dd467cdb26d6de8/go/src/infra/libs/testexpectations/expectations_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7601fb2e24a207e2582b96d6a0354670b1f5745d

commit 7601fb2e24a207e2582b96d6a0354670b1f5745d
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Aug 17 22:02:28 2017

[som] Attempt to add test expectation information to test failure alerts

Bug: 603982,622359
Change-Id: Ia3fe45f12b14eaef5f19f91d5f15efd6b1ae3093
Reviewed-on: https://chromium-review.googlesource.com/617609
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/7601fb2e24a207e2582b96d6a0354670b1f5745d/go/src/infra/monitoring/analyzer/step/test_step.go
[modify] https://crrev.com/7601fb2e24a207e2582b96d6a0354670b1f5745d/go/src/infra/libs/testexpectations/expectations.go
[modify] https://crrev.com/7601fb2e24a207e2582b96d6a0354670b1f5745d/go/src/infra/libs/testexpectations/expectations_test.go
[modify] https://crrev.com/7601fb2e24a207e2582b96d6a0354670b1f5745d/go/src/infra/monitoring/analyzer/step/test_step_test.go
[modify] https://crrev.com/7601fb2e24a207e2582b96d6a0354670b1f5745d/go/src/infra/appengine/sheriff-o-matic/som/analyze.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/010f098723fc831693943dbfac7c082aeb1082b4

commit 010f098723fc831693943dbfac7c082aeb1082b4
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu Nov 09 21:04:27 2017

[som] TA/DA: allow adding expectations for tests that don't already
have them in any existing expectations files.

Also:
- adds parameters for the expectation edit page so we can link to
"add these modifiers and expectations to this test".
- adds a countdown timer to update the UI to show seconds remaining
until next retry.

Bug: 622359,603982
Change-Id: I4b2bf9478204809638734947e0925a90326e56c6
Reviewed-on: https://chromium-review.googlesource.com/761198
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/testexpectations/expectations_test.go
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/testexpectations/expectations.go
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-test-expectations/som-test-expectations.js
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/handler/layout_tests.go
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/frontend/queue.yaml
[modify] https://crrev.com/010f098723fc831693943dbfac7c082aeb1082b4/go/src/infra/appengine/sheriff-o-matic/som/analyzer/step/test_step.go