sg

Had a meeting with machenbach@. We've reviewed V8 requirements and decided to start with Flakiness Surface as most flakes in V8 do not cause false rejections and the team would not benefit much from Chromium Try Flakes.

The first step is to add reporting of the test results to the test-results.appspot.com. Michael said he'll work on this soon.

One concern was that in V8 a single step may run the same test multiple times with varying configurations. After some discussion, we've decided to include these configuration options into the test name, but this is suboptimal since these options can easily take over 500 characters. Long-term we are planning to introduce a separate param called "config" that will allow to specify which config a given test was run in. This is also beneficial for other test types, e.g. GPU and Android tests can report platform name in the config, which is currently stripped from the step name during normalization. I'll write a proposal for this.

I wrote a proposal for adding configs to the test-results server: https://docs.google.com/document/d/1rFS9r9dIgTOgHnm84vtAxMtDAUZG1KfHk4Mle1ssS90/edit (sorry, internal only).

Since, at this point there are no AI for me, I'm going to assign this to Michael. Please re-assign back to me when you are done adding code reporting test results into the v8 test launcher.

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/d5ecdedf1397e70153e3c51857a2353d21b7d4d2

commit d5ecdedf1397e70153e3c51857a2353d21b7d4d2
Author: Michael Achenbach <machenbach@chromium.org>
Date: Thu Mar 02 14:26:04 2017

[test] Add new json test results output for flakiness dashboard

BUG=chromium:689108

Change-Id: Id39d247f267131b1abf968546c7555b3657b14f3
Reviewed-on: https://chromium-review.googlesource.com/439566
Reviewed-by: Jakob Kummerow <jkummerow@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>
Cr-Commit-Position: refs/heads/master@{#43546}
[modify] https://crrev.com/d5ecdedf1397e70153e3c51857a2353d21b7d4d2/tools/run-tests.py
[modify] https://crrev.com/d5ecdedf1397e70153e3c51857a2353d21b7d4d2/tools/testrunner/local/progress.py

Michael, is there something I can do to facilitate this work moving forward?

The parts on v8-infra side are still missing. You could dig into that as I'll not have time for the next 2-3 weeks.

1) pass the parameter added in comment 5 to the v8 test runner.
2) receive the output and merge it for swarming tests
3) write an upload hook

Not sure if there are suitable existing methods for 2+3

There is certainly a method for 3, probably also one for 2. Didn't you want to limit this to specific test types only to reduce potential impact?

I'd wanna limit it to a specific bot first.

Do you have a bot in mind? Or would any one bot work?

Any would work.

After talking to Michael offline, v8 does not experience much flakiness, therefore perhaps it's better to focus on other projects in EMEA that would benefit more from better flakiness tooling. Removing relevant labels. Michael, please let me know if you'd like this work to go forward at some point in the future.