@akeshet - is this work immediate? FixIt?

It's a fixit sized task, but might be more urgent.

-> jkop@ starter bug

I see two options here:

1) have a list of "hw_tests" and a list of "experimental_hw_tests", only move flaky hw_tests to experimental_hw_tests (sometimes we may end up with moving all hw_tests to experimental_hw_tests)
2) have a boolean "hw_tests_experimental" flag, treat all hw_tests as experimental for the build. (sometimes only a few hw_tests are flaky; marking all hw_tests as experimental may not be able to catch some bad CLs)

which option are we going to adopt?

I think specifying this flag for the entire HWTest stage is sufficient, rather than at the per-test level. Given that, clarifying questions:

1) Do you mean a list of individual tests, or a list of stages? Would this be specified in chromeos_config, or elsewhere?

2) This sounds like my preferred approach. It could be a boolean flag that is part of the hwtest config object (i.e. each entry of a configs hw_tests=[...] list could independently specify this boolean)

I think that the quicker but less granular approach is probably better for most uses, but obviously I'm new and haven't dealt with the problem directly.

Additionally, there already is a closely-related per-test config option: "warn_only". See cs/symbol:HWTestConfig. The use for this as distinct from that seems small.

My origin thought for comment #7 1) is
In HWTest Stage, we run a list of important hw tests.
In ExperimentalHWTest stage, we run a list of experimental hw tests and this stage is ignorable if failed.

If we handle the warn_only correctly mentioned in #9, #7 1) would just be duplicated. We can proceed with 2). 

Returning to this.

Copying earlier comments from canceled CL:

>akeshet@:
>Good find. If that already exists and does what we intend, then I guess all we need for this bug is to a) document it somewhere sensible b) make it easy to turn on for paladin builders (maybe next to the area in chromeos_config where we select which boards should run hwtest should, we also have a set of (board,suite) that should be warn_only)

I believe he was referring to the http://chromiumos-lab.appspot.com/ build-cancel regexen, there. If so, the most natural option would seem to be changing the status format from 

Lab is 'status' [build REs] (comment) 

to

Lab is 'status' [build REs]{(board RE, suite RE),(board RE, suite RE),...} (comment) 

If we don't need to switch tests rapidly, and just need better docs, I have some here: doc/1kg1XC6Oz7Qj30wCQ2L5f1kv5XANjWq0kkG8uYw9OwBE/edit

I don't know where that should live, though.

the doc link is private, can you paste the shared link ?

Should be public now, same link.