Could we get Sheriff-o-Matic to understand the structuring of webkit tests? It looks like SOM is already able to understand the individual test failing?

Initially maybe just using directory level grouping is a good start and then extending it to do some type of "longest common prefix" type grouping (maybe with some type of minimum depth?).

You'll probably want some type of special handling for virtual prefix too.

For example;

Grouping these together makes sense;
 - virtual/gpu/fast/canvas/canvas-clip-rule.html
 - virtual/gpu/fast/canvas/canvas-path-context-clip.html

Grouping these together make sense;
 - virtual/gpu/fast/canvas/canvas-clip-rule.html
 - gpu/fast/canvas/canvas-clip-rule.html

Grouping these together *doesn't* really make sense;
 - animations/missing-from-to.html
 - nfc/mock-nfc.html

It might or might not make sense to group things under css3/.* - A problem affecting css3/motion-path and css3/calc could be related or they might not...

---

An alternative might just let the user specify a regex to use for grouping stuff together? Maybe even when you merge two failing webkit_tests together you could calculate the regex automatically using the "longest common prefix" bit?

If SOM needs some more information or annotations from the output of buildbots to make this style of thing work, please do ask and I'll see what we can do..

Thanks for filing this bug! I think we definitely need to figure out something to make webkit_tests handled better. 

I like the idea of maybe using some sort of directory level grouping as a simple heuristic. I think user regexes is maybe adding a bit more mental overhead to the user than we would want for Sheriff-o-Matic, so I'd like to avoid that approach if possible. But I think we should be able to come up with some sort of system that at least makes the groups a bit more distinct than they are now.