Currently the chromium trybot recipe attempts to handle failures on the main waterfalls by comparing the failures that arise in a tryjob with the current patch against the failures that happen if the patch is deapplied; if the same failure happens both with and without the patch, we decide that the failure isn't the patch's fault and ignore it for purposes of determining if the tryjob fails.
However, there are a class of changes for which we *don't* do this, i.e., the failure is considered fatal and we don't even try to deapply the patch. I think the original thinking for this was that we thought we couldn't figure out what to re-run because the configuration had changed. In other words, if we add a new test step as part of the patch, and then de-applied the patch, we weren't smart enough to figure out that we didn't need to run the step for comparison.
Given that not being able to de-apply the patch means that a class of CL will then fail tryjobs for reasons that aren't the CL's fault, this is bad, and we should try to make this happen as rarely as possible (ideally, never).
Comment 1 by dpranke@chromium.org
, Dec 2 2017