idea #1 is out, you can't easily turn off the default bots.

Is this actually blocking the roll? Is there a way we can easily manually work around this?

If this only happens 1-2 times a month (or less) I would either manually work around it when it happens. So, that's what I'd probably do if we can for now.

If it happens frequently, we'd need to automate this. In this case, I'd probably modify the importer to *also* run the default jobs w/ dcheck enabled, and then have the importer merge the expectations from the two sets of builders.

We intentionally only run release+dcheck in the CQ, because we can't afford to run both release and debug tests, and release by itself won't catch debug failures (but dcheck catches most of them). We also don't want to have to run both release and release+dcheck, since that's double the work.

How does that sound?

So far, it happens once a month or less, and working around it without changing any try bots would be OK.

Manually working around it involves patching in the CL made by the autoroller, and adding a Crash expectation to get it passed the CQ. (Actually, does that sound correct? The new crash expectation applies only to builders with DCHECK enabled, not to normal Release builds.)

It's also possible to change the w3c-test-autoroller without changing any try bots: We'd want to also look at results from the CQ try bots, and if there are crash results there, then we'd add a Crash expectation and try again.


Anyway, hypothetically, do you think it would be a bad idea to do add separate layout test CQ builders that build Release without DCHECK and run just the layout tests? This would imply adding another 3-4 CQ builders which do a build of content_shell etc. and run layout tests for each CQ run. This seems a little more correct for layout tests, but maybe not worth it overall?

I think it's not worth it overall to add more builders, but we can maybe revisit once we have swarmed layout tests running well all the times it should be running and we can  see the impact on fleet capacity.

Does a dry run not use the same configuration as the CQ does? If it does, then maybe a dry run can be used to get the baselines? (And if not, why isn't everything on fire?)

A CQ dry run uses the same configuration as the normal CQ as far as I know -- both use Release builds with DCHECK on.

The reason why we can't use *only* the CQ bots *_chromium_rel_ng to get the baselines now is that they don't cover all supported platforms -- they only cover one version of mac, windows and linux, rather than all supported versions.

But once a dry run is done, we can indeed check the results of those runs and see which tests crashed on those bots; if a test crashed on (some of) the CQ bots but not on the other Blink try bots, then that will usually indicate that it crashed due to tripping a DCHECK.

Next action for this issue: Add functionality to the auto-import code (in webkitpy/w3c/deps_updater.py) to check the results of the CQ, and if there are Crash results, add a Crash expectation for the test, then try the CQ again.

As long as there aren't often new crashes in tests triggered by DCHECK, this is probably not very high priority.

A thought:

In general, and not just for import, there's no set of expectations that's really specific and correct that lets layout tests with DCHECK failures get through the CQ. The most specific and correct expectation would be:
  crbug.com/xxx [ Debug ] test [ Crash ]

But the configuration for CQ builders is still considered release, so this expectation wouldn't apply then. This means that to get through the CQ, we have to use the expectation:
  crbug.com/xxx test [ Crash Pass ]

But this doesn't explicitly say that the crash is related to DCHECK, this test will apply non-flaky on the continuous builders.

In most cases, for non-imported tests, the best way to solve this would be to never commit any tests that trigger DCHECK failures in the first place. But for imported tests we can't control this.

Anyway, my best idea now for how to handle this would still be to modify the auto-importer code (now in webkitpy/w3c/test_importer.py) to check the results of the CQ, and if there are Crash results, add a flaky crash expectation and try the CQ again. Another alternative would be to trigger *_chromium_rel_ng after first uploading the import CL, update expectations to include both the results there and on the try bots.

 Issue 752406  has been merged into this issue.

robertma@, this is a P2 issue that hasn't seen any activity for 60 days. Should we demote it to P3 if it's not causing regular problems, or should it actually be fixed soonish?

It only causes problems every once in a while (every 3-4 weeks?), but when this happens the wpt-import bot remains red for quite a while until we either manually skip the offending tests or fix the code that's triggering the DCHECKs, which affects the Q4 OKR to reduce WPT import latency.

This last happened earlier this week: builds #2521 up to #2536 (Sept 30th to Oct 2nd) were red until https://chromium-review.googlesource.com/689357 fixed the underlying issue.

Sorry this is actually my first time noticing this issue.

In the past three months, the importer was tripped by DCHECK twice I think, which is lower than the threshold (1 or 2 times a month, same as suggested by dpranke@) I'd like to prioritize automation. It's therefore safe to downgrade to P3.

Regarding the potential pragmatic fix discussed here (esp. comment 11), I think it's blocked on issue 739522 (rebaselining from CQ), which is also a P3.

 Issue 830490  has been merged into this issue.

It happened in  issue 830490  again last week, but wasn't all bad, the human involvement probably increased the speed at which the Blink bugs were fixed.