New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature

Blocked on:
issue 674692



Sign in to add a comment
link

Issue 725160: [WPT Import] Handle expectations for other build types besides basic Release builds.

Reported by qyears...@chromium.org, May 22 2017 Project Member

Issue description

This is a general bug, covering the below specific cases:

 - Some tests behave differently with DCHECK on, as is the case for CQ bots (bug 674692).
 - Relatedly, some tests behave differently in debug builds.
 - Some tests fail in ASAN/MSAN or Leak bots ( bug 662570 ).

My thoughts about the general way to solve these problems is: We need to use more try bots and use their results when updating expectations. For example:

 - *_chromium_rel_ng CQ bots could be used to get the results when DCHECK is enabled
 - *_blink_dbg bots on tryserver.blink could also provide results for debug builds
 - new try bots could be added for ASAN/MSAN or Leak bots

In the wpt-update-expectations, some important changes will need to be made:
 - "Debug" specifiers should be added for Debug-specific failures/timeouts/crashes (but we never add debug-specific baselines).
 - ASAN/MSAN/Leak expectations for Timeout/Failure/Crash also need to be added to their own special expectations files (again, we don't want to add baselines).
 

Comment 1 by qyears...@chromium.org, May 22 2017

Blockedon: 674692 662570

Comment 2 by qyears...@chromium.org, Jun 16 2017

Cc: jsb...@chromium.org foolip@chromium.org jeffcarp@chromium.org
 Issue 662570  has been merged into this issue.

Comment 3 by qyears...@chromium.org, Jun 16 2017

Blockedon: -662570
Summary: [WPT Import] Handle expectations for other build types besides basic Release builds. (was: wpt-update-expectations doesn't handle results from build types other than basic Release builds. )

Comment 4 by foolip@chromium.org, Jul 3 2017

Components: Blink>Infra>Ecosystem

Comment 5 by foolip@chromium.org, Jul 3 2017

Components: -Blink>Infra>Predictability

Comment 6 by sheriffbot@chromium.org, Jul 3 2018

Project Member
Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 7 by foolip@chromium.org, Jul 18 2018

Cc: robertma@chromium.org
robertma@, what do you think about this avenue? It seems to me that right now our biggest problem is actually that we don't consider CQ bots, and that simply by triggering those and using the results, we'd have sufficient coverage?

Comment 8 by robertma@chromium.org, Jul 18 2018

Labels: -Type-Bug -Hotlist-Recharge-Cold Type-Feature
Status: Available (was: Untriaged)
Re #7: using the CQ bot to rebaseline is specifically issue 739522, which would solve the DCHECK problem; I might try to find some time to think about the implementation there.

On the other hand, this issue has a much larger scope and won't be happening any time soon.

Sign in to add a comment