New issue
Advanced search Search tips

Issue 711733 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: ----



Sign in to add a comment

SoM entries for failing webkit tests are listing the wrong test failures

Project Member Reported by dpranke@chromium.org, Apr 14 2017

Issue description

At the moment (2017-04-14 11:06 PDT), there are three sets of failures reported for the webkit_tests, one each for mac, linux, and windows. Each of the entries contains a list of failing tests, but the lists of failing tests do not match what the builders are actually reporting.

Examples:


(1)

webkit_tests failing on chromium.webkit/WebKit Mac10.11 (dbg)
[Help]
 Bug 708330    Bug 709188  
Active for: 29h 31m 7s
Examine
 (2)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
WebKit Mac10.11 (dbg) [19 out of the last 19 builds have failed]
41 tests failed:
...... too many results, data snipped.... ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled-clipped.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled-late-composite.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled-late-noncomposite.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe-composited.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe-scrolled.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-iframe.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-composited-scroll-clip.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-composited-scrolled.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-composited.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-layout-change-2.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-layout-change.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-scrolled-late-composite.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-scrolled-late-noncomposite.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div-scrolled.html ( Code Search )
compositing/gestures/gesture-tapHighlight-1-overflow-div.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe-composited-inner.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe-composited-outer.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-inner-late-composite.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-inner.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-outer-late-composite.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-outer.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-iframe.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-inner-scroll-inner.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-inner-scroll-outer.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-inner.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-outer-scroll-inner.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-outer-scroll-outer.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-outer.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-scrolled-inner.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div-scrolled-outer.html ( Code Search )
compositing/gestures/gesture-tapHighlight-2-overflow-div.html ( Code Search )
compositing/gestures/gesture-tapHighlight-adjustment-clipping.html ( Code Search )
compositing/gestures/gesture-tapHighlight-composited-img.html ( Code Search )
compositing/gestures/gesture-tapHighlight-form-input-text.html ( Code Search )
compositing/gestures/gesture-tapHighlight-imagemap.html ( Code Search )
compositing/gestures/gesture-tapHighlight-img-and-text-2.html ( Code Search )
compositing/gestures/gesture-tapHighlight-img-and-text.html ( Code Search )
compositing/gestures/gesture-tapHighlight-img-transformed.html ( Code Search )
compositing/gestures/gesture-tapHighlight-img.html ( Code Search )

But on the builder, 

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.11%20%28dbg%29/builds/8344
Unexpected Failures:
* external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm
* paint/invalidation/text-match-document-change.html
* virtual/disable-spinvalidation/paint/invalidation/text-match-document-change.html


(2) 

webkit_tests failing on chromium.webkit/WebKit Linux Trusty (dbg)
[Help]
 Bug 709188    Bug 710824  
Active for: 10h 13m 54s
Examine
 (5)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
WebKit Linux Trusty (dbg) [5 out of the last 5 builds have failed]
41 tests failed:
...... too many results, data snipped.... ( Code Search )
editing/input/emacs-ctrl-o.html ( Code Search )
editing/pasteboard/emacs-cntl-y-001.html ( Code Search )
editing/pasteboard/emacs-ctrl-a-k-y.html ( Code Search )
editing/pasteboard/emacs-ctrl-k-with-move.html ( Code Search )
editing/pasteboard/emacs-ctrl-k-y-001.html ( Code Search )
external/wpt/accelerometer/idlharness.https.html ( Code Search )
external/wpt/assumptions/canvas-background.html ( Code Search )
external/wpt/assumptions/html-elements.html ( Code Search )
external/wpt/assumptions/initial-color.html ( Code Search )
external/wpt/assumptions/medium-font-size.html ( Code Search )
external/wpt/assumptions/min-font-size.html ( Code Search )
external/wpt/auxclick/auxclick_event-manual.html ( Code Search )
external/wpt/clear-site-data/navigation.html ( Code Search )
external/wpt/css-values/unset-value-storage.html ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-002.xht ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-003.xht ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-004.xht ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-005.xht ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-006.xht ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-007.xht ( Code Search )
external/wpt/css/CSS2/linebox/inline-formatting-context-023.xht ( Code Search )
external/wpt/css/CSS2/linebox/line-height-126.xht ( Code Search )
external/wpt/css/CSS2/linebox/vertical-align-sub-001.xht ( Code Search )
external/wpt/css/CSS2/linebox/vertical-align-super-001.xht ( Code Search )
external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html ( Code Search )
external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html ( Code Search )
external/wpt/gyroscope/idlharness.https.html ( Code Search )
external/wpt/html/obsolete/requirements-for-implementations/the-marquee-element-0/marquee-events.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-input-email-add-manual.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-input-password-add-manual.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-input-search-add-manual.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-input-tel-add-manual.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-input-text-add-manual.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-input-url-add-manual.html ( Code Search )
external/wpt/html/semantics/forms/constraints/tooShort-textarea-add-manual.html ( Code Search )
external/wpt/html/the-xhtml-syntax/parsing-xhtml-documents/xhtml-mathml-dtd-entity-support.htm ( Code Search )
external/wpt/longtask-timing/longtask-attributes.html ( Code Search )
external/wpt/longtask-timing/longtask-in-childiframe-crossorigin.html ( Code Search )
external/wpt/longtask-timing/longtask-in-childiframe.html ( Code Search )
external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm ( Code Search )

on the builder:
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.11%20%28dbg%29/builds/8344

Unexpected Failures:
* external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm
* external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html
 
(so, two of the failures are right)

(3) 

webkit_tests failing on chromium.webkit/WebKit Win7 (dbg)
[Help]
 Bug 710824    Bug 709030    Bug 710864  
Active for: 2h 27m 46s
Examine
 (6)
 LINK/FILE BUG
 SNOOZE
Builders this step failed on:
WebKit Win7 (dbg) [2 out of the last 2 builds have failed]
41 tests failed:
...... too many results, data snipped.... ( Code Search )
editing/input/emacs-ctrl-o.html ( Code Search )
editing/input/linux_ltr_composition_underline.html ( Code Search )
editing/input/linux_rtl_composition_underline.html ( Code Search )
editing/pasteboard/emacs-cntl-y-001.html ( Code Search )
editing/pasteboard/emacs-ctrl-a-k-y.html ( Code Search )
editing/pasteboard/emacs-ctrl-k-with-move.html ( Code Search )
editing/pasteboard/emacs-ctrl-k-y-001.html ( Code Search )
editing/selection/linux_selection_color.html ( Code Search )
external/wpt/accelerometer/idlharness.https.html ( Code Search )
external/wpt/assumptions/canvas-background.html ( Code Search )
external/wpt/assumptions/html-elements.html ( Code Search )


on the builder:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Win7%20%28dbg%29/builds/9501

Unexpected Failures:
* external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-18.html
* virtual/service-worker-navigation-preload-disabled/webexposed/global-interface-listing.html

 
som.png
991 KB View Download
Labels: -Pri-2 Pri-1
Status: Available (was: Untriaged)
uploaded a dispatcher snapshot to 
gs://sheriff-o-matic.appspot.com/debug-snapshots/crbug-711733/broken-webkit-tests.tar.gz

should be able to replay that. the alerts will be slightly different since it took me a while to get snapshotting to work and some time had passed.
Cc: -seanmccullough@chromium.org
Owner: seanmccullough@chromium.org
Status: Started (was: Available)
Think I found the problem. CL out for review soon.
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/82f69457f005dab4ca99a1b61e8c26e4b8c558ff

commit 82f69457f005dab4ca99a1b61e8c26e4b8c558ff
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Fri Apr 14 21:14:33 2017

[som] Trust is_unexpected in test result logs.

This was causing tests that were marked Skip/WontFix to be included
in unexpected test failures, which led to alerts indicating way too
many test failures.

Bug:711733

Change-Id: I04fdffd6f273baa73d1a5633f88bba80a54e28e7
Reviewed-on: https://chromium-review.googlesource.com/478056
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/82f69457f005dab4ca99a1b61e8c26e4b8c558ff/go/src/infra/monitoring/analyzer/step/test_step.go
[modify] https://crrev.com/82f69457f005dab4ca99a1b61e8c26e4b8c558ff/go/src/infra/monitoring/analyzer/step/test_step_test.go

Status: Fixed (was: Started)
Running the fix CL on that snapshot had this effect on the "webkit_tests failing on chromium.webkit/WebKit Linux Trusty (dbg)" alert:

Before (70ish tests indicated):

"reason": {
  "test_names": [
    "...... too many results, data snipped....",
    "editing/input/emacs-ctrl-o.html",
    "editing/pasteboard/emacs-cntl-y-001.html",
    "editing/pasteboard/emacs-ctrl-a-k-y.html",
    "editing/pasteboard/emacs-ctrl-k-with-move.html",
    "editing/pasteboard/emacs-ctrl-k-y-001.html",
    "external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm",
    "external/wpt/accelerometer/idlharness.https.html",
    "external/wpt/assumptions/canvas-background.html",
    "external/wpt/assumptions/html-elements.html",
    "external/wpt/assumptions/initial-color.html",
    "external/wpt/assumptions/medium-font-size.html",
    "external/wpt/assumptions/min-font-size.html",
    "external/wpt/auxclick/auxclick_event-manual.html",
    "external/wpt/clear-site-data/navigation.html",
    "external/wpt/css-values/unset-value-storage.html",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-002.xht",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-003.xht",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-004.xht",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-005.xht",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-006.xht",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-007.xht",
    "external/wpt/css/CSS2/linebox/inline-formatting-context-023.xht",
    "external/wpt/css/CSS2/linebox/line-height-126.xht",
    "external/wpt/css/CSS2/linebox/vertical-align-sub-001.xht",
    "external/wpt/css/CSS2/linebox/vertical-align-super-001.xht",
    "external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html",
    "external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html",
    "external/wpt/gyroscope/idlharness.https.html",
    "external/wpt/html/obsolete/requirements-for-implementations/the-marquee-element-0/marquee-events.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-input-email-add-manual.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-input-password-add-manual.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-input-search-add-manual.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-input-tel-add-manual.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-input-text-add-manual.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-input-url-add-manual.html",
    "external/wpt/html/semantics/forms/constraints/tooShort-textarea-add-manual.html",
    "external/wpt/html/the-xhtml-syntax/parsing-xhtml-documents/xhtml-mathml-dtd-entity-support.htm",
    "external/wpt/longtask-timing/longtask-attributes.html",
    "external/wpt/longtask-timing/longtask-in-childiframe-crossorigin.html",
    "external/wpt/longtask-timing/longtask-in-childiframe.html"
   ],                                    ],
 ...
}

After (only three tests indicated):

"reason": {
 "test_names": [                                               
   "external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm",
   "external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html",
   "external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html"
  ],
 ...
}

Cc: seanmccullough@chromium.org
 Issue 712087  has been merged into this issue.

Sign in to add a comment