SoM entries for failing webkit tests are listing the wrong test failures |
|||
Issue descriptionAt the moment (2017-04-14 11:06 PDT), there are three sets of failures reported for the webkit_tests, one each for mac, linux, and windows. Each of the entries contains a list of failing tests, but the lists of failing tests do not match what the builders are actually reporting. Examples: (1) webkit_tests failing on chromium.webkit/WebKit Mac10.11 (dbg) [Help] Bug 708330 Bug 709188 Active for: 29h 31m 7s Examine (2) LINK/FILE BUG SNOOZE Builders this step failed on: WebKit Mac10.11 (dbg) [19 out of the last 19 builds have failed] 41 tests failed: ...... too many results, data snipped.... ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled-clipped.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled-late-composite.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled-late-noncomposite.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe-composited-scrolled.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe-composited.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe-scrolled.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-iframe.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-composited-scroll-clip.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-composited-scrolled.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-composited.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-layout-change-2.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-layout-change.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-scrolled-late-composite.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-scrolled-late-noncomposite.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div-scrolled.html ( Code Search ) compositing/gestures/gesture-tapHighlight-1-overflow-div.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe-composited-inner.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe-composited-outer.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-inner-late-composite.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-inner.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-outer-late-composite.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe-scrolled-outer.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-iframe.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-inner-scroll-inner.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-inner-scroll-outer.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-inner.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-outer-scroll-inner.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-outer-scroll-outer.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-composited-outer.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-scrolled-inner.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div-scrolled-outer.html ( Code Search ) compositing/gestures/gesture-tapHighlight-2-overflow-div.html ( Code Search ) compositing/gestures/gesture-tapHighlight-adjustment-clipping.html ( Code Search ) compositing/gestures/gesture-tapHighlight-composited-img.html ( Code Search ) compositing/gestures/gesture-tapHighlight-form-input-text.html ( Code Search ) compositing/gestures/gesture-tapHighlight-imagemap.html ( Code Search ) compositing/gestures/gesture-tapHighlight-img-and-text-2.html ( Code Search ) compositing/gestures/gesture-tapHighlight-img-and-text.html ( Code Search ) compositing/gestures/gesture-tapHighlight-img-transformed.html ( Code Search ) compositing/gestures/gesture-tapHighlight-img.html ( Code Search ) But on the builder, https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.11%20%28dbg%29/builds/8344 Unexpected Failures: * external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm * paint/invalidation/text-match-document-change.html * virtual/disable-spinvalidation/paint/invalidation/text-match-document-change.html (2) webkit_tests failing on chromium.webkit/WebKit Linux Trusty (dbg) [Help] Bug 709188 Bug 710824 Active for: 10h 13m 54s Examine (5) LINK/FILE BUG SNOOZE Builders this step failed on: WebKit Linux Trusty (dbg) [5 out of the last 5 builds have failed] 41 tests failed: ...... too many results, data snipped.... ( Code Search ) editing/input/emacs-ctrl-o.html ( Code Search ) editing/pasteboard/emacs-cntl-y-001.html ( Code Search ) editing/pasteboard/emacs-ctrl-a-k-y.html ( Code Search ) editing/pasteboard/emacs-ctrl-k-with-move.html ( Code Search ) editing/pasteboard/emacs-ctrl-k-y-001.html ( Code Search ) external/wpt/accelerometer/idlharness.https.html ( Code Search ) external/wpt/assumptions/canvas-background.html ( Code Search ) external/wpt/assumptions/html-elements.html ( Code Search ) external/wpt/assumptions/initial-color.html ( Code Search ) external/wpt/assumptions/medium-font-size.html ( Code Search ) external/wpt/assumptions/min-font-size.html ( Code Search ) external/wpt/auxclick/auxclick_event-manual.html ( Code Search ) external/wpt/clear-site-data/navigation.html ( Code Search ) external/wpt/css-values/unset-value-storage.html ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-002.xht ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-003.xht ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-004.xht ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-005.xht ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-006.xht ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-007.xht ( Code Search ) external/wpt/css/CSS2/linebox/inline-formatting-context-023.xht ( Code Search ) external/wpt/css/CSS2/linebox/line-height-126.xht ( Code Search ) external/wpt/css/CSS2/linebox/vertical-align-sub-001.xht ( Code Search ) external/wpt/css/CSS2/linebox/vertical-align-super-001.xht ( Code Search ) external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html ( Code Search ) external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html ( Code Search ) external/wpt/gyroscope/idlharness.https.html ( Code Search ) external/wpt/html/obsolete/requirements-for-implementations/the-marquee-element-0/marquee-events.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-input-email-add-manual.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-input-password-add-manual.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-input-search-add-manual.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-input-tel-add-manual.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-input-text-add-manual.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-input-url-add-manual.html ( Code Search ) external/wpt/html/semantics/forms/constraints/tooShort-textarea-add-manual.html ( Code Search ) external/wpt/html/the-xhtml-syntax/parsing-xhtml-documents/xhtml-mathml-dtd-entity-support.htm ( Code Search ) external/wpt/longtask-timing/longtask-attributes.html ( Code Search ) external/wpt/longtask-timing/longtask-in-childiframe-crossorigin.html ( Code Search ) external/wpt/longtask-timing/longtask-in-childiframe.html ( Code Search ) external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm ( Code Search ) on the builder: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.11%20%28dbg%29/builds/8344 Unexpected Failures: * external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm * external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html (so, two of the failures are right) (3) webkit_tests failing on chromium.webkit/WebKit Win7 (dbg) [Help] Bug 710824 Bug 709030 Bug 710864 Active for: 2h 27m 46s Examine (6) LINK/FILE BUG SNOOZE Builders this step failed on: WebKit Win7 (dbg) [2 out of the last 2 builds have failed] 41 tests failed: ...... too many results, data snipped.... ( Code Search ) editing/input/emacs-ctrl-o.html ( Code Search ) editing/input/linux_ltr_composition_underline.html ( Code Search ) editing/input/linux_rtl_composition_underline.html ( Code Search ) editing/pasteboard/emacs-cntl-y-001.html ( Code Search ) editing/pasteboard/emacs-ctrl-a-k-y.html ( Code Search ) editing/pasteboard/emacs-ctrl-k-with-move.html ( Code Search ) editing/pasteboard/emacs-ctrl-k-y-001.html ( Code Search ) editing/selection/linux_selection_color.html ( Code Search ) external/wpt/accelerometer/idlharness.https.html ( Code Search ) external/wpt/assumptions/canvas-background.html ( Code Search ) external/wpt/assumptions/html-elements.html ( Code Search ) on the builder: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Win7%20%28dbg%29/builds/9501 Unexpected Failures: * external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-18.html * virtual/service-worker-navigation-preload-disabled/webexposed/global-interface-listing.html
,
Apr 14 2017
Think I found the problem. CL out for review soon.
,
Apr 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/82f69457f005dab4ca99a1b61e8c26e4b8c558ff commit 82f69457f005dab4ca99a1b61e8c26e4b8c558ff Author: Sean McCullough <seanmccullough@chromium.org> Date: Fri Apr 14 21:14:33 2017 [som] Trust is_unexpected in test result logs. This was causing tests that were marked Skip/WontFix to be included in unexpected test failures, which led to alerts indicating way too many test failures. Bug:711733 Change-Id: I04fdffd6f273baa73d1a5633f88bba80a54e28e7 Reviewed-on: https://chromium-review.googlesource.com/478056 Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org> Commit-Queue: Sean McCullough <seanmccullough@chromium.org> [modify] https://crrev.com/82f69457f005dab4ca99a1b61e8c26e4b8c558ff/go/src/infra/monitoring/analyzer/step/test_step.go [modify] https://crrev.com/82f69457f005dab4ca99a1b61e8c26e4b8c558ff/go/src/infra/monitoring/analyzer/step/test_step_test.go
,
Apr 14 2017
Running the fix CL on that snapshot had this effect on the "webkit_tests failing on chromium.webkit/WebKit Linux Trusty (dbg)" alert:
Before (70ish tests indicated):
"reason": {
"test_names": [
"...... too many results, data snipped....",
"editing/input/emacs-ctrl-o.html",
"editing/pasteboard/emacs-cntl-y-001.html",
"editing/pasteboard/emacs-ctrl-a-k-y.html",
"editing/pasteboard/emacs-ctrl-k-with-move.html",
"editing/pasteboard/emacs-ctrl-k-y-001.html",
"external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm",
"external/wpt/accelerometer/idlharness.https.html",
"external/wpt/assumptions/canvas-background.html",
"external/wpt/assumptions/html-elements.html",
"external/wpt/assumptions/initial-color.html",
"external/wpt/assumptions/medium-font-size.html",
"external/wpt/assumptions/min-font-size.html",
"external/wpt/auxclick/auxclick_event-manual.html",
"external/wpt/clear-site-data/navigation.html",
"external/wpt/css-values/unset-value-storage.html",
"external/wpt/css/CSS2/linebox/inline-formatting-context-002.xht",
"external/wpt/css/CSS2/linebox/inline-formatting-context-003.xht",
"external/wpt/css/CSS2/linebox/inline-formatting-context-004.xht",
"external/wpt/css/CSS2/linebox/inline-formatting-context-005.xht",
"external/wpt/css/CSS2/linebox/inline-formatting-context-006.xht",
"external/wpt/css/CSS2/linebox/inline-formatting-context-007.xht",
"external/wpt/css/CSS2/linebox/inline-formatting-context-023.xht",
"external/wpt/css/CSS2/linebox/line-height-126.xht",
"external/wpt/css/CSS2/linebox/vertical-align-sub-001.xht",
"external/wpt/css/CSS2/linebox/vertical-align-super-001.xht",
"external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html",
"external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html",
"external/wpt/gyroscope/idlharness.https.html",
"external/wpt/html/obsolete/requirements-for-implementations/the-marquee-element-0/marquee-events.html",
"external/wpt/html/semantics/forms/constraints/tooShort-input-email-add-manual.html",
"external/wpt/html/semantics/forms/constraints/tooShort-input-password-add-manual.html",
"external/wpt/html/semantics/forms/constraints/tooShort-input-search-add-manual.html",
"external/wpt/html/semantics/forms/constraints/tooShort-input-tel-add-manual.html",
"external/wpt/html/semantics/forms/constraints/tooShort-input-text-add-manual.html",
"external/wpt/html/semantics/forms/constraints/tooShort-input-url-add-manual.html",
"external/wpt/html/semantics/forms/constraints/tooShort-textarea-add-manual.html",
"external/wpt/html/the-xhtml-syntax/parsing-xhtml-documents/xhtml-mathml-dtd-entity-support.htm",
"external/wpt/longtask-timing/longtask-attributes.html",
"external/wpt/longtask-timing/longtask-in-childiframe-crossorigin.html",
"external/wpt/longtask-timing/longtask-in-childiframe.html"
], ],
...
}
After (only three tests indicated):
"reason": {
"test_names": [
"external/wpt/XMLHttpRequest/event-readystatechange-loaded.htm",
"external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html",
"external/wpt/css/vendor-imports/mozilla/mozilla-central-reftests/variables/variable-declaration-17.html"
],
...
}
,
Apr 17 2017
|
|||
►
Sign in to add a comment |
|||
Comment 1 by seanmccullough@chromium.org
, Apr 14 2017Status: Available (was: Untriaged)