As an example:

deqp/functional/gles3/pixelbufferobject.html is marked as fail on Mac AMD

https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py

However, on this bot, it's simply "passed"

https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Retina%20Release%20(AMD)

kbr: can you help finding an owner for this?

This is probably a side-effect of the switch to the typ test harness. I think it swallows all logging from successful test runs and I don't know whether that behavior can be overridden.

It should be easy to override this behavior by passing:
"-vv --passthrough" (maybe the 2nd flag is not needed, I can't recall)

typ has no understanding of the annotations that you seem to be using. We'd have to figure out what the old code did to detect an unexpected pass and see what sort of equivalent we might have.

er, by "have" I mean "need to implement".

#11 & 12: Dirk, seems like the way they did this in the past is just simply do logging.warning("expected to fail, but passed") & later do a text search for those.

I think this bug can be easily fixed by #10

okay, yeah, then -vv should be sufficient, and I don't think --passthrough is needed.

-vv produces too much output, but --passthrough provides what's desired with minimal addition to the logging. Here are four runs of:

./content/test/gpu/run_gpu_integration_test.py webgl_conformance --browser=canary --webgl-conformance-version=2.0.0 --test-filter=conformance_attribs --show-stdout --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc" -v

with the expectation:

self.Fail('conformance/attribs/gl-bindAttribLocation-aliasing.html', ['mac']);

added to webgl2_conformance_expectations.py .

The four runs are:

1) The above command
2) The above command with -vv instead of -v
3) The above command with --passthrough added
4) The above command with -vv --passthrough

I haven't figured out why --passthrough makes the number of lines in the output go up from 631 to 739, but -vv increases it to 39942, which is not workable.

Deleted: output.txt 133 KB

output.txt 133 KB View Download

Deleted: output-vv.txt 1.8 MB

output-vv.txt 1.8 MB View Download

Deleted: output-passthrough.txt 139 KB

output-passthrough.txt 139 KB View Download

Deleted: output-vv-passthrough.txt 3.5 MB

output-vv-passthrough.txt 3.5 MB View Download

https://codereview.chromium.org/2832383002/ up for review adding this command line arg.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5d00435578e14a407425e7bd19c7be8727254fe8

commit 5d00435578e14a407425e7bd19c7be8727254fe8
Author: kbr <kbr@chromium.org>
Date: Tue Apr 25 00:26:20 2017

Add --passthrough command line arg for Telemetry-based GPU tests.

This will re-enable display of warnings that a test was expected to
fail, but passed.

BUG= 710065 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2832383002
Cr-Commit-Position: refs/heads/master@{#466835}

[modify] https://crrev.com/5d00435578e14a407425e7bd19c7be8727254fe8/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/5d00435578e14a407425e7bd19c7be8727254fe8/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/5d00435578e14a407425e7bd19c7be8727254fe8/testing/buildbot/chromium.gpu.json
[modify] https://crrev.com/5d00435578e14a407425e7bd19c7be8727254fe8/testing/buildbot/client.v8.fyi.json

I think this is fixed. If someone can verify with recent runs on the waterfalls it would be greatly appreciated.

I verified, the "expected to fail, but passed" is back!

Thank you for fixing this.  This helps a lot for updating test expectations.

"The right way" to do this kind of updating test expectation should be through flakiness dashboard, I think. The dashboard has full knowledge of the test expectations & what actually happens 

In theory that would be the best way, but typ doesn't know about the test expectations and outputs "expected": "PASS" for all tests in its JSON output. I don't know how much work it would be to change this, but it's come up before.