New issue
Advanced search Search tips

Issue 917718 link

Starred by 2 users

Issue metadata

Status: Unconfirmed
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

[css-values] calc-with-time-angle-and-frequency-values test is highly unreliable, transition-delay testing causes side effects

Reported by goo...@gtalbot.org, Dec 24

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0

Steps to reproduce the problem:
The test in question is

https://chromium.googlesource.com/chromium/src/+/c825d655f6aaf73484f9d56e9012793f5b9668cc/third_party/WebKit/LayoutTests/css3/calc/calc-with-time-angle-and-frequency-values.html

which has been imported into:

http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-time-angle-and-frequency-values.html

Additional important reduced tests trying to demonstrate test unreliability:

With first test with a transition-delay of 1sec:

http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-angle-values-1s-GT.html

Chromium 71 reports no error!

- - - -

With first test with a transition-delay of 0sec: 

http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-angle-values-0s-1error-GT.html

now Chromium 71 reports 1 error

- - - -

With first test with a transition-delay of 0sec and 2 errors: 

http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-angle-values-0s-2errors-GT.html

Chromium 71 reports only 1 error.

What is the expected behavior?
Every error (failures, lack of implementations) in the test should be reported. 'transition-delay' testing should have no impact (no side effects) on subsequent calc() angle unit sub-tests.

What went wrong?
http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-angle-values-1s-GT.html

should report 1 test failure

while

http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-angle-values-0s-2errors-GT.html

should report 2 errors, not just 1 error.

Did this work before? N/A 

Chrome version: 71.0.3578.80  Channel: stable
OS Version: 
Flash Version: irrelevant

Each individual sub-test in that meta test

calc-with-time-angle-and-frequency-values.html

could be correct and could be passed by Chromium 71; right now, I do not know. But, as designed, if a browser has a faulty implementation or absence of implementation or several faulty or absent implementation of angles, then the meta test will not report all of the failures or will not report any failure! That meta-test, as designed, does not report reliable results or does not seem capable of reporting reliably sub-test failures. That meta-test (calc-with-time-angle-and-frequency-values) is not trustworthy with regards to each individual potential sub-test failures. And it seems to me that if 'transition-delay' is > 0 and if testing 'transition-delay' starts before testing calc() angle units, then sub-test failures are not reported.
 
I just checked the above tests with Chromium 73.0.3651.0 and I got the same results.
Proposed replacement
====================

http://www.gtalbot.org/BrowserBugsSection/CSS3Values/chromium-calc-with-time-angle-and-frequency-values-improv-GT.html

There are still a few issues with that test, like fractional issues (fraction of thousandth) with some sub-tests but at least the issues mentioned in this bug report are corrected and improved.
Labels: Needs-Triage-M71
Cc: phanindra.mandapaka@chromium.org
Components: Tests>Telemetry
Labels: Triaged-ET

Comment 5 by benhenry@google.com, Jan 16 (6 days ago)

Components: Test>Telemetry

Comment 6 by benhenry@google.com, Jan 16 (6 days ago)

Components: -Tests>Telemetry

Sign in to add a comment