webkit_tests failing on chromium.webkit/WebKit Linux Trusty MSAN |
|||||||||
Issue descriptionwebkit_tests failing on chromium.webkit/WebKit Linux Trusty MSAN Builders failed on: - WebKit Linux Trusty MSAN: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20MSAN I can't seem to find any correlation between those failing tests, even after pinging some devs. The iirfilter one (the one that started failing earlier is code that hasn't been touched in 2 weeks). Could some trooper look at the machine itself to see what's going on?
,
Mar 10 2017
I looked at the machine but doesn't see anything that I can recognize as "wrong". The only thing I can do at this moment is simply re-provision the slave, thus making it impossible to investigate further. If this is acceptable, I am ready to execute this in next 10 minutes.
,
Mar 10 2017
,
Mar 10 2017
Reprovisioning slave161-c1 after go ahead form fserb@
,
Mar 10 2017
slave161-c1 O> Deleting instance slave161-c1... slave161-c1 O> Running: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances delete -q slave161-c1 --zone us-central1-c --project google.com:chromecompute slave161-c1 O> Executing command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances delete -q slave161-c1 --zone us-central1-c --project google.com:chromecompute slave161-c1 O> == Output from command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances delete -q slave161-c1 --zone us-central1-c --project google.com:chromecompute slave161-c1 E> Deleted [https://www.googleapis.com/compute/v1/projects/google.com:chromecompute/zones/us-central1-c/instances/slave161-c1]. slave161-c1 O> Adding instance slave161-c1... slave161-c1 O> Running: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances create slave161-c1 --machine-type n1-standard-8 --zone us-central1-c --project google.com:chromecompute --image chrome-trusty-16102500-3c5d0899e78 --image-project google.com:chromecompute --boot-disk-size 500GB --boot-disk-type pd-standard --address 104.154.8.172 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chrome-trusty-16102500-3c5d0899e78 --metadata-from-file cipd_deployments=/usr/local/google/home/tandrii/s/infra_internal/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json slave161-c1 O> Executing command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances create slave161-c1 --machine-type n1-standard-8 --zone us-central1-c --project google.com:chromecompute --image chrome-trusty-16102500-3c5d0899e78 --image-project google.com:chromecompute --boot-disk-size 500GB --boot-disk-type pd-standard --address 104.154.8.172 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chrome-trusty-16102500-3c5d0899e78 --metadata-from-file cipd_deployments=/usr/local/google/home/tandrii/s/infra_internal/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json slave161-c1 O> == Output from command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances create slave161-c1 --machine-type n1-standard-8 --zone us-central1-c --project google.com:chromecompute --image chrome-trusty-16102500-3c5d0899e78 --image-project google.com:chromecompute --boot-disk-size 500GB --boot-disk-type pd-standard --address 104.154.8.172 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chrome-trusty-16102500-3c5d0899e78 --metadata-from-file cipd_deployments=/usr/local/google/home/tandrii/s/infra_internal/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json slave161-c1 O> NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS slave161-c1 O> slave161-c1 us-central1-c n1-standard-8 10.240.0.22 104.154.8.172 RUNNING slave161-c1 E> Created [https://www.googleapis.com/compute/v1/projects/google.com:chromecompute/zones/us-central1-c/instances/slave161-c1].
,
Mar 10 2017
Slave will be back soon. If this solves the problem, then it the bot got corrupted. If this doesn't solve the problem, then there was some either infra or code change that is clearly reproducible.
,
Mar 10 2017
Bot is back on, it's still failing, so it is definitely not a bot issue.
,
Mar 11 2017
The failures are: Regressions: Unexpected timeouts (5) external/wpt/dom/events/EventListener-invoke-legacy.html [ Timeout ] external/wpt/svg/interfaces.html [ Timeout ] webaudio/BiquadFilter/biquad-automation.html [ Timeout ] webaudio/IIRFilter/iirfilter.html [ Timeout ]
,
Mar 11 2017
Some more output here: 13:48:16.373 5148 worker/1 webaudio/IIRFilter/iirfilter.html output stderr lines: 13:48:16.373 5148 Xlib: extension "RANDR" missing on display ":9". 13:48:16.375 22877 [11/11] webaudio/IIRFilter/iirfilter.html failed unexpectedly (test timed out)
,
Mar 11 2017
-Trooper Actually that looks like a red herring, since it's printed everywhere. They look like truely flaky tests. Not much a trooper can do at this point.
,
Mar 13 2017
I filed a related bug for a subset of the tests that are crashing on MSAN: bug 701170 , and uploaded a CL to add expectations: https://codereview.chromium.org/2751563003.
,
Mar 14 2017
Alright, at this point, since bug 701170 is fixed, we now have several timeout issues with particular tests: These have timed out since they were first added (sometimes flakily, sometimes passing), but only on the MSAN bot: external/wpt/dom/events/EventListener-invoke-legacy.html external/wpt/svg/interfaces.html This times out for as long as the flakiness dashboard shows (just a few days), sometimes flakily: webaudio/IIRFilter/iirfilter.html webaudio/AudioParam/audioparam-method-chaining.html This one appears to have started timing out just recently: webaudio/BiquadFilter/biquad-automation.html This has timed out recently, but also flakily timed out before: crypto/subtle/worker-subtle-crypto-concurrent.html Next step: Adding flaky timeout + slow expectations in MSANExpectations, and perhaps filing individual bugs?
,
Mar 14 2017
,
Mar 14 2017
,
Mar 14 2017
,
Mar 16 2017
MSAN builder is green again for now: https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20MSAN |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by fs...@chromium.org
, Mar 10 2017