New issue
Advanced search Search tips

Issue 700394 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2017
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: ----

Blocked on:
issue 701418
issue 701433
issue 701445



Sign in to add a comment

webkit_tests failing on chromium.webkit/WebKit Linux Trusty MSAN

Project Member Reported by fs...@chromium.org, Mar 10 2017

Issue description

webkit_tests failing on chromium.webkit/WebKit Linux Trusty MSAN

Builders failed on: 
- WebKit Linux Trusty MSAN: 
  https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20MSAN


I can't seem to find any correlation between those failing tests, even after pinging some devs. The iirfilter one (the one that started failing earlier is code that hasn't been touched in 2 weeks). Could some trooper look at the machine itself to see what's going on?

 

Comment 1 by fs...@chromium.org, Mar 10 2017

This is the bug I filled yesterday on this:
https://bugs.chromium.org/p/chromium/issues/detail?id=700139
Labels: Pri-0
I looked at the machine but doesn't see anything that I can recognize as "wrong". 

The only thing I can do at this moment is simply re-provision the slave, thus making it impossible to investigate further. If this is acceptable, I am ready to execute this in next 10 minutes.
Labels: -Pri-0 Pri-1
Reprovisioning slave161-c1 after go ahead form fserb@
slave161-c1 O> Deleting instance slave161-c1...
slave161-c1 O> Running: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances delete -q slave161-c1 --zone us-central1-c --project google.com:chromecompute
slave161-c1 O> Executing command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances delete -q slave161-c1 --zone us-central1-c --project google.com:chromecompute
slave161-c1 O> == Output from command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances delete -q slave161-c1 --zone us-central1-c --project google.com:chromecompute
slave161-c1 E> Deleted [https://www.googleapis.com/compute/v1/projects/google.com:chromecompute/zones/us-central1-c/instances/slave161-c1].
slave161-c1 O> Adding instance slave161-c1...
slave161-c1 O> Running: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances create slave161-c1 --machine-type n1-standard-8 --zone us-central1-c --project google.com:chromecompute --image chrome-trusty-16102500-3c5d0899e78 --image-project google.com:chromecompute --boot-disk-size 500GB --boot-disk-type pd-standard --address 104.154.8.172 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chrome-trusty-16102500-3c5d0899e78 --metadata-from-file cipd_deployments=/usr/local/google/home/tandrii/s/infra_internal/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json
slave161-c1 O> Executing command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances create slave161-c1 --machine-type n1-standard-8 --zone us-central1-c --project google.com:chromecompute --image chrome-trusty-16102500-3c5d0899e78 --image-project google.com:chromecompute --boot-disk-size 500GB --boot-disk-type pd-standard --address 104.154.8.172 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chrome-trusty-16102500-3c5d0899e78 --metadata-from-file cipd_deployments=/usr/local/google/home/tandrii/s/infra_internal/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json
slave161-c1 O> == Output from command: /usr/local/google/home/tandrii/google-cloud-sdk/bin/gcloud compute instances create slave161-c1 --machine-type n1-standard-8 --zone us-central1-c --project google.com:chromecompute --image chrome-trusty-16102500-3c5d0899e78 --image-project google.com:chromecompute --boot-disk-size 500GB --boot-disk-type pd-standard --address 104.154.8.172 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/userinfo.email --metadata image_name=chrome-trusty-16102500-3c5d0899e78 --metadata-from-file cipd_deployments=/usr/local/google/home/tandrii/s/infra_internal/infra_internal/ccompute/scripts/cipd/buildbot-trusty.json
slave161-c1 O> NAME         ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
slave161-c1 O> slave161-c1  us-central1-c  n1-standard-8               10.240.0.22  104.154.8.172  RUNNING
slave161-c1 E> Created [https://www.googleapis.com/compute/v1/projects/google.com:chromecompute/zones/us-central1-c/instances/slave161-c1].
Slave will be back soon. If this solves the problem, then it the bot got corrupted.
If this doesn't solve the problem, then there was some either infra or code change that is clearly reproducible.

Comment 7 by hinoka@chromium.org, Mar 10 2017

Bot is back on, it's still failing, so it is definitely not a bot issue.

Comment 8 by hinoka@chromium.org, Mar 11 2017

The failures are:

Regressions: Unexpected timeouts (5)
  external/wpt/dom/events/EventListener-invoke-legacy.html [ Timeout ]
  external/wpt/svg/interfaces.html [ Timeout ]
  webaudio/BiquadFilter/biquad-automation.html [ Timeout ]
  webaudio/IIRFilter/iirfilter.html [ Timeout ]

Comment 9 by hinoka@chromium.org, Mar 11 2017

Some more output here:

13:48:16.373 5148 worker/1 webaudio/IIRFilter/iirfilter.html output stderr lines:
13:48:16.373 5148   Xlib:  extension "RANDR" missing on display ":9".
13:48:16.375 22877 [11/11] webaudio/IIRFilter/iirfilter.html failed unexpectedly (test timed out)

Labels: -Infra-Troopers
-Trooper

Actually that looks like a red herring, since it's printed everywhere.

They look like truely flaky tests.  Not much a trooper can do at this point.
I filed a related bug for a subset of the tests that are crashing on MSAN:  bug 701170 , and uploaded a CL to add expectations: https://codereview.chromium.org/2751563003.
Owner: qyears...@chromium.org
Status: Started (was: Untriaged)
Alright, at this point, since  bug 701170  is fixed, we now have several timeout issues with particular tests:

These have timed out since they were first added (sometimes flakily, sometimes passing), but only on the MSAN bot:

external/wpt/dom/events/EventListener-invoke-legacy.html
external/wpt/svg/interfaces.html

This times out for as long as the flakiness dashboard shows (just a few days), sometimes flakily:

webaudio/IIRFilter/iirfilter.html
webaudio/AudioParam/audioparam-method-chaining.html

This one appears to have started timing out just recently:

webaudio/BiquadFilter/biquad-automation.html

This has timed out recently, but also flakily timed out before:

crypto/subtle/worker-subtle-crypto-concurrent.html

Next step: Adding flaky timeout + slow expectations in MSANExpectations, and perhaps filing individual bugs?
Blockedon: 701418
Blockedon: 701433
Blockedon: 701445
Status: Fixed (was: Started)
MSAN builder is green again for now:
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux%20Trusty%20MSAN

Sign in to add a comment