network_DefaultProfileCreation failing consistently on tricky |
|||||||||
Issue descriptionnetwork_DefaultProfileCreation is failing consistently in the bvt-cq stage with the following message: network_DefaultProfileCreation,Unhandled DBusException: org.freedesktop.DBus.Error.NoReply: Message did not receive a reply This is blocking the PFQ
,
Jun 13 2016
This appears to be due to a shill crash, e.g. from the failure in issue 618612: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=66214744 -> 'raw results logs' link -> network_DefaultProfileCreation/sysinfo/iteration.1/var/spool/crash/ https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/66214744-chromeos-test/chromeos4-row2-rack3-host12/network_DefaultProfileCreation/sysinfo/iteration.1/var/spool/crash/
,
Jun 13 2016
Lots of possible duplicates - there's one being tracked at the partner bug here: https://code.google.com/p/chrome-os-partner/issues/detail?id=52804
,
Jun 13 2016
Ugh. Those crashes do not have populated .dmp.txt files. From the first recent informational failure: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot-chrome-pfq-informational/builds/1083 Results: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/66459507-chromeos-test/chromeos4-row2-rack3-host18/network_DefaultProfileCreation/sysinfo/iteration.1/var/spool/crash/ We get call stacks but no symbols :(
,
Jun 13 2016
Re comment #3 - that issue was filed on Apr 27. These failures on tricky just started happening consistently since Jun 11 Also, it looks like we had some similar but slightly different failures starting June 8: FAIL: Unhandled DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name :1.116 was not provided by any .service files
,
Jun 13 2016
jdufault@ - Have you had a chance to look at this one? Go ahead and re-assign this to ihf@ once he gets in.
,
Jun 13 2016
,
Jun 14 2016
I have a revert open at [1], but I cannot verify the revert fixes the build just yet because the of the compile error, which should be fixed tomorrow. 1: https://codereview.chromium.org/2061093002/
,
Jun 14 2016
Another tricky failure instance in the latest build. FAIL: Unhandled DBusException: org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) jdufault: should this one be fixed too with your revert? I need a green PFQ by latest tomorrow morning. Any chance we can get this in quick?
,
Jun 14 2016
That looks like a similar symptom. Not sure if the revert will fix it quite yet, but there's a good chance.
,
Jun 14 2016
Re comment #5 - Hi stevenjb@, we are pretty certain we have seen this failure through April and May. See the thread I forwarded to you. It probably began manifesting consistently again since Jun 8/11.
,
Jun 14 2016
Hi jdufault@, stevenjb@, What is the rationale for reverting ejcaruso@'s commit? I might be missing something here, but the code looks unrelated to the bug. This test has been failing on tricky with in different ways since early March. Please see https://bugs.chromium.org/p/chromium/issues/detail?id=594336 which is assigned to me. ejcaruso@'s commit is dated April 19.
,
Jun 14 2016
We have seen crashes in Shill, which show up as failures in this test, on and off. This particular failure has been happening consistently only on tricky since that CL landed. We are not certain it is the cause, which is why we are testing the revert before landing it, but that CL does cause Shill to exercise new code, so it's not entirely a shot in the dark.
,
Jun 14 2016
The previous tryjob failed due to a compile failure in ToT, I started a new one prior to the suspected compile failure: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chrome-pfq-informational/builds/23
,
Jun 14 2016
The revert passed, so that CL does indeed appear to be triggering the shill crashes. I'm going to commit the revert so that we can get a green pfq build.
,
Jun 14 2016
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chrome-pfq-informational/builds/24 includes the change and also passed. What should we do about this?
,
Jun 14 2016
I don't know. network_DefaultProfileCreation passed in the last tricky run with the revert. That said, BuildPackages has been failing since yesterday morning, so it's possible that some other fix went in? Let's: 1. Let tricky-tot-chrome-pfq-informational cycle green 2. Do another tot trybot run with your CL 3. Commit your CL and watch tricky-tot-chrome-pfq-informational Let's do #3 first thing tomorrow if that is OK?
,
Jun 14 2016
OK, tricky-tot-chrome-pfq-informational is now green. I started a tryjob with: cbuildbot --remote tricky-tot-chrome-pfq-informational -G 2067913002 Tryjob is here: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chrome-pfq-informational/builds/25 If that cycles green, we can put 2067913002 into the CQ then keep an eye on tricky-tot-chrome-pfq-informational. Downgrading to P1 since the PFQ informational builders went green.
,
Jun 14 2016
Sounds good. Thanks.
,
Jun 15 2016
New failure for network_DefaultProfileCreation immediately after the reland. https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/tricky-tot- chrome-pfq-informational/builds/1152
,
Jun 15 2016
I have a fix up: https://chromium-review.googlesource.com/#/c/352773/ It's currently working through a buildbot run: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/chrome-pfq-informational/builds/26
,
Jun 17 2016
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/connectivity/shill/+/874b2bd05deb6506a09463f970e5271e17579c9e commit 874b2bd05deb6506a09463f970e5271e17579c9e Author: Eric Caruso <ejcaruso@chromium.org> Date: Wed Jun 15 19:54:07 2016 wifi: do not send random MAC request until supplicant appears If supplicant hasn't appeared yet, wait to send the request to enable random MAC until we have connected to it. BUG= chromium:619615 , chromium:620395 TEST=cbuildbot --remote tricky-tot-chrome-pfq-informational Change-Id: I5d9dd915774d9eeaf6e25deaa35c3db1ee90351c [modify] https://crrev.com/874b2bd05deb6506a09463f970e5271e17579c9e/wifi/wifi.cc
,
Jun 17 2016
,
Jul 1 2016
Bulk verified
,
Sep 6 2016
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/connectivity/shill/+/2a2d915bbe877fc9b9a999a2885afb5bade2c202 commit 2a2d915bbe877fc9b9a999a2885afb5bade2c202 Author: Eric Caruso <ejcaruso@chromium.org> Date: Wed Jun 15 19:54:07 2016 wifi: do not send random MAC request until supplicant appears If supplicant hasn't appeared yet, wait to send the request to enable random MAC until we have connected to it. BUG= chromium:619615 , chromium:620395 TEST=cbuildbot --remote tricky-tot-chrome-pfq-informational Change-Id: I5d9dd915774d9eeaf6e25deaa35c3db1ee90351c (cherry picked from commit 874b2bd05deb6506a09463f970e5271e17579c9e) [modify] https://crrev.com/2a2d915bbe877fc9b9a999a2885afb5bade2c202/wifi/wifi.cc
,
Sep 6 2016
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/connectivity/shill/+/2a2d915bbe877fc9b9a999a2885afb5bade2c202 commit 2a2d915bbe877fc9b9a999a2885afb5bade2c202 Author: Eric Caruso <ejcaruso@chromium.org> Date: Wed Jun 15 19:54:07 2016 wifi: do not send random MAC request until supplicant appears If supplicant hasn't appeared yet, wait to send the request to enable random MAC until we have connected to it. BUG= chromium:619615 , chromium:620395 TEST=cbuildbot --remote tricky-tot-chrome-pfq-informational Change-Id: I5d9dd915774d9eeaf6e25deaa35c3db1ee90351c (cherry picked from commit 874b2bd05deb6506a09463f970e5271e17579c9e) [modify] https://crrev.com/2a2d915bbe877fc9b9a999a2885afb5bade2c202/wifi/wifi.cc |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by steve...@chromium.org
, Jun 13 2016Cc: kirtika@chromium.org steve...@chromium.org
Owner: ihf@chromium.org
Status: Assigned (was: Untriaged)