HWTest bvt-arc aborted cheets_CTS_N* and cheets_GTS* tests stuck on "waiting for cache lock" |
|||||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of johnylin@google.com Regression started on 1784 (ChromeOS version 10240.0.0)- https://uberchromegw.corp.google.com/i/chromeos/builders/banon-release/builds/1784 From the host log of one cheets_GTS* test - https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/166859207-chromeos-test After the step of querying "/etc/lsb-release", it takes forever to finish "cache lock", and timeout in the end. Log: 12/25 05:21:15.730 DEBUG| utils:0280| [stdout] CHROMEOS_RELEASE_VERSION=10248.0.0 12/25 05:21:15.730 DEBUG| utils:0280| [stdout] CHROMEOS_AUSERVER=https://tools.google.com/service/update2 12/25 05:21:15.830 INFO | tradefed_test:0268| Waiting for cache lock... 12/25 05:21:16.490 INFO | tradefed_test:0268| Waiting for cache lock... ... 12/25 06:20:58.369 INFO | tradefed_test:0268| Waiting for cache lock... 12/25 06:21:03.758 INFO | autoserv:0685| Results placed in /usr/local/autotest/results/165220320-chromeos-test/chromeos6-row1-rack19-host7 12/25 06:21:03.758 DEBUG| autoserv:0693| autoserv is running in drone chromeos-server100.mtv.corp.google.com.
,
Jan 2 2018
After there are more boards having this. edgar- https://uberchromegw.corp.google.com/i/chromeos/builders/edgar-release/builds/1818 eve- https://uberchromegw.corp.google.com/i/chromeos/builders/eve-release/builds/1239 pyro- https://uberchromegw.corp.google.com/i/chromeos/builders/pyro-release/builds/1333 reks- https://uberchromegw.corp.google.com/i/chromeos/builders/reks-release/builds/1813 relm- https://uberchromegw.corp.google.com/i/chromeos/builders/relm-release/builds/1816 ultima- https://uberchromegw.corp.google.com/i/chromeos/builders/ultima-release/builds/1807 veyron_mighty- https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_mighty-release/builds/1816 veyron_speedy- https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_speedy-release/builds/1813 wizpig- https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-release/builds/1822
,
Jan 3 2018
banon- https://uberchromegw.corp.google.com/i/chromeos/builders/banon-release/builds/1816 (Also tracked at: b.corp.google.com/issues/70539494) Ilja, the above builds still break with the changes to timeout. PTAL. (Assigned to you since you are the owner of the original bug).
,
Jan 3 2018
On cs100 and 104 there was an old lock from December 23/29th. I deleted them. It should have been broken on its own, but jobs kept getting aborted before breaking. I may have missed some servers. But basically the self repair in case CTS jobs get killed while holding the lock isn't working properly. I will change the logic and take the age of the lock into account.
,
Jan 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fee326e2b0b30910d7c684ddee736829da04549a commit fee326e2b0b30910d7c684ddee736829da04549a Author: Ilja H. Friedel <ihf@chromium.org> Date: Thu Jan 04 04:56:08 2018 tradefed_test: move increment. Fixes bug. Right now we don't increment at all. BUG= chromium:798273 TEST=Inspection. Change-Id: Ic010410df849947ccb321c618b3c76ff96dd2deb Reviewed-on: https://chromium-review.googlesource.com/849758 Tested-by: Ilja H. Friedel <ihf@chromium.org> Trybot-Ready: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/fee326e2b0b30910d7c684ddee736829da04549a/server/cros/tradefed_test.py
,
Jan 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f5d976d47b1f84108a2a6a90aa833115b11374da commit f5d976d47b1f84108a2a6a90aa833115b11374da Author: Ilja H. Friedel <ihf@chromium.org> Date: Thu Jan 04 04:56:27 2018 tradefed_test: move increment. Fixes bug. Right now we don't increment at all. BUG= chromium:798273 TEST=Inspection. Change-Id: Ic010410df849947ccb321c618b3c76ff96dd2deb Reviewed-on: https://chromium-review.googlesource.com/849758 Tested-by: Ilja H. Friedel <ihf@chromium.org> Trybot-Ready: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> (cherry picked from commit fee326e2b0b30910d7c684ddee736829da04549a) Reviewed-on: https://chromium-review.googlesource.com/849514 [modify] https://crrev.com/f5d976d47b1f84108a2a6a90aa833115b11374da/server/cros/tradefed_test.py
,
Jan 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e5d2bc273631fc0d897a6a7c79a31f3ed6ff1620 commit e5d2bc273631fc0d897a6a7c79a31f3ed6ff1620 Author: Ilja H. Friedel <ihf@chromium.org> Date: Thu Jan 04 04:56:44 2018 tradefed_test: move increment. Fixes bug. Right now we don't increment at all. BUG= chromium:798273 TEST=Inspection. Change-Id: Ic010410df849947ccb321c618b3c76ff96dd2deb Reviewed-on: https://chromium-review.googlesource.com/849758 Tested-by: Ilja H. Friedel <ihf@chromium.org> Trybot-Ready: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> (cherry picked from commit fee326e2b0b30910d7c684ddee736829da04549a) Reviewed-on: https://chromium-review.googlesource.com/849514 (cherry picked from commit f5d976d47b1f84108a2a6a90aa833115b11374da) Reviewed-on: https://chromium-review.googlesource.com/849515 [modify] https://crrev.com/e5d2bc273631fc0d897a6a7c79a31f3ed6ff1620/server/cros/tradefed_test.py
,
Jan 4 2018
I think this should clean up the bad server state, but will check tomorrow.
,
Jan 4 2018
Many of the builders turned green.
,
Jan 4 2018
Issue 799033 has been merged into this issue.
,
Jan 4 2018
I am going to nuke all old jobs so we only get the fixed ones in the queue.
,
Jan 4 2018
,
Jan 26 2018
Issue 798272 has been merged into this issue.
,
Jan 26 2018
Issue 804849 has been merged into this issue.
,
Dec 27
Looks like this was fixed a year ago, so closing this. Please feel free to reopen if this is not the case. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by johnylin@chromium.org
, Jan 2 2018Components: Infra>Client>ChromeOS