New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 798273 link

Starred by 4 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: ----

Blocked on:
issue 799182



Sign in to add a comment

HWTest bvt-arc aborted cheets_CTS_N* and cheets_GTS* tests stuck on "waiting for cache lock"

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Jan 2 2018

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of johnylin@google.com

Regression started on 1784 (ChromeOS version 10240.0.0)-
https://uberchromegw.corp.google.com/i/chromeos/builders/banon-release/builds/1784

From the host log of one cheets_GTS* test -
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/166859207-chromeos-test

After the step of querying "/etc/lsb-release", it takes forever to finish "cache lock", and timeout in the end.

Log:
12/25 05:21:15.730 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_VERSION=10248.0.0
12/25 05:21:15.730 DEBUG|             utils:0280| [stdout] CHROMEOS_AUSERVER=https://tools.google.com/service/update2
12/25 05:21:15.830 INFO |     tradefed_test:0268| Waiting for cache lock...
12/25 05:21:16.490 INFO |     tradefed_test:0268| Waiting for cache lock...
...
12/25 06:20:58.369 INFO |     tradefed_test:0268| Waiting for cache lock...
12/25 06:21:03.758 INFO |          autoserv:0685| Results placed in /usr/local/autotest/results/165220320-chromeos-test/chromeos6-row1-rack19-host7
12/25 06:21:03.758 DEBUG|          autoserv:0693| autoserv is running in drone chromeos-server100.mtv.corp.google.com.

 
Cc: frankhu@chromium.org sarthakkukreti@chromium.org
Components: Infra>Client>ChromeOS
Labels: -Pri-2 Pri-1
Owner: ihf@chromium.org
Status: Assigned (was: Available)
banon- 
https://uberchromegw.corp.google.com/i/chromeos/builders/banon-release/builds/1816

(Also tracked at: b.corp.google.com/issues/70539494)

Ilja, the above builds still break with the changes to timeout. PTAL. (Assigned to you since you are the owner of the original bug).

Comment 5 by ihf@chromium.org, Jan 3 2018

Labels: OS-Chrome
On cs100 and 104 there was an old lock from December 23/29th. I deleted them. It should have been broken on its own, but jobs kept getting aborted before breaking.

I may have missed some servers. But basically the self repair in case CTS jobs get killed while holding the lock isn't working properly. I will change the logic and take the age of the lock into account.
Project Member

Comment 6 by bugdroid1@chromium.org, Jan 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fee326e2b0b30910d7c684ddee736829da04549a

commit fee326e2b0b30910d7c684ddee736829da04549a
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Thu Jan 04 04:56:08 2018

tradefed_test: move increment.

Fixes bug. Right now we don't increment at all.

BUG=chromium:798273
TEST=Inspection.

Change-Id: Ic010410df849947ccb321c618b3c76ff96dd2deb
Reviewed-on: https://chromium-review.googlesource.com/849758
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/fee326e2b0b30910d7c684ddee736829da04549a/server/cros/tradefed_test.py

Project Member

Comment 7 by bugdroid1@chromium.org, Jan 4 2018

Labels: merge-merged-release-R64-10176.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f5d976d47b1f84108a2a6a90aa833115b11374da

commit f5d976d47b1f84108a2a6a90aa833115b11374da
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Thu Jan 04 04:56:27 2018

tradefed_test: move increment.

Fixes bug. Right now we don't increment at all.

BUG=chromium:798273
TEST=Inspection.

Change-Id: Ic010410df849947ccb321c618b3c76ff96dd2deb
Reviewed-on: https://chromium-review.googlesource.com/849758
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
(cherry picked from commit fee326e2b0b30910d7c684ddee736829da04549a)
Reviewed-on: https://chromium-review.googlesource.com/849514

[modify] https://crrev.com/f5d976d47b1f84108a2a6a90aa833115b11374da/server/cros/tradefed_test.py

Project Member

Comment 8 by bugdroid1@chromium.org, Jan 4 2018

Labels: merge-merged-release-R63-10032.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e5d2bc273631fc0d897a6a7c79a31f3ed6ff1620

commit e5d2bc273631fc0d897a6a7c79a31f3ed6ff1620
Author: Ilja H. Friedel <ihf@chromium.org>
Date: Thu Jan 04 04:56:44 2018

tradefed_test: move increment.

Fixes bug. Right now we don't increment at all.

BUG=chromium:798273
TEST=Inspection.

Change-Id: Ic010410df849947ccb321c618b3c76ff96dd2deb
Reviewed-on: https://chromium-review.googlesource.com/849758
Tested-by: Ilja H. Friedel <ihf@chromium.org>
Trybot-Ready: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
(cherry picked from commit fee326e2b0b30910d7c684ddee736829da04549a)
Reviewed-on: https://chromium-review.googlesource.com/849514
(cherry picked from commit f5d976d47b1f84108a2a6a90aa833115b11374da)
Reviewed-on: https://chromium-review.googlesource.com/849515

[modify] https://crrev.com/e5d2bc273631fc0d897a6a7c79a31f3ed6ff1620/server/cros/tradefed_test.py

Comment 9 by ihf@chromium.org, Jan 4 2018

I think this should clean up the bad server state, but will check tomorrow.

Comment 10 by ihf@chromium.org, Jan 4 2018

Many of the builders turned green.

Comment 11 by ihf@chromium.org, Jan 4 2018

Issue 799033 has been merged into this issue.

Comment 12 by ihf@chromium.org, Jan 4 2018

I am going to nuke all old jobs so we only get the fixed ones in the queue.
Blockedon: 799182
 Issue 798272  has been merged into this issue.
Cc: khmel@chromium.org ihf@chromium.org kinaba@chromium.org warx@chromium.org malaykeshav@chromium.org
 Issue 804849  has been merged into this issue.

Sign in to add a comment