New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users

Issue metadata

Status: Unconfirmed
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

canaries fail frequently as bvt-arc times out very often

Project Member Reported by ravisadineni@chromium.org, Feb 2 2018

Issue description

1. Bvt-arc times out quite often.  Here are some of the examples. 

2. Tests are not taking long enough to complete. I see that some of the server side tests even though scheduled never get started. Looks like an infra issue.

https://uberchromegw.corp.google.com/i/chromeos/builders/auron_paine-release/builds/1909
https://uberchromegw.corp.google.com/i/chromeos/builders/kefka-release/builds/1912
https://uberchromegw.corp.google.com/i/chromeos/builders/snappy-release/builds/1431

Many release builds are failing because of this from the last week.

 
Components: Infra>Client>ChromeOS

Comment 2 by jkop@chromium.org, Feb 2 2018

Owner: jkop@chromium.org
Status: Assigned (was: Untriaged)

Comment 3 by jkop@chromium.org, Feb 2 2018

It's a consistent set of tests that aren't being run. And they definitely are scheduled.

174091801	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted	
174091779	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted	
174091788	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted	
174091776	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted	



74084897	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted
174084902	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_ContainerSmokeTest	Build	Client	1 Completed	3 / 3
174084887	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted
174084892	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted
74084882	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted

Comment 4 by jkop@chromium.org, Feb 2 2018

Cc: shuqianz@chromium.org jrbarnette@chromium.org
Cc: martinroth@chromium.org akes...@chromium.org bmgordon@google.com shapiroc@chromium.org
 Issue 796275  has been merged into this issue.

Comment 6 by jkop@chromium.org, Feb 3 2018

Possibly related to crbug.com/808563; this may be downstream effects of overworked shards.

Comment 7 by jkop@chromium.org, Feb 5 2018

Owner: xixuan@chromium.org
Labels: -Pri-0 Pri-2
Cc: -martinroth@chromium.org
Owner: ----
Status: Unconfirmed (was: Assigned)
Why a suite gets aborted is hard to detect and correct due to it's caused by multiple factors:

1. We have 6 hwtest stages in release builder, which is very large.

2. Some of these HWTest stages share bvt DUTs. Once some tests fail and get retried, bvt DUTs will be occupied for long time, which delays other HWTest.

3. Sometimes, shard is full/devserver is full/mysql is slow/network is slow/... could cause a test being executed slower than usual, which delays HWTest stage.

So this bug is too general to be diagnosed accurately. It's a feature-level bug & I will leave it here.
Cc: norvez@chromium.org dgarr...@chromium.org
Labels: -Pri-2 Pri-1
Recent CQ also has this issue.

Builders failed on: 
- nyan_kitty-paladin: 
  https://luci-milo.appspot.com/buildbot/chromeos/nyan_kitty-paladin/4413
- hana-paladin: 
  https://luci-milo.appspot.com/buildbot/chromeos/hana-paladin/2358

Sign in to add a comment