New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users

Issue metadata

Status: Unconfirmed
Owner: ----
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Sign in to add a comment

Issue 808563: canaries fail frequently as bvt-arc times out very often

Reported by, Feb 2 2018 Project Member

Issue description

1. Bvt-arc times out quite often.  Here are some of the examples. 

2. Tests are not taking long enough to complete. I see that some of the server side tests even though scheduled never get started. Looks like an infra issue.

Many release builds are failing because of this from the last week.

Comment 1 by, Feb 2 2018

Components: Infra>Client>ChromeOS

Comment 2 by, Feb 2 2018

Status: Assigned (was: Untriaged)

Comment 3 by, Feb 2 2018

It's a consistent set of tests that aren't being run. And they definitely are scheduled.

174091801	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted	
174091779	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted	
174091788	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted	
174091776	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted	

74084897	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted
174084902	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_ContainerSmokeTest	Build	Client	1 Completed	3 / 3
174084887	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted
174084892	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted
74084882	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted

Comment 4 by, Feb 2 2018


Comment 5 by, Feb 2 2018

 Issue 796275  has been merged into this issue.

Comment 6 by, Feb 3 2018

Possibly related to; this may be downstream effects of overworked shards.

Comment 7 by, Feb 5 2018


Comment 8 by, Feb 5 2018

Labels: -Pri-0 Pri-2

Comment 9 by, Feb 5 2018


Comment 10 by, Feb 5 2018

Owner: ----
Status: Unconfirmed (was: Assigned)
Why a suite gets aborted is hard to detect and correct due to it's caused by multiple factors:

1. We have 6 hwtest stages in release builder, which is very large.

2. Some of these HWTest stages share bvt DUTs. Once some tests fail and get retried, bvt DUTs will be occupied for long time, which delays other HWTest.

3. Sometimes, shard is full/devserver is full/mysql is slow/network is slow/... could cause a test being executed slower than usual, which delays HWTest stage.

So this bug is too general to be diagnosed accurately. It's a feature-level bug & I will leave it here.

Comment 11 by, Feb 21 2018

Labels: -Pri-2 Pri-1
Recent CQ also has this issue.

Builders failed on: 
- nyan_kitty-paladin:
- hana-paladin:

Sign in to add a comment