New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: Unconfirmed
Owner: ----
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Sign in to add a comment

canaries fail frequently as bvt-arc times out very often

Project Member Reported by, Feb 2 Back to list

Issue description

1. Bvt-arc times out quite often.  Here are some of the examples. 

2. Tests are not taking long enough to complete. I see that some of the server side tests even though scheduled never get started. Looks like an infra issue.

Many release builds are failing because of this from the last week.

Components: Infra>Client>ChromeOS
Status: Assigned
It's a consistent set of tests that aren't being run. And they definitely are scheduled.

174091801	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted	
174091779	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted	
174091788	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted	
174091776	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted	

74084897	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted
174084902	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_ContainerSmokeTest	Build	Client	1 Completed	3 / 3
174084887	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted
174084892	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted
74084882	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted
 Issue 796275  has been merged into this issue.
Possibly related to; this may be downstream effects of overworked shards.
Labels: -Pri-0 Pri-2
Owner: ----
Status: Unconfirmed
Why a suite gets aborted is hard to detect and correct due to it's caused by multiple factors:

1. We have 6 hwtest stages in release builder, which is very large.

2. Some of these HWTest stages share bvt DUTs. Once some tests fail and get retried, bvt DUTs will be occupied for long time, which delays other HWTest.

3. Sometimes, shard is full/devserver is full/mysql is slow/network is slow/... could cause a test being executed slower than usual, which delays HWTest stage.

So this bug is too general to be diagnosed accurately. It's a feature-level bug & I will leave it here.
Labels: -Pri-2 Pri-1
Recent CQ also has this issue.

Builders failed on: 
- nyan_kitty-paladin:
- hana-paladin:

Sign in to add a comment