New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users

Issue metadata

Status: Unconfirmed
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment
link

Issue 808563: canaries fail frequently as bvt-arc times out very often

Reported by ravisadineni@chromium.org, Feb 2 2018 Project Member

Issue description

1. Bvt-arc times out quite often.  Here are some of the examples. 

2. Tests are not taking long enough to complete. I see that some of the server side tests even though scheduled never get started. Looks like an infra issue.

https://uberchromegw.corp.google.com/i/chromeos/builders/auron_paine-release/builds/1909
https://uberchromegw.corp.google.com/i/chromeos/builders/kefka-release/builds/1912
https://uberchromegw.corp.google.com/i/chromeos/builders/snappy-release/builds/1431

Many release builds are failing because of this from the last week.
 

Comment 1 by ravisadineni@chromium.org, Feb 2 2018

Components: Infra>Client>ChromeOS

Comment 2 by jkop@chromium.org, Feb 2 2018

Owner: jkop@chromium.org
Status: Assigned (was: Untriaged)

Comment 3 by jkop@chromium.org, Feb 2 2018

It's a consistent set of tests that aren't being run. And they definitely are scheduled.

174091801	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted	
174091779	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted	
174091788	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted	
174091776	kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted	



74084897	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases	Build	Server	1 Aborted
174084902	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_ContainerSmokeTest	Build	Client	1 Completed	3 / 3
174084887	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases	Build	Server	1 Aborted
174084892	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases	Build	Server	1 Aborted
74084882	auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases	Build	Server	1 Aborted

Comment 4 by jkop@chromium.org, Feb 2 2018

Cc: shuqianz@chromium.org jrbarnette@chromium.org

Comment 5 by bmgordon@chromium.org, Feb 2 2018

Cc: martinroth@chromium.org akes...@chromium.org bmgordon@google.com shapiroc@chromium.org
 Issue 796275  has been merged into this issue.

Comment 6 by jkop@chromium.org, Feb 3 2018

Possibly related to crbug.com/808563; this may be downstream effects of overworked shards.

Comment 7 by jkop@chromium.org, Feb 5 2018

Owner: xixuan@chromium.org

Comment 8 by xixuan@chromium.org, Feb 5 2018

Labels: -Pri-0 Pri-2

Comment 9 by martinroth@chromium.org, Feb 5 2018

Cc: -martinroth@chromium.org

Comment 10 by xixuan@chromium.org, Feb 5 2018

Owner: ----
Status: Unconfirmed (was: Assigned)
Why a suite gets aborted is hard to detect and correct due to it's caused by multiple factors:

1. We have 6 hwtest stages in release builder, which is very large.

2. Some of these HWTest stages share bvt DUTs. Once some tests fail and get retried, bvt DUTs will be occupied for long time, which delays other HWTest.

3. Sometimes, shard is full/devserver is full/mysql is slow/network is slow/... could cause a test being executed slower than usual, which delays HWTest stage.

So this bug is too general to be diagnosed accurately. It's a feature-level bug & I will leave it here.

Comment 11 by waihong@chromium.org, Feb 21 2018

Cc: norvez@chromium.org dgarr...@chromium.org
Labels: -Pri-2 Pri-1
Recent CQ also has this issue.

Builders failed on: 
- nyan_kitty-paladin: 
  https://luci-milo.appspot.com/buildbot/chromeos/nyan_kitty-paladin/4413
- hana-paladin: 
  https://luci-milo.appspot.com/buildbot/chromeos/hana-paladin/2358

Sign in to add a comment