canaries fail frequently as bvt-arc times out very often |
|||||||||
Issue description1. Bvt-arc times out quite often. Here are some of the examples. 2. Tests are not taking long enough to complete. I see that some of the server side tests even though scheduled never get started. Looks like an infra issue. https://uberchromegw.corp.google.com/i/chromeos/builders/auron_paine-release/builds/1909 https://uberchromegw.corp.google.com/i/chromeos/builders/kefka-release/builds/1912 https://uberchromegw.corp.google.com/i/chromeos/builders/snappy-release/builds/1431 Many release builds are failing because of this from the last week.
,
Feb 2 2018
,
Feb 2 2018
It's a consistent set of tests that aren't being run. And they definitely are scheduled. 174091801 kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases Build Server 1 Aborted 174091779 kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases Build Server 1 Aborted 174091788 kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases Build Server 1 Aborted 174091776 kefka-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases Build Server 1 Aborted 74084897 auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAppUsageHostTestCases Build Server 1 Aborted 174084902 auron_paine-release/R66-10367.0.0/bvt-arc/cheets_ContainerSmokeTest Build Client 1 Completed 3 / 3 174084887 auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccountManagerTestCases Build Server 1 Aborted 174084892 auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsDramTestCases Build Server 1 Aborted 74084882 auron_paine-release/R66-10367.0.0/bvt-arc/cheets_CTS_N.7.1_r13.x86.CtsAccelerationTestCases Build Server 1 Aborted
,
Feb 2 2018
,
Feb 2 2018
Issue 796275 has been merged into this issue.
,
Feb 3 2018
Possibly related to crbug.com/808563; this may be downstream effects of overworked shards.
,
Feb 5 2018
,
Feb 5 2018
,
Feb 5 2018
,
Feb 5 2018
Why a suite gets aborted is hard to detect and correct due to it's caused by multiple factors: 1. We have 6 hwtest stages in release builder, which is very large. 2. Some of these HWTest stages share bvt DUTs. Once some tests fail and get retried, bvt DUTs will be occupied for long time, which delays other HWTest. 3. Sometimes, shard is full/devserver is full/mysql is slow/network is slow/... could cause a test being executed slower than usual, which delays HWTest stage. So this bug is too general to be diagnosed accurately. It's a feature-level bug & I will leave it here.
,
Feb 21 2018
Recent CQ also has this issue. Builders failed on: - nyan_kitty-paladin: https://luci-milo.appspot.com/buildbot/chromeos/nyan_kitty-paladin/4413 - hana-paladin: https://luci-milo.appspot.com/buildbot/chromeos/hana-paladin/2358 |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by ravisadineni@chromium.org
, Feb 2 2018