Android boot test fails for strago boards
Reported by
jrbarnette@chromium.org,
Jul 19 2017
|
|||||||
Issue description
The wizpig and sentry paladins recently started failing with
complaints on two tests:
[Test-Logs]: cheets_StartAndroid.stress: FAIL: Android did not boot!
[Test-Logs]: cheets_MediaPlayerVideoHWDecodeUsed: retry_count: 2, FAIL: Android did not boot!
The first recorded CQ failure is this run:
https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/15439
That run may contain the changes that introduced the problem. It's
also possible than an Android- or Chrome PFQ run introduced the problem.
,
Jul 19 2017
,
Jul 19 2017
I'll take a look in ~30 mins.
,
Jul 19 2017
The build linked above is a successful run, and it seems to be green for ~15 hours now, am I missing something? FWIW we are seeing ToT ARC failures in EGL startup which we think might also be related to these multilib changes
,
Jul 19 2017
,
Jul 19 2017
I guess this is the wizpig failure: https://uberchromegw.corp.google.com/i/chromeos/builders/wizpig-paladin/builds/283 looking
,
Jul 19 2017
yes, it does appear it's the same issue we're running into because Surfaceflinger is continually crashing. Looks like it passed the CQ because the failures that would have been caught on wizpig were exceptions but are experimental builders anyway
,
Jul 19 2017
over to Josh who is attempting a minimal revert of the above blame list to fix
,
Jul 19 2017
We're also seeing failures in the PFQ, with 3 failing builds each on cyan, caroline, reef, veyron_minnie, starting yesterday night. I'm not entirely sure if it's the same issue: https://bugs.chromium.org/p/chromium/issues/detail?id=746573
,
Jul 19 2017
> We're also seeing failures in the PFQ, with 3 failing builds > each on cyan, caroline, reef, veyron_minnie, starting yesterday > night. I'm not entirely sure if it's the same issue: > https://bugs.chromium.org/p/chromium/issues/detail?id=746573 The cyan and caroline failures don't match the symptom. The reef and veyron_minnie changes _do_match the symptom. However: * The Chrome PFQ failures pre-date the CQ failures, so the timing is wrong. * The PFQ failures aren't showing up on the veyron_minnie or reef canary builders, or in the CQ. So, the Chrome PFQ failures are almost certainly caused by a Chrome bug.
,
Jul 19 2017
> The build linked above is a successful run, and it seems to be green for ~15 hours now, am I missing something? The CQ run passed because the strago builders are all still experimental, so the paladin failures were ignored.
,
Jul 19 2017
,
Jul 19 2017
Sent https://chromium-review.googlesource.com/q/topic:%22revert-arc-multilib%22+(status:open%20OR%20status:merged) to CQ to revert multilib-related patches until we properly sort out a fix.
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/04ffd34753ed1da48d22f7afe3a3b37822e979df commit 04ffd34753ed1da48d22f7afe3a3b37822e979df Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:58:24 2017 Revert "chromeos-base/arc-build-nyc: Add support for multilib" This reverts commit b4d8ff0fc902787624b8d9e864e95e177ad19b33. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device CQ-DEPEND=CL:578671 Change-Id: I07ed27e76613c6ad9141232402af6ed443636570 Reviewed-on: https://chromium-review.googlesource.com/578564 Tested-by: Josh Horwich <jhorwich@chromium.org> Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> [rename] https://crrev.com/04ffd34753ed1da48d22f7afe3a3b37822e979df/chromeos-base/arc-build-nyc/arc-build-nyc-0.0.1-r5.ebuild [modify] https://crrev.com/04ffd34753ed1da48d22f7afe3a3b37822e979df/chromeos-base/arc-build-nyc/arc-build-nyc-0.0.1.ebuild
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/be32fc06ea0c4696723b0fba2f1c0f252871ecf8 commit be32fc06ea0c4696723b0fba2f1c0f252871ecf8 Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:58:35 2017 Revert "profiles: arch/arm: Sync up make.defaults with upstream Gentoo" This reverts commit b9a92df6b963f171dbab7b26dcce9d13804cccf5. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: Ie78fb2fec0b7570f7958d65d1bb6e284630d6ad0 Reviewed-on: https://chromium-review.googlesource.com/578565 Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/be32fc06ea0c4696723b0fba2f1c0f252871ecf8/profiles/arch/arm/make.defaults
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/517ce487ecf2a173c5b6dd33338a432f5ac77e34 commit 517ce487ecf2a173c5b6dd33338a432f5ac77e34 Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:58:53 2017 Revert "profiles: Enable multilib infrastructure for amd64 chromeos profile" This reverts commit 1b244c91bc938eeec08bf9f2c45be7ed9d826ad5. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: Ia5986fec4be43ae7db031f626d3ffa8407eec582 Reviewed-on: https://chromium-review.googlesource.com/578566 Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/517ce487ecf2a173c5b6dd33338a432f5ac77e34/profiles/targets/chromeos/make.defaults [modify] https://crrev.com/517ce487ecf2a173c5b6dd33338a432f5ac77e34/profiles/default/linux/amd64/10.0/chromeos/use.force [modify] https://crrev.com/517ce487ecf2a173c5b6dd33338a432f5ac77e34/profiles/default/linux/amd64/10.0/chromeos/make.defaults [modify] https://crrev.com/517ce487ecf2a173c5b6dd33338a432f5ac77e34/profiles/targets/sdk/make.defaults [modify] https://crrev.com/517ce487ecf2a173c5b6dd33338a432f5ac77e34/profiles/default/linux/amd64/10.0/chromeos/parent
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/afd7bc6c683725f551c77de24ac3b2ae9b537d9c commit afd7bc6c683725f551c77de24ac3b2ae9b537d9c Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:59:03 2017 Revert "eclass/arc-build: Enable multilib support" This reverts commit a9e93c34622bbd833e5e8c59afc853bf0ef6e632. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: I50ada5bd457f47b2252161637fa23c6af8ebb5a4 Reviewed-on: https://chromium-review.googlesource.com/578667 Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/afd7bc6c683725f551c77de24ac3b2ae9b537d9c/eclass/arc-build.eclass
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/8b925adfda43ea58e1c8931587e4c587e34f33a3 commit 8b925adfda43ea58e1c8931587e4c587e34f33a3 Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:59:12 2017 Revert "x11-libs/arc-libdrm: Enable support for multilib" This reverts commit d1159dc9b5bef3d9a55e076d5c199a09eaadbf0b. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: I300e0927f0b63ef75439fe3fff0625ff4bd2d24f Reviewed-on: https://chromium-review.googlesource.com/578668 Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/8b925adfda43ea58e1c8931587e4c587e34f33a3/x11-libs/arc-libdrm/arc-libdrm-9999.ebuild
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/f6e49d6ace487dcb7beebb506c41f1d1070067d9 commit f6e49d6ace487dcb7beebb506c41f1d1070067d9 Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:59:20 2017 Revert "media-libs/arc-mesa: Add support for multilib" This reverts commit 502a21576d2ee14e88ade2afab275e2f03da35d8. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: I3e46de994549e266a8f057f11c01a5066063e673 Reviewed-on: https://chromium-review.googlesource.com/578669 Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/f6e49d6ace487dcb7beebb506c41f1d1070067d9/media-libs/arc-mesa/arc-mesa-9999.ebuild
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/9c3a15d8a2e0709db578a7779f78d1bbc0b2b2ae commit 9c3a15d8a2e0709db578a7779f78d1bbc0b2b2ae Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:59:30 2017 Revert "media-libs/arc-cros-gralloc: Enable support for multilib" This reverts commit ca06c13f221e182f9f23c312d906c8149a4e829f. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: Ib03aaa884ac49dfa9a188910dc61e7fa76e349a4 Reviewed-on: https://chromium-review.googlesource.com/578670 Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> Tested-by: Josh Horwich <jhorwich@chromium.org> [modify] https://crrev.com/9c3a15d8a2e0709db578a7779f78d1bbc0b2b2ae/media-libs/arc-cros-gralloc/arc-cros-gralloc-9999.ebuild
,
Jul 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/006cf3616ae609c0b632d15fd5feae78c9dbbf97 commit 006cf3616ae609c0b632d15fd5feae78c9dbbf97 Author: Josh Horwich <jhorwich@chromium.org> Date: Thu Jul 20 00:59:49 2017 Revert "x11-libs/arc-libpciaccess: Add support for multilib" This reverts commit b3ae6619f99498f81b21b59c374546f3f4a3e79b. Reason for revert: Unable to start surfaceflinger - /vendor/lib64/egl now has the libraries when trying to run 32-bit ABI Android. BUG= chromium:746548 TEST=Start x86 Android on a 64-bit Intel device Change-Id: I5f96c561461236df9d664c8cb8f580526196d0da Reviewed-on: https://chromium-review.googlesource.com/578671 Tested-by: Josh Horwich <jhorwich@chromium.org> Commit-Queue: Josh Horwich <jhorwich@chromium.org> Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org> [rename] https://crrev.com/006cf3616ae609c0b632d15fd5feae78c9dbbf97/x11-libs/arc-libpciaccess/arc-libpciaccess-0.12.902-r4.ebuild
,
Jul 20 2017
This should be fixed. 9761.0.0 picked the reverts. https://uberchromegw.corp.google.com/i/chromeos/builders/sentry-paladin/builds/287 : failed. 9760.0.0 https://uberchromegw.corp.google.com/i/chromeos/builders/sentry-paladin/builds/288 : passed. 9761.0.0 https://luci-milo.appspot.com/buildbot/chromeos/wizpig-release/1328 : failed. 9760.0.0 https://luci-milo.appspot.com/buildbot/chromeos/wizpig-release/1329 : passed. 9761.0.0 |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by jrbarnette@chromium.org
, Jul 19 2017Checking the canaries, wizpig, sentry, and edgar are not yet exhibiting the failure. If the problem were introduced through either PFQ, I'd have expected it to show up no later than 9759.0.0, but it didn't The CLs in the failed CQ run landed in the 9760.0.0 run, so it's likely that the bug made it in to the tree via the CQ. Here's the relevant blamelist: chromiumos-overlay | tfiga | 488744 | fails:0(15) | eclass/arc-build: Enable multilib support chromiumos-overlay | tfiga | 488743 | fails:0(8) | chromeos-base/arc-build-nyc: Add support for multilib chromiumos-overlay | tfiga | 488747 | fails:0(8) | x11-libs/arc-libdrm: Enable support for multilib chromiumos-overlay | tfiga | 488748 | fails:0(8) | media-libs/arc-mesa: Add support for multilib chromiumos-overlay | tfiga | 488749 | fails:0(8) | media-libs/arc-cros-gralloc: Enable support for multilib chromiumos-overlay | tfiga | 485001 | fails:0(7) | profiles: Enable multilib infrastructure for amd64 chromeos profile chromiumos-overlay | tfiga | 488746 | fails:0(7) | x11-libs/arc-libpciaccess: Add support for multilib platform2 | kitching | 566755 | fails:1(2) | hammerd: fix interface number bug ec-private-cr52 | domrizzo | *405253 | fails:0(2) | CR52: Generate k for ECDSA sign via HMAC-DRBG platform2 | kitching | 571342 | fails:1 | hammerd: fix segmentation faults arc-camera | henryhsu | 575125 | common: Add include file for DISALLOW_COPY_AND_ASSIGN chromiumos-overlay | tfiga | 568338 | profiles: arch/arm: Sync up make.defaults with upstream Gentoo ec-private-cr52 | domrizzo | *413153 | CR52: Add miscellaneous tests; pop count and fatal FIPS error. ec-private-cr52 | mschilder | *413988 | CR52: aes MCT driver kernel | naveen.m-AT-intel.com | 532756 | BACKPORT: ASoC: Intel: Skylake: Fix module load when module size > DMA buffer... +tfiga@ as the most likely owner of the root cause.