New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 820487 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Mar 2018
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

reef-chrome-pfq failed bvt-arc HwTest

Project Member Reported by x...@chromium.org, Mar 9 2018

Issue description

It started failing from https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq/builds/2070

Most errors are like "Devtools target crashed", "Connection reset by peer", "command execution error". It seems the device kept crashing on these tests, there are a lot of dmp files in the test result folder, see https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/182101291-chromeos-test/chromeos6-row4-rack9-host8/debug/ as an example.

Assign to Arc constables for investigation. 
 

Comment 1 by x...@chromium.org, Mar 9 2018

Seems to me that there are two types of crashes. One is during arc setup, the callstack is:
 0  libc-2.23.so!raise [raise.c : 54 + 0x10]
 1  libc-2.23.so!abort [abort.c : 89 + 0xa]
 2  libbase-core-395517.so!base::debug::BreakDebugger() [debugger_posix.cc : 219 + 0x5]
 3  libbase-core-395517.so!logging::LogMessage::~LogMessage() [logging.cc : 755 + 0x5]
 4  arc-setup!arc::ArcSetup::GetSystemImageFingerprint() [arc_setup.cc : 1395 + 0x36]
 5  arc-setup!arc::ArcSetup::GenerateHostSideCodeInternal(base::FilePath const&) [arc_setup.cc : 780 + 0xb]
 6  arc-setup!arc::ArcSetup::GenerateHostSideCode(base::FilePath const&) [arc_setup.cc : 793 + 0x8]
 7  arc-setup!arc::ArcSetup::OnSetup(bool) [arc_setup.cc : 1769 + 0xb]
 8  arc-setup!arc::ArcSetup::Run() [arc_setup.cc : 1919 + 0x8]
 9  arc-setup!main [main.cc : 24 + 0x8]
10  libc-2.23.so!__libc_start_main [libc-start.c : 289 + 0x1a]
11  arc-setup!_start + 0x29
12  0x7ffe5cf6acc8

The other is Chrome crash, the callstack is 
0  chrome!network::(anonymous namespace)::SimpleURLLoaderImpl::StartRequest(network::mojom::URLLoaderFactory*) [simple_url_loader.cc : 1275 + 0x7]
 1  chrome!network::(anonymous namespace)::SimpleURLLoaderImpl::DownloadToStringOfUnboundedSizeUntilCrashAndDie(network::mojom::URLLoaderFactory*, base::OnceCallback<void (std::__1::unique_ptr<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::default_delete<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >)>) [simple_url_loader.cc : 1254 + 0xb]
 2  chrome!chromeos::ServicesCustomizationDocument::DoStartFileFetch() [customization_document.cc : 607 + 0xc]
 3  chrome!chromeos::DelayNetworkCall(base::TimeDelta, base::RepeatingCallback<void ()> const&) [callback.h : 124 + 0x3]
 4  chrome!chromeos::ServicesCustomizationDocument::StartFileFetch() [customization_document.cc : 571 + 0xb]
 5  chrome!chromeos::ServicesCustomizationDocument::StartFetching() [customization_document.cc : 557 + 0x8]
 6  chrome!chromeos::ServicesCustomizationDocument::EnsureCustomizationApplied() [customization_document.cc : 519 + 0x8]
 7  chrome!chromeos::DelayNetworkCall(base::TimeDelta, base::RepeatingCallback<void ()> const&) [callback.h : 124 + 0x3]
 8  chrome!base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) [callback.h : 95 + 0x3]

And the latter one has been fixed in  Issue 820302 . 

Comment 2 by x...@chromium.org, Mar 9 2018

I just merged the fix in  Issue 820302  back to today's Chrome branch (3366) and kicked a new Chrome PFQ run to see if this issue goes away.
Components: Platform>ARC
similar bug on android: https://b.corp.google.com/issues/74414697

Comment 5 by x...@chromium.org, Mar 9 2018

Note: after merging back the revert (fix) in  Issue 820302  into 3366, the latest reef-chrome-pfq build passed. So it seems to me that the two types of crashes were both caused by the same root cause. 
Status: Fixed (was: Assigned)
thanks. closing it as fixed. See comment #5

Comment 7 by glevin@chromium.org, Mar 12 2018

Status: Assigned (was: Fixed)
We got 3 green pfq runs out of the fix (https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq), and then the problems reappeared with https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq/builds/2076.  Superficially, at least, they appear to be the same symptoms as before: multiple HWTest [bvt-arc] tests failing for various reasons, like "command execution error", "Unhandled AutoservRunError: Timeout encountered:", "Connection reset by peer", "Android did not boot!", and "Devtools target crashed".

Comment 8 by glevin@chromium.org, Mar 12 2018

Cc: jhorwich@chromium.org victorhsieh@chromium.org
+this week's ARC++ cops
Seems like someone has a potential fix up for review: https://chromium-review.googlesource.com/c/chromiumos/platform2/+/959050
And to also persist the knowledge in the Android bug here, the root cause is a malformed fingerprint which was causing arc-setup to assert:

[0310/084722:ERROR:arc_setup_util.cc(963)] Did not find a value for product while expanding ro.bootimage.build.fingerprint=google/coral/{product}_cheets:7.1.1/R67-10476.0.0/4645268:user/release-keys
Owner: victorhsieh@chromium.org
assigned to PST constable.
Cc: jacoblin@chromium.org za...@chromium.org reve...@chromium.org
 Issue 821016  has been merged into this issue.
https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq/builds/2093 is green again.  I'll mark this "Fixed" again if we get another good run or two.
Owner: glevin@chromium.org
I'll let glevin monitor this :)
Status: Fixed (was: Assigned)
We've had 7 green runs since 2093, and only one red one (https://uberchromegw.corp.google.com/i/chromeos/builders/reef-chrome-pfq/builds/2100), and those errors look rather different, so I think we'll call this fixed.

Sign in to add a comment