bisect showed it is https://chromium-review.googlesource.com/c/chromium/src/+/1379066

Tom, do you know what's the problem? If we cannot fix it quickly, can you revert the change or filter out chromeos builds?  This is blocking many engineers' work flow.

Building with _LIBCPP_ABI_UNSTABLE is intentional for Linux ChromeOS builds.  I couldn't repro because I couldn't figure out how to load the assistant, but I noticed libassistant.so gets built in a separate build system, and it has libc++ statically linked.  _LIBCPP_ABI_UNSTABLE changes the layout of std::string to apparently be more efficient for small string optimizations.  If any std::string's get passed over the shared library boundary, they'll need to be changed to C strings.  C++ objects shouldn't be passed across .so boundaries unless they were built in the same configuration.

xiaohuic@ the version of build that libassistant pulls in is from mid-October.  To fix the issue, just update build and buildtools.

Though I'd like to issue a warning: the way libassistant is built right now is very fragile and will likely break again in the future.  There are 2 ways you can fix this below.  I'd recommend the second.

1. Make sure there's no c++ objects getting passed over .so boundaries
2. Don't do a nested gn build and simply have Chrome depend on libassistant's BUILD.gn files directly, like how eg. v8 does.  Out of curiosity, why wasn't it done this way?

Hi thomasanderson@, to load the Assistant, you need to:
1. You may need to go to chrome://flags to turn on "Enable Google Assistant"
2. Go to chrome://settings and search Assistant and enable it
3. Press Assistant button or open it from Launcher->Search Box->Assistant icon.

I recommend doing 1 in addition to 2 too. C++ objects should not be passed across .so boundaries, that limits both sides of the boundary too much.

1. We do not have control of the LibAssistant API definition. We inherit it from the Cast/Assistant team. Changing the API would require buy-in from many other teams.
2. I am not sure how v8 is doing it. But we had run into many problems with libassistant integration and settled with the current setup. If you are interested here is a list of things we tried and explored.  Let me know if V8 is doing something different we could try.  But just directly dep on libasssitant's BUILD.gn target would not work, which is the first thing we tried when we started the project.
https://docs.google.com/document/d/1XJEXL7qRHvTFbaQBy5aLLbmowZ-PP1mYPwiiy7IVDJQ/edit

thomasanderson@ Since ChromeOS Chrome on linux build is purely for ChromeOS developers and it would take a while to work out a solution. Can we disable _LIBCPP_ABI_UNSTABLE just for ChromeOS Chrome on linux to unblock engineers?

btw: to repro follow comment#0's gn args and commandline args

I already told you how to fix it to unblock engineers in c#3: "To fix the issue, just update build and buildtools"

Can you be more specific? 

Also is it going to be less fragile if we disable "_LIBCPP_ABI_UNSTABLE"?

> Can you be more specific? 

These two dirs are out-of-date:
chromeos/assistant/libassistant/src/build
chromeos/assistant/libassistant/src/buildtools

You need to update the DEPS file that pulls in buildtools:
chromeos/assistant/libassistant/src/deps/DEPS

It looks like build isn't DEPS'ed in, but just copied.  So copy build manually from Chromium's toplevel src dir.

In particular, //chromeos/assistant/libassistant/src/build/config/posix/BUILD.gn is the file that's supposed to define _LIBCPP_ABI_UNSTABLE, but it's currently not because it's out-of-sync with //build/config/posix/BUILD.gn.

> Also is it going to be less fragile if we disable "_LIBCPP_ABI_UNSTABLE"?

No.  I'd strongly recommend trying again to directly depend on libassistant's BUILD.gn files.  //chromeos/assistant is currently taking up 13G because of duplicated repos needed for the standalone build.

I cannot uprev the build configs for libassistant freely. The Chromium dependency flow from the Chromium tree to Eureka's Chromium fork and then to libassistant's Chromium fork. These updates are managed by their respective teams.

It seems Eureka's fork has not been updated yet. https://cs.corp.google.com/eureka_internal/chromium/src/build/config/posix/BUILD.gn?sq&g=0

On top of that, we are tracking libassistant's stable release branch. It's unlikely we can merge this once it reaches libassistant's HEAD.  

I will see if I can get this one line change in manually.

re: directly depend on libassistant's BUILD.gn files
There are many problems with this approach when we tried it. We even kept it running for a few months before we gave it up. It's described in the doc in comment#7. But I am interested in your take if you have a few minutes to chat. A pair of fresh eyes may see something new.

Having "_LIBCPP_ABI_UNSTABLE" on both sides of .so fixed the first crash but still brings some weird errors. Below is an error that indicates the mojom callback was destroyed immediately but should have already been moved. Without "_LIBCPP_ABI_UNSTABLE" flag it works fine.

Can we whitelist "_LIBCPP_ABI_UNSTABLE" ChromeOS for now before we sort out all the implications? 

[175049:175049:0111/170551.996252:FATAL:settings.mojom.cc(655)] Check failed: !connected. AssistantSettingsManager::GetSettingsCallback was destroyed without first either being run or its corresponding binding being closed. It is an error to drop response callbacks which still correspond to an open interface pipe.
#0 0x7f5a30af9fcf base::debug::StackTrace::StackTrace()
#1 0x7f5a30a26d1a logging::LogMessage::~LogMessage()
#2 0x7f5a284b0015 chromeos::assistant::mojom::AssistantSettingsManager_GetSettings_ProxyToResponder::OnIsConnectedComplete()
#3 0x7f5a30dc15ac mojo::(anonymous namespace)::ResponderThunk::IsConnectedAsync()
#4 0x7f5a284aff20 std::__Cr::unique_ptr<>::reset()
#5 0x7f5a284afe99 base::internal::BindState<>::Destroy()
#6 0x7f5a28493449 base::internal::BindState<>::Destroy()
#7 0x7f5a28493737 std::__Cr::__function::__policy::__large_destroy<>()
#8 0x7f5a28491dcc chromeos::assistant::AssistantSettingsManagerImpl::GetSettings()
#9 0x7f5a284af585 chromeos::assistant::mojom::AssistantSettingsManagerStubDispatch::AcceptWithResponder()
#10 0x7f5a2847dec6 chromeos::assistant::mojom::AssistantSettingsManagerStub<>::AcceptWithResponder()
#11 0x7f5a30dbee47 mojo::InterfaceEndpointClient::HandleValidatedMessage()
#12 0x7f5a30dbe7e6 mojo::FilterChain::Accept()
#13 0x7f5a30dc0205 mojo::InterfaceEndpointClient::HandleIncomingMessage()
#14 0x7f5a30dc6668 mojo::internal::MultiplexRouter::ProcessIncomingMessage()
#15 0x7f5a30dc5b29 mojo::internal::MultiplexRouter::Accept()
#16 0x7f5a30dbe7e6 mojo::FilterChain::Accept()
#17 0x7f5a30db9536 mojo::Connector::ReadSingleMessage()
#18 0x7f5a30dba041 mojo::Connector::ReadAllAvailableMessages()
#19 0x7f5a30db9ee9 mojo::Connector::OnHandleReadyInternal()
#20 0x7f5a30dba857 mojo::SimpleWatcher::DiscardReadyState()
#21 0x7f5a30d7eb92 mojo::SimpleWatcher::OnHandleReady()
#22 0x7f5a30d7f0a1 _ZN4base8internal7InvokerINS0_9BindStateIMN4mojo13SimpleWatcherEFvijRKNS3_18HandleSignalsStateEEJNS_7WeakPtrIS4_EEijS5_EEEFvvEE7RunImplIRKS9_RKNSt4__Cr5tupleIJSB_ijS5_EEEJLm0ELm1ELm2ELm3EEEEvOT_OT0_NSI_16integer_sequenceImJXspT1_EEEE
#23 0x7f5a30a0adb1 base::debug::TaskAnnotator::RunTask()
#24 0x7f5a30a35e4f base::MessageLoopImpl::RunTask()
#25 0x7f5a30a364d3 base::MessageLoopImpl::DoWork()
#26 0x7f5a30b1d299 base::MessagePumpLibevent::Run()
#27 0x7f5a30a359f8 base::MessageLoopImpl::Run()
#28 0x7f5a30a671b5 base::RunLoop::Run()
#29 0x562b8df074bb ChromeBrowserMainParts::MainMessageLoopRun()
#30 0x7f5a2e34dc74 content::BrowserMainLoop::RunMainMessageLoopParts()
#31 0x7f5a2e350326 content::BrowserMainRunnerImpl::Run()
#32 0x7f5a2e34a8be content::BrowserMain()
#33 0x7f5a2ee8974c content::ContentMainRunnerImpl::RunServiceManager()
#34 0x7f5a2ee89351 content::ContentMainRunnerImpl::Run()
#35 0x7f5a205023b3 service_manager::Main()
#36 0x7f5a2ee877b4 content::ContentMain()
#37 0x562b8d09c6b3 ChromeMain
#38 0x7f5a20a412b1 __libc_start_main
#39 0x562b8d09c52a _start

Did you also update buildtools?  //chromeos/assistant/libassistant/src/buildtools/third_party/libc++ and //buildtools/third_party/libc++ also need to be in sync

You can disable _LIBCPP_ABI_UNSTABLE for now if updating repos is giving significant troubles.  Please use the condition "enable_cros_libassistant" for disabling in build/config/posix/BUILD.gn

No, I didn't update buildtools because of the process I mentioned in #12. 

I made a CL to skip _LIBCPP_ABI_UNSTABLE for now. Using condition "enable_cros_libassistant" looks a bit weird for //build/config/ because the gn flag is defined in /chromeos/assistant/assistant.gni. I used the default value of the flag instead which is "is_chromeos && is_chrome_branded".  Let me know if it's fine to include assistant.gni in //build/config, then I can change it.

https://chromium-review.googlesource.com/c/chromium/src/+/1409519

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/652495299553fa515c6246e98babd7617a87b7ce

commit 652495299553fa515c6246e98babd7617a87b7ce
Author: Xiaohui Chen <xiaohuic@google.com>
Date: Mon Jan 14 21:13:26 2019

Skip "_LIBCPP_ABI_UNSTABLE" for internal ChromeOS

Bug: 917533
Test: locally build Chrome for ChromeOS and no crash
Change-Id: I4c425465c3d407cc3bd000e5a501b5ede7731781
Reviewed-on: https://chromium-review.googlesource.com/c/1409519
Commit-Queue: Xiaohui Chen <xiaohuic@chromium.org>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#622601}
[modify] https://crrev.com/652495299553fa515c6246e98babd7617a87b7ce/build/config/posix/BUILD.gn