Naive question: Did they ever pass?  Looks like it's been failing as far back as the histroy goes (Sept 28).

No, they haven't.

(which is more of a reason to believe that I may not have everything set up correctly)

Ah, got it.  FWIW I am able to repro the bot failure on my workstation if I build with the same args:

maniscalco-z620 ~/chromium/src(master) $ cat out/bepcs/args.gn
import("//build/args/bots/chromium.fyi/blimp_engine_rel.gn")
use_goma = true

(gdb) run
Starting program: /usr/local/google/home/maniscalco/chromium/src/out/bepcs/content_browsertests --gtest_filter=OpenedByDOMTest.Popup --single_process
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/grte/v4/lib64/libthread_db.so.1".
Detected presence of a debugger, running without test timeouts.
Note: Google Test filter = OpenedByDOMTest.Popup
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from OpenedByDOMTest, where TypeParam = 
[ RUN      ] OpenedByDOMTest.Popup
[20159:20159:1007/113355:93458054252:INFO:fake_audio_manager.cc(23)] FakeAudioManager
[New Thread 0x7fffec965700 (LWP 20180)]
[New Thread 0x7fffe995f700 (LWP 20186)]
[New Thread 0x7fffea160700 (LWP 20185)]
[New Thread 0x7fffea961700 (LWP 20184)]
[New Thread 0x7fffeb162700 (LWP 20183)]
[New Thread 0x7fffeb963700 (LWP 20182)]
[New Thread 0x7fffec164700 (LWP 20181)]
[New Thread 0x7fffed166700 (LWP 20179)]
[New Thread 0x7fffed967700 (LWP 20178)]
[New Thread 0x7fffee168700 (LWP 20177)]
[New Thread 0x7fffee969700 (LWP 20176)]
[New Thread 0x7fffef16a700 (LWP 20175)]
[New Thread 0x7fffef96b700 (LWP 20174)]
[New Thread 0x7ffff016c700 (LWP 20173)]
[New Thread 0x7ffff096d700 (LWP 20172)]
[New Thread 0x7ffff116e700 (LWP 20171)]
[New Thread 0x7ffff196f700 (LWP 20163)]

Program received signal SIGILL, Illegal instruction.
views::DesktopFactoryOzone::GetInstance () at ../../ui/views/widget/desktop_aura/desktop_factory_ozone.cc:21
21        CHECK(impl_) << "DesktopFactoryOzone accessed before constructed";
(gdb) bt
#0  views::DesktopFactoryOzone::GetInstance () at ../../ui/views/widget/desktop_aura/desktop_factory_ozone.cc:21
#1  0x000000000429ab36 in views::CreateDesktopScreen () at ../../ui/views/widget/desktop_aura/desktop_screen_ozone.cc:12
#2  0x0000000001bfeb16 in content::Shell::PlatformInitialize (default_window_size=...) at ../../content/shell/browser/shell_views.cc:327
#3  0x0000000001bf2512 in content::Shell::Initialize () at ../../content/shell/browser/shell.cc:168
#4  0x0000000001c0ca48 in content::ShellBrowserMainParts::PreMainMessageLoopRun (this=0x18d2074b0880) at ../../content/shell/browser/shell_browser_main_parts.cc:185
#5  0x00000000016e9f65 in content::BrowserMainLoop::PreMainMessageLoopRun (this=0x18d207459a80) at ../../content/browser/browser_main_loop.cc:964
#6  0x0000000001a5abb6 in Run (this=<optimized out>) at ../../base/callback.h:64
#7  content::StartupTaskRunner::RunAllTasksNow (this=<optimized out>) at ../../content/browser/startup_task_runner.cc:45
#8  0x00000000016e8571 in content::BrowserMainLoop::CreateStartupTasks (this=0x18d207459a80) at ../../content/browser/browser_main_loop.cc:854
#9  0x00000000016ec957 in content::BrowserMainRunnerImpl::Initialize (this=0x18d20746e7e0, parameters=...) at ../../content/browser/browser_main_runner.cc:140
#10 0x0000000001bf32ac in ShellBrowserMain (parameters=..., main_runner=std::unique_ptr<content::BrowserMainRunner> containing 0x18d20746e7e0) at ../../content/shell/browser/shell_browser_main.cc:23
#11 0x0000000001be9742 in content::ShellMainDelegate::RunProcess (this=<optimized out>, process_type=..., main_function_params=...) at ../../content/shell/app/shell_main_delegate.cc:295
#12 0x0000000001649e30 in content::RunNamedProcessTypeMain (process_type="", main_function_params=..., delegate=0x18d207452750) at ../../content/app/content_main_runner.cc:398
#13 0x000000000164a84b in content::ContentMainRunnerImpl::Run (this=0x18d20746bfa0) at ../../content/app/content_main_runner.cc:779
#14 0x0000000001642eb0 in content::ContentMain (params=...) at ../../content/app/content_main.cc:20
#15 0x0000000001bb7b19 in content::BrowserTestBase::SetUp (this=0x18d2074a1c80) at ../../content/public/test/browser_test_base.cc:308
#16 0x0000000001bb3413 in content::ContentBrowserTest::SetUp (this=0x18d2074a1c80) at ../../content/public/test/content_browser_test.cc:94
#17 0x0000000001d15c38 in HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x18d2074a1c80, method=<error reading variable: DWARF-2 expression error: `DW_OP_stack_value' operations must be used either alone or in conjunction with DW_OP_piece or DW_OP_bit_piece.>, 
    location=<optimized out>) at ../../testing/gtest/src/gtest.cc:2458
#18 testing::Test::Run (this=0x18d2074a1c80) at ../../testing/gtest/src/gtest.cc:2470
#19 0x0000000001d169a3 in testing::TestInfo::Run (this=0x18d207426600) at ../../testing/gtest/src/gtest.cc:2656
#20 0x0000000001d16e37 in testing::TestCase::Run (this=<optimized out>) at ../../testing/gtest/src/gtest.cc:2774
#21 0x0000000001d1de67 in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at ../../testing/gtest/src/gtest.cc:4647
#22 0x0000000001d1daba in HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x18d207383200, method=<error reading variable: access outside bounds of object referenced via synthetic pointer>, location=<optimized out>)
    at ../../testing/gtest/src/gtest.cc:2458
---Type <return> to continue, or q <return> to quit---

iirc that's the __builtin_trap here: https://codesearch.chromium.org/chromium/src/base/logging.h?rcl=0&l=461

On non-official builds, that'd show up as a more normal-looking CHECK failure.

I ran it locally with the error message below:

shenghuazhang@shenghuazhang0:/usr/local/google/code/chromium/src$ out/bepcs/content_browsertests --gtest_filter=OpenedByDOMTest.Popup --single_process
Note: Google Test filter = OpenedByDOMTest.Popup
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from OpenedByDOMTest, where TypeParam = 
[ RUN      ] OpenedByDOMTest.Popup
[117716:117716:1007/115202:946592284838:INFO:fake_audio_manager.cc(23)] FakeAudioManager
[117740:117740:1007/115202:946592401218:ERROR:broker_posix.cc(41)] Invalid node channel message
Illegal instruction (core dumped)

#7: that "Illegal instruction (core dumped)" is the same SIGILL as #5.

content_browsertests with chromeos=0 is currently unsupported by Ozone. We are likely blocked until  crbug.com/295089  is resolved.

I will remove content_browsertest from the waterfall in the meantime.

(source of that info: discussion with rjkroege@)

Re #10, nice find!  Looks like that bug you referenced has been marked fixed.  Is there a different bug we should be be Blockedon ? (presumably one that isn't yet filed?)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9d1eef3834a132c7911a82ac5064e10adde81ac2

commit 9d1eef3834a132c7911a82ac5064e10adde81ac2
Author: kmarshall <kmarshall@chromium.org>
Date: Fri Nov 18 19:12:19 2016

Remove content_browsertests from Blimp Linux Engine buildbot recipe.

The tests are failing because Ozone isn't yet in a state where it
can run tests on non-ChromeOS configurations. See the bug for more
information.

BUG= 653917 
R=maniscalco@chromium.org,dpranke@chromium.org

Review-Url: https://codereview.chromium.org/2496283005
Cr-Commit-Position: refs/heads/master@{#433258}

[modify] https://crrev.com/9d1eef3834a132c7911a82ac5064e10adde81ac2/testing/buildbot/chromium.fyi.json

owner=jbudorick for verification

https://build.chromium.org/p/chromium.fyi/builders/Blimp%20Linux%20Engine/builds/2305