Bob, do you want to take a shot at this? The ToT bots look pretty green.

inglorion, did you have a chance to look at this?

Thanks, Nico. Yes, I'll take this one.

And thanks Hans too!

Need r326226 for issue 469376.

We need r326854 for issue 818141

Need at least r326722 to make sure arm64 builds work. Need at least r326862 for issue 819236.

The latest Clang that passes tests on my system is r324295. Trying to figure out what's wrong here.

Trying r326862.

https://chromium-review.googlesource.com/c/chromium/src/+/954003

Trying r326975: https://chromium-review.googlesource.com/#/c/chromium/src/+/954407

Trying r327100: https://crrev.com/c/954407

need r327124 for /order: support in lld

(the win failures at the latest attempt also show up on http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/)

r327105 broke a .mm debug info test on Win; complained on the commit. If that's not fixed soon, we can revert to unblock us.

The execname failure started in http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/9300 , almost certainly due to http://llvm.org/viewvc/llvm-project?revision=326710&view=revision

Trying r327184: https://crrev.com/c/954407

I replied on the r326710 thread and reverted in r327266.
Kicked off packaging of that in https://chromium-review.googlesource.com/#/c/chromium/src/+/959003

Thanks Hans! That looks happier. Want me to roll from here or do you want to take over?

Let's try to exploit our time zone coverage and both work towards a roll.

Looks like the goma team helped with pushing the package in #19. I've started tryjobs on that.

Try fuchsia failure in the tryjob for #19 is real. I've started bisecting, but since content_unittests is a large target it will take a while. The linux_android_rel_ng failure also looks real, and if we're lucky it's the same root cause.

Bisection points to r326991, but it will not be easy to come up with a test case for that.

https://bugs.llvm.org/show_bug.cgi?id=36729 for the fuchsia test failure. I'm still not sure exactly what's going on, but it's isolated to a single function in Skia and there's small asm diff.

Is this arm (since android also has something? Looked different though?) or fuchsia specific? phosek, any recent fuchsia clang changes that could make tests fail?

The two recent changes are setting -fno-common and -fsanitize=safe-stack -fstack-protector-strong by default for Fuchsia, but this doesn't seem to be either of those?

The fuchsia tests are x86_64. The bisection last night was faulty (test doesn't always fail) so I'm re-running it. Hoping for a proper smoking gun this time.

We will want at least Clang r327560 to get some fixes rnk committed, including the fix for  issue 818086 .

Trying r327571: https://crrev.com/c/954407

Trying r327580, which contains a fix for a broken test. https://crrev.com/c/954407

Thanks! I've pushed that to goma and trying it here:
https://chromium-review.googlesource.com/c/chromium/src/+/959003/5

The linux_android_rel_ng failure is still there. I had optimistically hoped it would've gone away :-/

It should be possible to bisect this by running the test on swarming, but it will be painful.
My notes:

Some instructions:
https://www.chromium.org/developers/testing/isolated-testing/for-swes#TOC-Run-a-test-built-locally-on-Swarming

Configure gn as on the bot, but pointing to local clang:
$ cat out/release/args.gn 
clang_base_path="/work/llvm.combined/build.release"
clang_use_chrome_plugins = false
llvm_force_head_revision = true
dcheck_always_on = true
ffmpeg_branding = "Chrome"
is_component_build = false
is_debug = false
proprietary_codecs = true
strip_absolute_paths_from_debug_symbols = true
strip_debug_info = true
symbol_level = 1
target_os = "android"

Build:

$ ninja -C out/release -t clean && ninja -C out/release chrome_public_test_apk

Create isolate:

$ echo gn > out/release/mb_type
$ tools/swarming_client/isolate.py archive -I https://isolateserver.appspot.com -i out/release/chrome_public_test_apk.isolate -s out/release/chrome_public_test_apk.isolated --verbose
(copy the hash for the isolate printed by the last command)
$ export hash=21db8835e6002c568dfbd56d0b49de325126f4cb

Trigger the test run with this giant command:
(Using the hash from above for the --isolated argument.)

$ tools/swarming_client/swarming.py trigger --swarming https://chromium-swarm.appspot.com --isolate-server https://isolateserver.appspot.com --dimension device_os KTU84P --dimension device_type hammerhead --dimension os Android --dimension pool Chrome --named-cache swarming_module_cache_vpython .swarming_module_cache/vpython --idempotent --user hans@chromium.org --cipd-package 'bin:infra/tools/luci/logdog/butler/${platform}:git_revision:ff387eadf445b24c935f1cf7d6ddd279f8a6b04c' --cipd-package '.swarming_module:infra/python/cpython/${platform}:version:2.7.14.chromium14' --cipd-package '.swarming_module:infra/tools/luci/logdog/butler/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c' --cipd-package '.swarming_module:infra/tools/luci/vpython-native/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c' --cipd-package '.swarming_module:infra/tools/luci/vpython/${platform}:git_revision:e1abc57be62d198b5c2f487bfb2fa2d2eb0e867c' --env-prefix PATH .swarming_module --env-prefix PATH .swarming_module/bin --env-prefix VPYTHON_VIRTUALENV_ROOT .swarming_module_cache/vpython --isolated $hash -- --gtest_filter=*ContextMenuLoadUrlParamsTest#testOpenInNewTabSanitizeReferrer

That command prints a URL and command for fetching the results.

There's also tools/run-swarmed.py which you can make work for android with minimal changes (but it does require a change).

Sigh, the test insists on passing when I run it like this. Maybe it needs more tests running at the same time to fail or something.

Grumble grumble, trying again with Clang r327580 specifically, it does fail that test.
I don't like this.

Tried again, building Clang from r327580 myself and the test passed:
https://chromium-swarm.appspot.com/task?id=3c436719eb4cec10&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3c436cd8360aab10&refresh=10&show_raw=1
The test is failing very consistently on the trybot, so it doesn't seem flaky.

Is it because I built Clang myself and not with the packaging script? That's worth a shot I suppose..

(Why didn't we catch this with the ToT bots? The ToTAndroid bot doesn't run tests. But didn't it use to? Maybe this got lost in the big test spec simplification recently..)

If you end up rolling further it might be nice to get r327668 for issue 810154.

> The ToTAndroid bot doesn't run tests. But didn't it use to?

I don't recall ever seeing it run tests. ToTAndroidCFI does, but we aren't running chrome_public_test_apk there because, as I recall, at least one of its tests doesn't pass reliably on M.

I mean it doesn't pass even without CFI.

However, looking at ToTAndroidCFI, a number of tests are currently failing, and they seem to be approximately the same tests that failed on the roll CL. ToTAndroidCFI started failing like that in r327248, which would seem to point to that as the culprit.

r327248 moved _GLOBAL_OFFSET_TABLE_ from .got to .got.plt. All of the failing tests seem to relate to ffmpeg video playback. And ffmpeg contains asm that refers to _GLOBAL_OFFSET_TABLE_:

https://cs.chromium.org/chromium/src/third_party/ffmpeg/libavutil/arm/asm.S?type=cs&q=_GLOBAL_OFFSET_TABLE_+-file:elfutils&sq=package:chromium&l=204

I bet that code relies on _GLOBAL_OFFSET_TABLE_ being at the start of .got.

Here is an example of a function that uses the macro.

00000394 <fft32_neon>:
 394:   b510            push    {r4, lr}
 396:   4604            mov     r4, r0
 398:   f7ff fe90       bl      bc <fft16_neon>
 39c:   f104 0040       add.w   r0, r4, #64     ; 0x40
 3a0:   f7ff fe48       bl      34 <fft8_neon>
 3a4:   f104 0060       add.w   r0, r4, #96     ; 0x60
 3a8:   f7ff fe44       bl      34 <fft8_neon>
 3ac:   4620            mov     r0, r4
 3ae:   e8bd 4010       ldmia.w sp!, {r4, lr}
 3b2:   f8df c010       ldr.w   ip, [pc, #16]   ; 3c4 <fft32_neon+0x30>
 3b6:   44fc            add     ip, pc
 3b8:   4903            ldr     r1, [pc, #12]   ; (3c8 <fft32_neon+0x34>)
 3ba:   f85c 1001       ldr.w   r1, [ip, r1]
 3be:   f04f 0204       mov.w   r2, #4
 3c2:   e71d            b.n     fffffe3e <_GLOBAL_OFFSET_TABLE_+0xfffffe3e>
 3c4:   0000000a        .word   0x0000000a
                        3c4: R_ARM_REL32        _GLOBAL_OFFSET_TABLE_
 3c8:   00000000        .word   0x00000000
                        3c8: R_ARM_GOT32        ff_cos_32_fixed

This function expects the R_ARM_GOT32 to resolve to the difference between _GLOBAL_OFFSET_TABLE_ and the GOT entry for ff_cos_32_fixed, but R_ARM_GOT32 actually resolves to the difference between .got and the GOT entry.

I tried fixing this in lld but it doesn't seem trivial, so I will revert r327248 with a reproducer.

The revert is r327688.

Thanks pcc!

I had a shower thought this morning, realizing I hadn't been rebuilding lld in my bisection yesterday, only clang, which would explain why I couldn't reproduce the failure.

I've verified now that r327248 is indeed the culprit for this test failure too.

I'll start packaging r327688 (or something more recent which includes it).

Looking pretty good so far: https://chromium-review.googlesource.com/c/chromium/src/+/959003/7

I've seen linux_chromium_cfi_rel_ng fail on a whitespace change before so that might be benign, the chromeos_asan_rel_ng bot usually times out. The mach_chromium_asan_rel_ng redness is new, hopefully it's a flake.

A fresh whitespace change to compare with here: https://chromium-review.googlesource.com/c/chromium/src/+/966701

I'll check back in on this a little later.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6705a9b1d17e8f56c9a8229c971a7286ce1fbe42

commit 6705a9b1d17e8f56c9a8229c971a7286ce1fbe42
Author: Hans Wennborg <hans@chromium.org>
Date: Fri Mar 16 17:23:38 2018

Roll Clang 325667:327688

Clang r326867 turned on SafeStack when targeting Fuchsia by default.
Turn it off in our build config until all the tests pass.

Bug:  817298 , 821951

Change-Id: Id2d9f911eb62fe6823365baeba55a4ccc0b484ce
Reviewed-on: https://chromium-review.googlesource.com/959003
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#543745}
[modify] https://crrev.com/6705a9b1d17e8f56c9a8229c971a7286ce1fbe42/build/config/fuchsia/BUILD.gn
[modify] https://crrev.com/6705a9b1d17e8f56c9a8229c971a7286ce1fbe42/tools/clang/scripts/update.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/309e36bd5d011d57f5e27fb18c6438673129da65

commit 309e36bd5d011d57f5e27fb18c6438673129da65
Author: Hans Wennborg <hans@chromium.org>
Date: Mon Mar 19 14:19:50 2018

Run tests on chromium.clang's ToTAndroid bot

We've had poor Android test coverage on the ToT bots, as shown by the
recent clang roll. Historically that's been due to flakiness of running
Android tests. But now that everything's on swarming it should be
better, and this uses the config from linux_android_rel_ng which runs on
the CQ and seems to be very stable.

Bug:  817298 
Change-Id: Ie992c49c9a86037529f8ac6bd70c7ce7b732850c
Reviewed-on: https://chromium-review.googlesource.com/968243
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#544022}
[modify] https://crrev.com/309e36bd5d011d57f5e27fb18c6438673129da65/testing/buildbot/chromium.clang.json
[modify] https://crrev.com/309e36bd5d011d57f5e27fb18c6438673129da65/testing/buildbot/waterfalls.pyl

The roll seems to be sticking so far, but since it landed on Friday, let's wait at least until end of today MTV time before landing dependent changes.

This appears to be sticking.

Tracking bug for the next roll:  https://crbug.com/823655 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ecc84c5100f4301f65f3af16d72054f3dec26496

commit ecc84c5100f4301f65f3af16d72054f3dec26496
Author: Hans Wennborg <hans@chromium.org>
Date: Tue Apr 03 14:46:15 2018

mb: Use KitKat Phone Tester (rel)'s build config for ToTAndroid

The KitKat Phone Tester config is used by the linux_android_rel_ng CQ bot.
ToTAndroid now runs the same tests as that bot, but they don't pass due to
build config differences.

It would of course be nice if the tests passed without the official codecs etc.
but that shouldn't be a distraction from keeping the ToT bots green in the
config tested by the CQ.

Bug:  817298 
Change-Id: I4b89d6c050d39038af8faddbd4225b394048b1f7
Reviewed-on: https://chromium-review.googlesource.com/992234
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#547701}
[modify] https://crrev.com/ecc84c5100f4301f65f3af16d72054f3dec26496/tools/mb/mb_config.pyl

The Android tests that I added are still failing. Tracking that in  crbug.com/840756