+Clang folks and adding Build-Toolchain. Is there a Clang-specific component or label?

Build-Toolchain ist some CrOS thing I believe. Wr use just "clang" as label for clang-related things. Sounds like it's not confirmed that this is clang-related yet though?

Nope it isn't confirmed yet that it is a clang problem. If Geoff can confirm the roll caused this, it could also be some undefined behavior in ANGLE that gets triggered by new optimizations in clang.

I can't seem to repro any of these failures locally with ToT Chrome, debug or release builds, non-tsan.

Geoff has been able to reproduce this with is_debug=false, however the previous version of clang doesn't exist on the server anymore. hans@ can we have that old version back so we can test?

We never delete clangs, all that we ever build are at http://commondatastorage.googleapis.com/chromium-browser-clang/index.html?path=Linux_x64/ . Just editing the revision in tools/clang/scripts/update.py and running runhooks will get it back. It might not work on goma anymore after a while, but the last 2-3 builds usually work on goma.

When I locally revert https://chromium.googlesource.com/chromium/src/+/a545baf74563a7bf60c5c6eb2e044e76cad48e0e and then gclient sync I get the following error:

pdating Clang to 300839-2...
Downloading prebuilt clang
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-300839-2.tgz 
HTTP Error 404: Not Found
Failed to download prebuilt clang clang-300839-2.tgz
Use --force-local-build if you want to build locally.
Exiting.

Do you have any local changes? The script is supposed to download 300839-1, not 300839-2. (See CLANG_SUB_REVISION on both lhs and rhs of https://codereview.chromium.org/2848923002/diff/1/tools/clang/scripts/update.py)

Ah, yes, some other CL changed the CLANG_SUB_REVISION so the revert only reverted the CLANG_REVISION.

Can confirm that the clang roll is what broke this test.  The crash is in dEQP's code to generate random shader permutations:

#0  0x00007ffff7b88140 in typeinfo for rsg::FloatLiteral () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#1  0x00007ffff75c013e in rsg::Expression::createRandomLValue(rsg::GeneratorState&, rsg::ConstValueRangeAccess) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#2  0x00007ffff75c26eb in rsg::AssignOp::createNextChild(rsg::GeneratorState&) () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#3  0x00007ffff75d1dbd in rsg::ExpressionGenerator::generate(rsg::Expression*) () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#4  0x00007ffff75d1c44 in rsg::ExpressionGenerator::generate(rsg::ValueRange const&, int) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#5  0x00007ffff75d82b2 in rsg::ExpressionStatement::ExpressionStatement(rsg::GeneratorState&) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#6  0x00007ffff75daa5f in rsg::Statement* rsg::(anonymous namespace)::create<rsg::ExpressionStatement>(rsg::GeneratorState&) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#7  0x00007ffff75d8690 in rsg::BlockStatement::createNextChild(rsg::GeneratorState&) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#8  0x00007ffff75d200e in rsg::FunctionGenerator::generate() () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#9  0x00007ffff75d6a15 in rsg::ShaderGenerator::generate(rsg::ShaderParameters const&, rsg::Shader&, std::vector<rsg::ShaderInput*, std::allocator<rsg::ShaderInput*> > const&)
    () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#10 0x00007ffff75d4e09 in rsg::ProgramGenerator::generate(rsg::ProgramParameters const&, rsg::Shader&, rsg::Shader&) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#11 0x00007ffff785d8e2 in deqp::gls::RandomShaderCase::init() () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#12 0x00007ffff78e865d in tcu::RandomOrderExecutor::executeInner(tcu::TestCase*, std::string const&) ()
   from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#13 0x00007ffff78e859e in tcu::RandomOrderExecutor::execute(std::string const&) () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#14 0x00007ffff7484852 in deqp_libtester_run(char const*) () from /usr/local/google/home/geofflang/src/chromium/src/out/Release/./libangle_deqp_libgles3.so
#15 0x000000000040e393 in (anonymous namespace)::dEQP_GLES3_Default_Test::TestBody() ()
#16 0x000000000042d916 in testing::Test::Run() ()
#17 0x000000000042e330 in testing::TestInfo::Run() ()
#18 0x000000000042e857 in testing::TestCase::Run() ()
#19 0x0000000000435817 in testing::internal::UnitTestImpl::RunAllTests() ()
#20 0x0000000000435497 in testing::UnitTest::Run() ()
#21 0x0000000000488581 in base::TestSuite::Run() ()
#22 0x0000000000414833 in (anonymous namespace)::RunHelper(base::TestSuite*) ()
#23 0x00000000004899aa in base::(anonymous namespace)::LaunchUnitTestsInternal(base::Callback<int (), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> const&, int, int, bool, base::Callback<void (), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> const&) ()
#24 0x0000000000489e3d in base::LaunchUnitTestsSerially(int, char**, base::Callback<int (), (base::internal::CopyMode)1, (base::internal::RepeatMode)1> const&) ()
#25 0x00000000004147c0 in main ()

Given that the crash is in typeinfo, I'm thinking it's a compiler bug.  I'm not sure how to proceed.

Huh, typeinfo? Do you use rtti?

dEQP is a third party test suite that uses RTTI and exceptions, currently it's only on the GPU FYI bots.

The GN config for it: https://cs.chromium.org/chromium/src/third_party/angle/src/tests/BUILD.gn?l=218&rcl=82830edeacade1dd52ba8bc2338ecb195641c5e3

Can you make a reduced reproducer?

That will take some time I think, it's not obvious where typeinfo is being used in the code from the stack trace.  In the meantime, we should suppress the failures I think

Up to you. We look at the bots on https://build.chromium.org/p/chromium.fyi/console?category=clang%20tot before rolls. Nothing in Chromium land uses RTTI, so we don't test that when rolling compilers. If you make a reduced repro, we can make sure this specific problem is fixed in the next roll though. (Other folks also use clang, and it's possible someone else reports this and that it gets fixed on its own eventually if it's really a compiler bug.)

Another thing to keep in mind that sometimes compiler rolls just expose previously-existing undefined behavior in the code, so even if a compiler roll triggered it, it's not 100% certain it's a compiler bug. (But it's well possible.)

For what it's worth, to repro this bug on Linux:

GN args:
build_angle_deqp_tests = true
is_debug = false

build target:
angle_deqp_gles3_tests

flags:
angle_deqp_gles3_tests --gtest_filter=dEQP_GLES3.Default/functional_shaders_random_trigonometric_fragment_77 --single-process-tests

If there is a reasonable way to bisect clang, I'd try that too.

You can pass --force-local-build to update.py and change the revision there, bit that'll build clang locally (which is slow in itself and also means you can't use goma), but even if the clang revision is known we can't revert it upstream without a reduced repro.

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/1a310b847e107d411d167d701473217c431236ad

commit 1a310b847e107d411d167d701473217c431236ad
Author: Corentin Wallez <cwallez@chromium.org>
Date: Wed May 17 17:01:13 2017

dEQP expectations: suppress failures caused by a clang roll

Hopefully this makes the bot green while we investigate whether the bug
is in dEQP or clang, and find a fix.

BUG= 722345 

Change-Id: I2813881addb4ebfe23443244d175b4c6db698a7d
Reviewed-on: https://chromium-review.googlesource.com/507767
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>

[modify] https://crrev.com/1a310b847e107d411d167d701473217c431236ad/src/tests/deqp_support/deqp_gles2_test_expectations.txt
[modify] https://crrev.com/1a310b847e107d411d167d701473217c431236ad/src/tests/deqp_support/deqp_gles3_test_expectations.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/13d7727c2c3179ca83db74d20de29954c428fe52

commit 13d7727c2c3179ca83db74d20de29954c428fe52
Author: jmadill <jmadill@chromium.org>
Date: Wed May 17 21:01:37 2017

Roll ANGLE 995c2ed..4465330

https://chromium.googlesource.com/angle/angle.git/+log/995c2ed..4465330

BUG= 722345 ,None

TBR=geofflang@chromium.org

TEST=bots

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2889913003
Cr-Commit-Position: refs/heads/master@{#472566}

[modify] https://crrev.com/13d7727c2c3179ca83db74d20de29954c428fe52/DEPS

This bug has caused angle_deqp_gles2_gl_tests to start flaking on the linux_optional_gpu_tests_rel trybot, causing many GPU-related CLs to either take a very long time to commit, or get kicked out of the CQ.

One example failure:

https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_optional_gpu_tests_rel/9079

I see that that ANGLE roll with the failure suppressions landed at r472566, but the tryjob above had got_revision_cp of "refs/heads/master@{#472601}", so I don't think those failure suppressions are working. I think they need to be changed to SKIP. Raising to P1.

I think we should consider reverting the clang roll because if it has bugs in its RTTI implementation then who knows what other kinds of bugs it might have. We need help reducing these tests to something for the clang developers.

Let me take this temporarily while changing these failure expectations.

We're not reverting the clang roll over a feature not used on any waterfall we support. It's not even clear the compiler is to blame.

Uploaded https://chromium-review.googlesource.com/508102/ skipping these tests.

The compiler is almost surely at fault. These tests are running fine on multiple platforms using compilers from multiple vendors. Regardless of whether we revert the clang roll we still need some help to produce a reduced test case for the clang developers.

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/df22132f386c762352883eddde0b6822cb5a6f8b

commit df22132f386c762352883eddde0b6822cb5a6f8b
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 18 03:15:28 2017

Skip, rather than fail, crashing functional_shaders_random tests.

BUG= 722345 
TBR=geofflang@chromium.org

Change-Id: Ie4c488b730ed2b51ce8f62d5bf7961dcc8ac0104
Reviewed-on: https://chromium-review.googlesource.com/508102
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>

[modify] https://crrev.com/df22132f386c762352883eddde0b6822cb5a6f8b/src/tests/deqp_support/deqp_gles2_test_expectations.txt
[modify] https://crrev.com/df22132f386c762352883eddde0b6822cb5a6f8b/src/tests/deqp_support/deqp_gles3_test_expectations.txt

The functional_shader_random tests has been passing since clang roll 301384:303273 https://codereview.chromium.org/2884383004, but this clang roll is reverted due to breaking cc_unittests on Linux MSAN.

OK, good to know. Hopefully that issue will be fixed upstream in clang and these tests can be re-enabled after the next clang roll. In the meantime, rolling forward ANGLE with the skip expectations in https://codereview.chromium.org/2891973002/ .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2347997fc357bfa237223290cf55b37f9234e478

commit 2347997fc357bfa237223290cf55b37f9234e478
Author: kbr <kbr@chromium.org>
Date: Thu May 18 17:50:43 2017

Roll ANGLE 4465330..9f10b77

https://chromium.googlesource.com/angle/angle.git/+log/4465330..9f10b77

BUG=722684, 722345 
TBR=geofflang@chromium.org
TEST=bots

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2891973002
Cr-Commit-Position: refs/heads/master@{#472867}

[modify] https://crrev.com/2347997fc357bfa237223290cf55b37f9234e478/DEPS

The above roll didn't fix the problem. The issue is that some of the names in the test expectations were slightly wrong, so the suppressions weren't applying. Fixing.

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/bb580b1b8b124bacb44fc3de06a57ddd375bd987

commit bb580b1b8b124bacb44fc3de06a57ddd375bd987
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 18 18:26:31 2017

Fix names of some crashing shader tests.

The names were wrong, so the test expectations weren't applying.

BUG= 722345 

Change-Id: Ic64abf08e512dbf3c297ee89a74b614cd4590b58
Reviewed-on: https://chromium-review.googlesource.com/508166
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>

[modify] https://crrev.com/bb580b1b8b124bacb44fc3de06a57ddd375bd987/src/tests/deqp_support/deqp_gles2_test_expectations.txt
[modify] https://crrev.com/bb580b1b8b124bacb44fc3de06a57ddd375bd987/src/tests/deqp_support/deqp_gles3_test_expectations.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd66f55270c94cae7c3579da95294a6f56dce60d

commit bd66f55270c94cae7c3579da95294a6f56dce60d
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 18 18:38:21 2017

Stop manually invoking trybots in roll_angle.py.

The CQ_INCLUDE_TRYBOTS directive works now, and "git cl try -b
[builder] -m [master]" doesn't work with Gerrit. Always try
win_clang_dbg now, too.

BUG= 722345 
NOTRY=true

Change-Id: I41bdb1064fdffafc3996e5cd17d415c09657f7b7
Reviewed-on: https://chromium-review.googlesource.com/508848
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#472880}
[modify] https://crrev.com/bd66f55270c94cae7c3579da95294a6f56dce60d/tools/roll_angle.py

>  Always try win_clang_dbg now, too.

That's backed by like 1 machine. win_clang (without _dbg) is backed by the general chromium win tryserver pool. You'll have to wait much less if you use win_clang instead of win_clang_dbg.

Ok, we currently roll only once a day, I'll keep and eye on the cycle time of that bot for rolls.

I remember it caught some issues in the past that the non-debug one didn't but with release asserts on it may no longer be an issue.

Thanks Nico, swapping it out in https://chromium-review.googlesource.com/508850/ .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/732dbeaf182678e15e315d6586aec467efe24f17

commit 732dbeaf182678e15e315d6586aec467efe24f17
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 18 19:42:35 2017

Use win_clang instead of win_clang_dbg for ANGLE rolls.

BUG= 722345 

Change-Id: Id19c425ad057a388d1db3e96aa6f605d0b6dbf64
Reviewed-on: https://chromium-review.googlesource.com/508850
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Commit-Queue: Geoff Lang <geofflang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#472905}
[modify] https://crrev.com/732dbeaf182678e15e315d6586aec467efe24f17/tools/roll_angle.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f2259156351db352fb546f7a62a2dc33011ceb4c

commit f2259156351db352fb546f7a62a2dc33011ceb4c
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 18 19:58:30 2017

Remove win_clang from manually added tryservers.

It's part of the default set now.

BUG= 722345 
TBR=jmadill@chromium.org

Change-Id: I1eb3340f707df8e90809301d8c4e86b1b945530e
Reviewed-on: https://chromium-review.googlesource.com/507956
Reviewed-by: Nico Weber <thakis@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#472909}
[modify] https://crrev.com/f2259156351db352fb546f7a62a2dc33011ceb4c/tools/roll_angle.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6a77bbc3b1aa49c3fd821b5b793b79dc1c4e80c9

commit 6a77bbc3b1aa49c3fd821b5b793b79dc1c4e80c9
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu May 18 23:06:30 2017

Roll ANGLE 9f10b77..bb580b1

https://chromium.googlesource.com/angle/angle.git/+log/9f10b77..bb580b1

BUG= 722345 
TBR=geofflang@chromium.org
TEST=bots

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.win:win_clang_dbg

Change-Id: If6296ac15cee26f29215750857d3fb101deb96cb
Reviewed-on: https://chromium-review.googlesource.com/508477
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#472962}
[modify] https://crrev.com/6a77bbc3b1aa49c3fd821b5b793b79dc1c4e80c9/DEPS

These failures are suppressed now per https://luci-milo.appspot.com/buildbot/tryserver.chromium.linux/linux_optional_gpu_tests_rel/?limit=200 . Geoff, could you take this back and look into making a reduced test case for the clang developers? Thanks.

Comment 24 suggests that this is already fixed in clang trunk. Yesterday's roll (https://codereview.chromium.org/2897453002/) looks like it might stick, so maybe your tests are already passing again?

Thanks. Trying removing these in https://chromium-review.googlesource.com/c/510031/ .

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/538c7bcfd3a51d045a2d90a7ea332979dc75c377

commit 538c7bcfd3a51d045a2d90a7ea332979dc75c377
Author: Kenneth Russell <kbr@chromium.org>
Date: Tue May 23 14:41:27 2017

Remove failure expectations for shader_random tests.

Apparently the current clang roll has fixed these already.

BUG= 722345 

Change-Id: I59f07f197fdfd1eb6e000390ee93d41938fe4d42
Reviewed-on: https://chromium-review.googlesource.com/510031
Commit-Queue: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>

[modify] https://crrev.com/538c7bcfd3a51d045a2d90a7ea332979dc75c377/src/tests/deqp_support/deqp_gles2_test_expectations.txt
[modify] https://crrev.com/538c7bcfd3a51d045a2d90a7ea332979dc75c377/src/tests/deqp_support/deqp_gles3_test_expectations.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/257f536a7a0d55b86cfd1a874aee483f99a46f7e

commit 257f536a7a0d55b86cfd1a874aee483f99a46f7e
Author: Corentin Wallez <cwallez@chromium.org>
Date: Wed May 24 16:01:39 2017

Roll ANGLE bb580b1..9e3bd31

https://chromium.googlesource.com/angle/angle.git/+log/bb580b1..9e3bd31

BUG= 722345 

TBR=geofflang@chromium.org

TEST=bots

CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Change-Id: I3e3d88f8e7eb2c44452878a1e6bd13e874bf7a8d
Reviewed-on: https://chromium-review.googlesource.com/513419
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Commit-Queue: Corentin Wallez <cwallez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#474309}
[modify] https://crrev.com/257f536a7a0d55b86cfd1a874aee483f99a46f7e/DEPS