I am getting close to fix all existing failures. It seems like a perfect time to setup a ubsan trybot.

Initially, I will make it an optional, but the goal is to make it triggered for each incoming CL.

Nico, do you have any objections?  Any known issues I need to address prior to doing that? 

No objections to an optional trybot. Making it triggered for each incoming CL has a higher bar though: It requires eng review (see https://www.chromium.org/developers/testing/chromium-build-infrastructure/tour-of-the-chromium-buildbot "Adding new build configurations and tests to the main Chromium waterfall & Commit Queue"), and default CQ also needs enough hardware resources (which for Linux bots should be doable).

Thank you for the link. I think it's what I wanted to hear.

I will finish my cleanups, set up an optional trybot, measure time (and reduce it, if needed), and then try to go through that review.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89

commit 3c0d76d7ccbc0957ab0edf78333b77661b8ffe89
Author: krasin <krasin@chromium.org>
Date: Thu Nov 03 00:27:20 2016

Add a config for optional UBSan trybot.

The plan is to eventually make it a part of CQ, but the first
step is to set it up as optional, measure performance and
other things, and then go through a technical review for getting
the trybot into the CQ.

BUG=612594

Review-Url: https://codereview.chromium.org/2468363002

[modify] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/masters/master.tryserver.chromium.linux/master.cfg
[modify] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/masters/master.tryserver.chromium.linux/slaves.cfg
[modify] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/scripts/slave/recipe_modules/chromium_tests/trybots.py
[add] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_linux_chromium_ubsan_rel_ng.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89

commit 3c0d76d7ccbc0957ab0edf78333b77661b8ffe89
Author: krasin <krasin@chromium.org>
Date: Thu Nov 03 00:27:20 2016

Add a config for optional UBSan trybot.

The plan is to eventually make it a part of CQ, but the first
step is to set it up as optional, measure performance and
other things, and then go through a technical review for getting
the trybot into the CQ.

BUG=612594

Review-Url: https://codereview.chromium.org/2468363002

[modify] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/masters/master.tryserver.chromium.linux/master.cfg
[modify] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/masters/master.tryserver.chromium.linux/slaves.cfg
[modify] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/scripts/slave/recipe_modules/chromium_tests/trybots.py
[add] https://crrev.com/3c0d76d7ccbc0957ab0edf78333b77661b8ffe89/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_linux_chromium_ubsan_rel_ng.json

An optional UBSan trybot should appear once the tryserver master is restarted.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/dc6d4c8e8bc30ee2937a1525aef64e80af2a3dc6

commit dc6d4c8e8bc30ee2937a1525aef64e80af2a3dc6
Author: phajdan.jr <phajdan.jr@chromium.org>
Date: Thu Nov 03 12:22:40 2016

tryserver.chromium.linux: move linux_asan_rel_ng back to CQ pool

It seems to have been moved out by mistake in https://codereview.chromium.org/2468363002/

BUG=661974, 612594

Review-Url: https://codereview.chromium.org/2477673002

[modify] https://crrev.com/dc6d4c8e8bc30ee2937a1525aef64e80af2a3dc6/masters/master.tryserver.chromium.linux/slaves.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/41b70014bbe50f899125b5f00a464a6b68d0427b

commit 41b70014bbe50f899125b5f00a464a6b68d0427b
Author: krasin <krasin@chromium.org>
Date: Mon Nov 07 23:54:14 2016

Add mb_config for the new ubsan try bot.

As of now, it refers to just is_ubsan_vptr, but I aim to add other ubsan bits
as well (in particular, is_ubsan_null).

BUG=612594

Review-Url: https://codereview.chromium.org/2473953007
Cr-Commit-Position: refs/heads/master@{#430426}

[modify] https://crrev.com/41b70014bbe50f899125b5f00a464a6b68d0427b/tools/mb/mb_config.pyl

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f37d4947c6fd5ccb4189f2e24a6f0031ee37676d

commit f37d4947c6fd5ccb4189f2e24a6f0031ee37676d
Author: krasin <krasin@chromium.org>
Date: Thu Dec 01 21:15:26 2016

Fix invalid casts from ComplexFeature to SimpleFeature.

UBSan and CFI made the following invalid cast:
../../tools/json_schema_compiler/test/features_generation_unittest.cc:102:12: runtime error: downcast of address 0x1f55b1c64040 which does not point to an object of type 'APIFeature' (aka 'extensions::SimpleFeat
ure')
0x1f55b1c64040: note: object is of type 'extensions::ComplexFeature'

It is currently impossible to catch regressions like these with the CQ, but there is a work on getting a UBSan Vptr trybot there.

BUG=612594, 670404 

Review-Url: https://codereview.chromium.org/2542163002
Cr-Commit-Position: refs/heads/master@{#435713}

[modify] https://crrev.com/f37d4947c6fd5ccb4189f2e24a6f0031ee37676d/tools/json_schema_compiler/test/features_generation_unittest.cc