Add linux_chromium_tsan_rel_ng to CQ |
|||||||||||||||||
Issue descriptionfdoray made the bot green a few weeks ago by solving issues 665945 , 665996 , and 663850 . But it's red again (issues 672812 , 672852 , 587199 , 672859 ). This bot is actually really good (I often include it on my CLs -- CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng), but it's unfortunate that it's very often red (it's been red in > last 200 runs according to logs, i.e. > 10 days). It's fast enough for CQ I think as well and given TSAN errors keep sneaking in and there doesn't appear to be good monitoring of the bot, it'd be worthwhile to add to CQ IMO. Any idea why it's not already on CQ? Is it a matter of scale in the lab? If so, how can we make sure it's clearly assigned to a sheriff rotation? We've
,
Dec 13 2016
,
Dec 13 2016
,
Dec 14 2016
,
Dec 15 2016
,
Dec 15 2016
Bot is now green, working on getting it on CQ, yay!
,
Dec 16 2016
Presumably this would move from chromium.memory.full waterfall to main waterfall before it's on the CQ right?
,
Dec 16 2016
Perhaps? +dpranke? The longer it's not on CQ, the more manual monitoring I need to do..
,
Dec 16 2016
Yes, we should move it to the main waterfall.
,
Dec 20 2016
From dpranke@ offline, the steps are: 1) post a CL to the build repo that moves linux_chromium_tsan_rel_ng from the 'optionalbuilders2' pool to the 'cq_builders' pool in slaves.cfg: https://cs.chromium.org/chromium/build/masters/master.tryserver.chromium.linux/slaves.cfg?q=linux_chromium_tsan_rel_ng+slaves.cfg&sq=package:chromium&dr=C&l=7 2) add a 'linux_chromium_tsan_rel_ng' entry to: https://cs.chromium.org/chromium/src/infra/config/cq.cfg?rcl=0&l=56 mirroring the 'headless' entry (so it'll run as a 10% experiment). Watch that run for a while to make sure the builder is stable. 3) Up the experiment percentage to 50%. Watch. Confirm w/ infra that we have the capacity to do 100% (should be fine at the moment). 4) Remove the experiment line, so that it's a full member of the CQ.
,
Dec 20 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/0588337c19680108b3c858be14f3660d1aa40886 commit 0588337c19680108b3c858be14f3660d1aa40886 Author: Gabriel Charette <gab@chromium.org> Date: Tue Dec 20 18:44:02 2016 Promote linux_chromium_tsan_rel_ng to a CQ builder. BUG= 672860 Change-Id: Iba7b1ff58112b899e00236b96c86ef2f1c624bca Reviewed-on: https://chromium-review.googlesource.com/422473 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Gabriel Charette <gab@chromium.org> [modify] https://crrev.com/0588337c19680108b3c858be14f3660d1aa40886/masters/master.tryserver.chromium.linux/master.cfg [modify] https://crrev.com/0588337c19680108b3c858be14f3660d1aa40886/masters/master.tryserver.chromium.linux/slaves.cfg
,
Dec 21 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/a4dc1bd4c41dd72b2fa83bed1047db1072b30554 commit a4dc1bd4c41dd72b2fa83bed1047db1072b30554 Author: Gabriel Charette <gab@chromium.org> Date: Wed Dec 21 19:28:04 2016 Move linux_chromium_tsan_rel_ng to main chromium.memory waterfall. BUG= 672860 Change-Id: I3ac7513bbe3ca18423ee6524829d5883726f4c24 Reviewed-on: https://chromium-review.googlesource.com/422437 Commit-Queue: Gabriel Charette <gab@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/masters/master.chromium.memory.full/master.cfg [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/masters/master.chromium.memory.full/slaves.cfg [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/masters/master.chromium.memory/master_linux_cfg.py [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/masters/master.chromium.memory/slaves.cfg [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipe_modules/chromium_tests/chromium_memory.py [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipe_modules/chromium_tests/chromium_memory_full.py [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipe_modules/chromium_tests/trybots.py [rename] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipes/chromium.expected/full_chromium_memory_Linux_TSan_Builder.json [rename] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipes/chromium.expected/full_chromium_memory_Linux_TSan_Tests.json [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipes/chromium.expected/tsan.json [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipes/chromium.py [modify] https://crrev.com/a4dc1bd4c41dd72b2fa83bed1047db1072b30554/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_linux_linux_chromium_tsan_rel_ng.json
,
Dec 21 2016
,
Dec 22 2016
,
Dec 23 2016
So the above two CLs landed and all masters were restarted (master.tryserver.chromium.linux, master.chromium.memory, and master.chromium.memory.full), but Linux TSAN Builder is now offline..? https://build.chromium.org/p/chromium.memory/builders/Linux%20TSan%20Builder
,
Dec 23 2016
Problem appears to be with vm13-m1 being offline (or at least not picking up builds). It was associated with previous "Chromium Linux TSan Builder" on chromium.memory.full and now associated with "Linux TSan Builder" on chromium.memory. https://build.chromium.org/p/chromium.memory/buildslaves/vm13-m1 Could it be because the builder's name changed (dropped "Chromium")?
,
Dec 23 2016
Could be, the restart might've just caught things at a weird time. File a bug @ go/bug-a-trooper to get it restarted.
,
Dec 23 2016
,
Dec 28 2016
,
Jan 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1933025de3e1564c12bf7f1b60ca7add98bb62c4 commit 1933025de3e1564c12bf7f1b60ca7add98bb62c4 Author: gab <gab@chromium.org> Date: Thu Jan 05 19:06:16 2017 Add linux_chromium_tsan_rel_ng to 10% of CQs. BUG= 672860 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng Review-Url: https://codereview.chromium.org/2596533002 Cr-Commit-Position: refs/heads/master@{#441721} [modify] https://crrev.com/1933025de3e1564c12bf7f1b60ca7add98bb62c4/infra/config/cq.cfg
,
Jan 10 2017
Folks, I may be missing something, but there appears to be no per-test TSan logs at https://build.chromium.org/p/chromium.memory/waterfall?builder=Linux%20TSan%20Tests How the devs are supposed to use it?
,
Jan 10 2017
Interesting, hadn't noticed that... is it not running the tests? Another missing config? The matching try bot definitely does run the tests that I'm sure (or at least I'm sure it did before I touched the configs, haven't used it since).
,
Jan 10 2017
,
Jan 11 2017
@dpranke: any idea (re #21-22)?
,
Jan 12 2017
You need to move the "Linux TSan Tests" entries to the chromium.memory.json file.
,
Jan 13 2017
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f99432e478be31dc7e2f63326dca8a542375793e commit f99432e478be31dc7e2f63326dca8a542375793e Author: gab <gab@chromium.org> Date: Fri Jan 13 23:23:42 2017 Move Linux TSan Tests config to chromium.memory.json Also cleaning up chromium.full.json and chromium.memory.fyi.json which I think are no longer used (or at least don't run TSAN?) BUG= 672860 # Commenting out because it's catching known http://crbug.com/674545 which is being fixed. # CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng Review-Url: https://codereview.chromium.org/2621143006 Cr-Commit-Position: refs/heads/master@{#443720} [modify] https://crrev.com/f99432e478be31dc7e2f63326dca8a542375793e/testing/buildbot/chromium.full.json [modify] https://crrev.com/f99432e478be31dc7e2f63326dca8a542375793e/testing/buildbot/chromium.memory.full.json [modify] https://crrev.com/f99432e478be31dc7e2f63326dca8a542375793e/testing/buildbot/chromium.memory.fyi.json [modify] https://crrev.com/f99432e478be31dc7e2f63326dca8a542375793e/testing/buildbot/chromium.memory.json
,
Jan 14 2017
,
Jan 16 2017
,
Jan 18 2017
Re. #21 (@glider): fixed after master restart, woohoo, thanks for noticing!
,
Jan 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a0f0a395c1604aadbfe97a5815044fbd245d7a11 commit a0f0a395c1604aadbfe97a5815044fbd245d7a11 Author: gab <gab@chromium.org> Date: Thu Jan 19 19:17:21 2017 Up TSAN bot CQ experiment to 50% It's been very stable for 24 hours @ 10%: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_tsan_rel_ng BUG= 672860 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng Review-Url: https://codereview.chromium.org/2617823004 Cr-Commit-Position: refs/heads/master@{#444813} [modify] https://crrev.com/a0f0a395c1604aadbfe97a5815044fbd245d7a11/infra/config/cq.cfg
,
Jan 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2f08dd5ad90c907e1575be2e67ae7b18fc062ef0 commit 2f08dd5ad90c907e1575be2e67ae7b18fc062ef0 Author: gab <gab@chromium.org> Date: Tue Jan 24 02:27:48 2017 Up Linux TSAN CQ experiment to 100% Builders/testers are scaling well (no pending build requests backlog): https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_tsan_rel_ng BUG= 672860 R=dpranke@chromium.org Review-Url: https://codereview.chromium.org/2649153002 Cr-Commit-Position: refs/heads/master@{#445610} [modify] https://crrev.com/2f08dd5ad90c907e1575be2e67ae7b18fc062ef0/infra/config/cq.cfg
,
Jan 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4afe10c428a371521326f42b312cdcdf9e87a7dd commit 4afe10c428a371521326f42b312cdcdf9e87a7dd Author: gab <gab@chromium.org> Date: Wed Jan 25 21:51:22 2017 Augment content_browsertests to 5 shards for Linux TSan Tests R=dpranke@chromium.org BUG= 672860 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng Review-Url: https://codereview.chromium.org/2651253002 Cr-Commit-Position: refs/heads/master@{#446129} [modify] https://crrev.com/4afe10c428a371521326f42b312cdcdf9e87a7dd/testing/buildbot/chromium.memory.json
,
Jan 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5c98db656aa52625137af840e098bbce26db7010 commit 5c98db656aa52625137af840e098bbce26db7010 Author: gab <gab@chromium.org> Date: Thu Jan 26 12:41:40 2017 Enable linux_chromium_tsan_rel_ng on CQ for realz. It's working great @ 100%, time to make it real: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_tsan_rel_ng BUG= 672860 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng Review-Url: https://codereview.chromium.org/2652923003 Cr-Commit-Position: refs/heads/master@{#446308} [modify] https://crrev.com/5c98db656aa52625137af840e098bbce26db7010/infra/config/cq.cfg
,
Jan 26 2017
@gab - please don't link to internal sites like viceroy w/o making sure that the R-V-G label is applied. You can use shortened links if you don't want to use the label.
,
Jan 26 2017
Looks like browser_tests, cc_unittests, and interactive_ui_tests are still problematic: https://codereview.chromium.org/2654283002/. I will call this bug fixed per TSan running on CQ and follow-up on the other bugs to figure out next steps there.
,
Jan 26 2017
,
Jan 26 2017
[-RVG] Oops my bad, re-posting comment #36 properly: PS: CQ is now running linux_chromium_tsan_rel_ng on all runs by default :-O!! Links for reference (and to keep an eye on in first couple days): - the CQ try bot: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_tsan_rel_ng - its throughput: https://goto.google.com/dzaaq
,
Feb 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8f6e755711ae6d4d6640c409e462cdd048c7cfe8 commit 8f6e755711ae6d4d6640c409e462cdd048c7cfe8 Author: danakj <danakj@chromium.org> Date: Fri Feb 10 17:33:07 2017 Enable cc_unittests for tsan bot now that they pass. R=gab@chromium.org BUG= 437454 , 672860 Review-Url: https://codereview.chromium.org/2688033002 Cr-Commit-Position: refs/heads/master@{#449647} [modify] https://crrev.com/8f6e755711ae6d4d6640c409e462cdd048c7cfe8/testing/buildbot/chromium.memory.json |
|||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||
Comment 1 by gab@chromium.org
, Dec 9 2016