Issue metadata
Sign in to add a comment
|
Linux MSan bots are taking too long to compile |
||||||||||||||||||||||
Issue descriptionThe Linux MSan (https://build.chromium.org/p/client.webrtc/builders/Linux%20MSan) and linux_msan (https://build.chromium.org/p/tryserver.webrtc/builders/linux_msan/builds/) bots are taking too long to compile. The compile step used to take usually less than 5 minutes, and now it takes almost an hour. For the trybot it seems like the problem began with this build (it takes 1h 4m 28s to compile!): https://build.chromium.org/p/tryserver.webrtc/builders/linux_msan/builds/16150 For the bot in the main waterfall it seems like it began with: https://build.chromium.org/p/client.webrtc/builders/Linux%20MSan/builds/7775 (1h 1m 36s to compile) Or perhaps with the build before: https://build.chromium.org/p/client.webrtc/builders/Linux%20MSan/builds/7774
,
Jan 20 2017
Another data point, both https://build.chromium.org/p/client.webrtc/builders/Linux%20MSan/builds/7777 and https://build.chromium.org/p/client.webrtc/builders/Linux%20MSan/builds/7781 take place on slave16-c3. So even though the latter build is just 725 objects, it takes 51 minutes to build (both after the roll)
,
Jan 20 2017
This has severe impact on our tryserver and CQ performance, since these slow bots have become the bottleneck and jobs gets queued up waiting for these VMs to finish.
,
Jan 20 2017
,
Jan 20 2017
Note that the last commit to MSan is 286980 from 2 months ago, while the Clang version change is 289575:289944. Maybe something has changed in the register allocator? It usually bears much more stress under MSan than in native builds.
,
Jan 20 2017
A graph that shows the increase in build times for linux_msan https://viceroy.corp.google.com/chrome_infra/Buildbot/overview_v2?builder=linux_msan&duration=5529596&master=master.tryserver.webrtc&refresh=-1#_VG_-H3Naic3
,
Jan 20 2017
Kostya, Nico, there is ongoing discussion about LLVM getting slower on llvm-dev. Any idea if the issues mentioned there are related to this particular bug?
,
Jan 23 2017
,
Jan 23 2017
I see 93% compile time regression on sqlite.c in the following clang revision range: 289575:289944. I'll narrow it down and file an upstream bug.
,
Jan 23 2017
I also see that it is fixed in the current ToT, as well as in the possible next roll candidate, r292414 (and they are even 1 or 2% faster on the two files I've tested). So, I guess we wait for the clang roll. Assigning to Nico since I could not find a tracking bug for that.
,
Jan 23 2017
Can you file a bug for the roll and make this one blocked on that?
,
Jan 25 2017
,
Jan 25 2017
,
Jan 27 2017
Is there an estimated time?
,
Mar 1 2017
clang roll landed in chromium. i think you need to merge that to webrtc, and then let us know if it helped.
,
Mar 1 2017
Yay, the bots are now cycling way faster (~4x), so this is definitely fixed now! Thanks all! |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by ehmaldonado@chromium.org
, Jan 20 2017