30 minutes is an extremely long time. Cc'ing a couple folks who work on the layers repo.

(forgot to attach this screenshot of the trace view)

Deleted: Screen Shot 2018-05-22 at 12.10.51 PM.png 318 KB

	Screen Shot 2018-05-22 at 12.10.51 PM.png 318 KB View Download

I've noticed the same thing recently in our CI.  NDK builds of core_validation are taking 4 minutes per ABI on a single workstation thread.

Do you have a chart of when this slow down started?

No, sorry. Should be pretty easy to bisect locally though.

Okay, I'll work on isolating when it started, see if it can be fixed in source.

I don't know what your build is boiling down to, but the NDK build is always slower than desktop clang.  We've gotten it down to 8-10 minutes (in the past) on Travis CI using ccache for the shader toolchain and splitting the ABIs to separate jobs[1].  It has crept up to 20-30 minutes per ABI in the last 6 months.  I notice that x86 Linux builds have also slowed considerably (from <5 minutes to 10-20 mins) so this should be easy to prioritize a fix.

[1] https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/.travis.yml

Awesome. Thanks Cody. Nico is it possible to add Cody's email to the list of contributors? Same would go for ianelliot, tobine, and courtneygo.

From what I understand, https://www.chromium.org/getting-involved/get-bug-editing-privileges says someone just needs to email comitters@chromium.org and ask. Want me to do that?

I'll do it, thanks for sharing that link.

We were able to identify a major compile time regression from about 6 weeks ago, just landed a change that should greatly improve things.  Nico, can you let us know how things looks now?

Just curious, what makes build time such a slow and how did you fix it?

This probably first need to roll into angle and that into chromium before I can check, correct? Is there a tracking bug for that roll?

There was a large static STL map in a header being included in several files.  Making it non-static cleared up the compile time.  It also improved performance since not every file including it was using the new map yet.

https://github.com/KhronosGroup/Vulkan-ValidationLayers/pull/132

Yeah, the repos split, so we have to get that working first, then we'll get the latest source.  Adding blocking bug.

Tobin is helping integrate the new repos with the fix.

Any chance that the definition of the std::maps could be moved to a separate translation unit, completely out of the header file? It's still being included from lots of source files even if most of them now skip over it. I see that several files define VALIDATION_ERROR_MAP_IMPL but I'm not sure why.

I've been trying to move as many static/const objects/arrays from header files as they tend to cause duplicate globals as well as harming compile time, and moving them out entirely is ideal.

The roll from  issue angleproject:2558  landed. Is it possible to check if the compile time sped up on the affected bots?

I kicked off a new linux_arm try job on https://chromium-review.googlesource.com/c/chromium/src/+/1068996

The good news is that the compile did get 14 minutes faster: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_arm/982

The less good news is that while these files build faster now, they still take several orders of magnitude longer than anything else in the build; one takes as long to compile as building all the other files even (we end up waiting 200s for just that file at the end of the build), see the trace:

https://chromium-build-stats.appspot.com/ninja_log/2018/06/21/swarm1937-c4/ninja_log.swarm1937-c4.chrome-bot.20180621-074705.14479.gz/trace.html

Oh wait, I didn't scroll down. We still wait 1100s on these files, not 200. So almost 20 minutes of the build are due to these files still.

Tobin or Cody, any ideas? Is it possible there are headers still being included that have repeated code in them? LVL objects are quite noticeable outliers in that build.

I grabbed the .ninja_log file from https://chromium-build-stats.appspot.com/ninja_log/2018/06/21/swarm1937-c4/ninja_log.swarm1937-c4.chrome-bot.20180621-074705.14479.gz/lastbuild and ran my build summarizing script on it. This avoids having to manually eyeball where the bottlenecks are. The weighted time is a measure of serialization - it's the length of time for a step divided by how many steps are running at a time. 1043.6 s is a long time. Some other vulkan-validation-layers also have long CPU times and weighted times:

>python c:\src\depot_tools\post_build_ninja_summary.py --log-file=ninja_log
    Longest build steps:
           0.7 weighted s to build obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_parameter_validation/parameter_validation.o (54.5 s CPU time)
           0.7 weighted s to build obj/content/common/common/content_message_generator.o (54.9 s CPU time)
           1.2 weighted s to build obj/third_party/sqlite/chromium_sqlite3/sqlite3.o (93.1 s CPU time)
           1.3 weighted s to build nacl_loader_unittests (1.3 s CPU time)
           3.0 weighted s to build newlib_pnacl_nonsfi/nacl_helper_nonsfi_unittests_main, newlib_pnacl_nonsfi/exe.unstripped/nacl_helper_nonsfi_unittests_main (18.7 s CPU time)
           9.4 weighted s to build obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_threading/threading.o (262.7 s CPU time)
          23.4 weighted s to build obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_parameter_validation/parameter_validation_utils.o (318.7 s CPU time)
          29.5 weighted s to build obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_object_tracker/object_tracker_utils.o (337.2 s CPU time)
          52.1 weighted s to build obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_unique_objects/unique_objects.o (382.3 s CPU time)
        1043.6 weighted s to build obj/third_party/angle/third_party/vulkan-validation-layers/VkLayer_core_validation/core_validation.o (1374.4 s CPU time)
    Time by build-step type:
           0.8 s weighted time to generate 95 .h files (65.6 s CPU time)
           0.8 s weighted time to generate 8 .so (linking) files (1.5 s CPU time)
           4.4 s weighted time to generate 14 (no extension found) files (21.5 s CPU time)
          59.0 s weighted time to generate 1190 mojo files (4716.3 s CPU time)
        1493.0 s weighted time to generate 10448 .o files (29400.3 s CPU time)
    1560.8 s weighted time (34330.1 s CPU time, 22.0x parallelism)
    14502 build steps completed, average of 9.29/s

Testing on my Windows workstation I see core_validation.obj file taking ~60 s to build with goma, or ~30 s to build locally.

vulkan-validation-layers\src\layers\core_validation.cpp is quite a big file - ~13,000 lines - and if everything after the last #include is removed then the local compile time drops below 10 s. Most of the rest of the compilation cost comes from #include "core_validation.h"

FWIW.

If I understand correctly: the object is so large that it breaks the sharding algorithm and ends up dominating the build time. Many other threads are underutilized. And it doesn't work well with Goma.

What about breaking this file up into smaller chunks?

I don't think it's _just_ sharding. The file is "only" 13000 lines, which isn't all that much for a source file. (included files probably expand to way more than that in every single cc file we build.) The file probably does something that triggers pathological behavior in the compiler somehow. The trick is probably to find out what exactly happens.

Breaking the file up will make compile times worse in many cases, especially for non-goma users. Note that the jumbo effort is a build configuration that effectively merges translation units in order to improve build times by avoiding redundant processing of shared header files. If this was broken into two .cc files then each one would take roughly twenty seconds to compile (10 s for header files, 10 s for half of the .cc) for a potential savings of ten seconds in some cases and a probable slowdown in many other cases.

I grabbed an ETW trace of compiling core_validation.cpp file on Windows. It looks like the time is roughly evenly split between parsing and backend, but without learning more about clang I can't say much more. I've attached a screenshot showing that. I could share an ETW trace through other channels - it's too big to attach here - or it's fairly easy to record one. I can help explain the process. On Linux I would assume that perf would do the job nicely also - one advantage on Windows is that the symbols for clang-cl.exe are on Chrome's symbol server so they download automatically.

I did enough analysis to determine that memory allocation and freeing use about 9% of CPU time, although it's not clear that that leads to anything useful.

Deleted: Compiling_core_validation.PNG 31.6 KB

	Compiling_core_validation.PNG 31.6 KB View Download

The file doesn't change very often. Monthly? You technically don't need to compile it every time. Just when the repos have been rolled forward.

Bots only rebuild things if they don't have an up-to-date version. However, linux_arm isn't a default-cq bot, and so it's likely that it only gets a few builds per month and doesn't have this file lying around.

Checking in .o files for all the build configs we support isn't going to happen.

It's not reasonable to have files that take 20 min to compile. I'm sure there's something that can be done to make this build way faster.

The file only takes ~2 minutes to compile on my gLinux workstation. The entire project compiles on Travis in ~13 minutes, which is in line with other projects
https://travis-ci.org/KhronosGroup/Vulkan-ValidationLayers/jobs/395463719

I agree the compile can probably be sped up, but I think there might be something compounding the problem here. Is everything else in the build cached making validation stand out?

I don't think we're out of line with other vulkan tools. Are you seeing problems with them as well? For instance, spirv-tools takes ~11 mins:
https://travis-ci.org/KhronosGroup/SPIRV-Tools/jobs/395462110

Are you guys using ccache? That can help tremendously.

I don't know if we build these.

Maybe it's made worse when targeting arm? You're probably building for x86?

No, the numbers I'm quoting are targeting armeabi-v7a using a relatively new NDK, usually about one release behind latest public.

Maybe your build flags are different? We use -Oz, and a bunch of other flags.

I'm seeing this when building the official NDK (rather than upstream development). There does seem to be something pathological going on. It is probably build flag related. I'll raise the priority of this with LunarG.

Thanks Cody!

Correction, I removed mips/mips64 from our target ABI list and build time dropped back to ~2.5 minutes instead of (what felt like) half an hour. Is your build by chance hitting mips, which has been removed from the NDK as of r17?

No, it's doing a regular chrome build for arm.

If you have a chrome checkout, here are the args.gn:

https://logs.chromium.org/v/?s=chromium%2Fbuildbucket%2Fcr-buildbucket.appspot.com%2F8943102479427226544%2F%2B%2Fsteps%2Fgenerate_build_files__with_patch_%2F0%2Fstdout


dcheck_always_on = true
is_component_build = false
is_debug = false
strip_absolute_paths_from_debug_symbols = true
symbol_level = 1
target_cpu = "arm"
use_goma = true

I had core_validation.cpp take 8 minutes and 1.6g of RAM before I killed it because I thought clang had hung.

I realise you only need to compile it when you get a cache miss, but even so, this is a terrible productivity killer.

Probably the file could be bisected to find out where the time is being spent.

On https://chromium-build-stats.appspot.com/ninja_log/2018/10/25/swarm2173-c4/ninja_log.swarm2173-c4.chrome-bot.20181025-075020.16472.gz/trace.html it took over 3 minutes on Windows, so this isn't arm-only or anything.

This issue is finally getting some attention upstream, I'll update when fixes/workarounds have been applied.

Cool, thanks Cody.

If that file has large tables of not-constexpr-constructible objects, this might apply: http://llvm.org/viewvc/llvm-project?view=revision&revision=345329

(i.e. maybe a similar change can be made in the vklayer code)

@thakis, there is one large table there, but its initializer list contains only hard coded C-strings and function pointers. That's probably not the issue here.

That file is 14k lines of C++ code though, so it's no surprise it compiles so slowly. They could simply chop up that file into multiple files, exactly the same way they've already separated buffer functions for example.

Shabi, there's something weird happening with these particular files. gles2_cmd_decoder.cc is about 20k LoC and seems to compile pretty quickly AFAIK:

https://cs.chromium.org/chromium/src/gpu/command_buffer/service/gles2_cmd_decoder.cc

Interesting. I'm curious to know what would end up being the real problem, but a guess would be that compared to gles2_cmd_decoder, core_validation.cpp is full of  unordered_map and other complex templates, so the code looks shorter but there is more to compile and especially optimize.

I'd try bisecting: Delete half the file and look at compile time, do same for other half. One half is probably much slower. Keep that half, cut it in half again and repeat, until you've found the part of the file that's slow to build.

Clang is having a hard time filling the map of validation errors in https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/layers/vk_validation_error_messages.h

Tracked internally here: https://buganizer.corp.google.com/issues/118397036

We're getting workarounds applied upstream in the meantime.

That's exactly the situation from comment 42, right? Just making an array of const char* pairs and then building the map from that at runtime should do the trick, I think.

Yeah, comment 42 applies.

Workaround applied upstream, should see improvement after the following commit:
https://github.com/KhronosGroup/Vulkan-ValidationLayers/commit/4d9b2f82b04efa37c1b2695bf5a8621cbd1e3aeb

Awesome Cody. Let's roll the layers in ANGLE.

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/d00a9a1447d48299800ac62b8955c861cd362d10

commit d00a9a1447d48299800ac62b8955c861cd362d10
Author: Tobin Ehlis <tobine@google.com>
Date: Mon Oct 29 19:53:33 2018

Vulkan: Roll VK deps forward

Roll VK Headers/Tools/Loader/ValidationLayers forward.
There's a build improvement as well as many new checks.

Added new source file dep, convert_to_renderpass2.cpp, used by core_
validation for added RenderPass2 checks.

Bug:  angleproject:2937 
Bug: chromium:845519
Change-Id: Id5c30c5bfc684c365a8ad97c3289bccc02afb3b3
Reviewed-on: https://chromium-review.googlesource.com/c/1305673
Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Tobin Ehlis <tobine@google.com>

[modify] https://crrev.com/d00a9a1447d48299800ac62b8955c861cd362d10/third_party/vulkan-validation-layers/BUILD.gn
[modify] https://crrev.com/d00a9a1447d48299800ac62b8955c861cd362d10/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dffd4c21bd1bdb66f8058df88174f2a53983df6e

commit dffd4c21bd1bdb66f8058df88174f2a53983df6e
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Oct 30 01:11:33 2018

Roll src/third_party/angle 0da64b5e1009..3420dcb0cff1 (5 commits)

https://chromium.googlesource.com/angle/angle.git/+log/0da64b5e1009..3420dcb0cff1


git log 0da64b5e1009..3420dcb0cff1 --date=short --no-merges --format='%ad %ae %s'
2018-10-29 ynovikov@chromium.org Revert "Roll DEQP version forward"
2018-10-29 syoussefi@chromium.org Vulkan: Remaining disjoint timer query functions
2018-10-29 jmadill@chromium.org Remove Context::gatherParams.
2018-10-29 tobine@google.com Roll DEQP version forward
2018-10-29 tobine@google.com Vulkan: Roll VK deps forward


Created with:
  gclient setdep -r src/third_party/angle@3420dcb0cff1

The AutoRoll server is located here: https://autoroll.skia.org/r/angle-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:845519
TBR=ynovikov@chromium.org

Change-Id: I52228a106f71061d97e67e79b3259dd543b60817
Reviewed-on: https://chromium-review.googlesource.com/c/1306219
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#603715}
[modify] https://crrev.com/dffd4c21bd1bdb66f8058df88174f2a53983df6e/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/7cba6be85a1468b113f64c5a8beff4dde4ae8217

commit 7cba6be85a1468b113f64c5a8beff4dde4ae8217
Author: Yuly Novikov <ynovikov@chromium.org>
Date: Tue Oct 30 15:38:13 2018

Revert "Vulkan: Roll VK deps forward"

This reverts commit d00a9a1447d48299800ac62b8955c861cd362d10.

Reason for revert: Suspect for breaking Debug bots.
Original change's description:
> Vulkan: Roll VK deps forward
> 
> Roll VK Headers/Tools/Loader/ValidationLayers forward.
> There's a build improvement as well as many new checks.
> 
> Added new source file dep, convert_to_renderpass2.cpp, used by core_
> validation for added RenderPass2 checks.
> 
> Bug:  angleproject:2937 
> Bug: chromium:845519
> Change-Id: Id5c30c5bfc684c365a8ad97c3289bccc02afb3b3
> Reviewed-on: https://chromium-review.googlesource.com/c/1305673
> Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
> Reviewed-by: Jamie Madill <jmadill@chromium.org>
> Commit-Queue: Tobin Ehlis <tobine@google.com>

TBR=ynovikov@chromium.org,jmadill@chromium.org,cnorthrop@google.com,tobine@google.com,syoussefi@chromium.org

Change-Id: I5c48315dd8c80510e7dea54cb384c08dce5b6bcb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  angleproject:2937 , chromium:845519
Reviewed-on: https://chromium-review.googlesource.com/c/1307695
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>

[modify] https://crrev.com/7cba6be85a1468b113f64c5a8beff4dde4ae8217/third_party/vulkan-validation-layers/BUILD.gn
[modify] https://crrev.com/7cba6be85a1468b113f64c5a8beff4dde4ae8217/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/78d5c8d9bf4b90181e5e253e7b49e27169c5228b

commit 78d5c8d9bf4b90181e5e253e7b49e27169c5228b
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Oct 30 17:09:24 2018

Roll src/third_party/angle 3420dcb0cff1..7cba6be85a14 (1 commits)

https://chromium.googlesource.com/angle/angle.git/+log/3420dcb0cff1..7cba6be85a14


git log 3420dcb0cff1..7cba6be85a14 --date=short --no-merges --format='%ad %ae %s'
2018-10-30 ynovikov@chromium.org Revert "Vulkan: Roll VK deps forward"


Created with:
  gclient setdep -r src/third_party/angle@7cba6be85a14

The AutoRoll server is located here: https://autoroll.skia.org/r/angle-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:845519
TBR=ynovikov@chromium.org

Change-Id: I8968557636a6c70cf7cda4ecc30d3f1df8ae7a37
Reviewed-on: https://chromium-review.googlesource.com/c/1307814
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#603934}
[modify] https://crrev.com/78d5c8d9bf4b90181e5e253e7b49e27169c5228b/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/angle/angle/+/317a9ebdb019b02fda925072a5572265826676e9

commit 317a9ebdb019b02fda925072a5572265826676e9
Author: Tobin Ehlis <tobine@google.com>
Date: Fri Nov 30 16:21:53 2018

Vulkan: Roll VK deps forward

Roll VK Headers/Loader/ValidationLayers forward.
There's a build improvement as well as many new checks.

Added new source file dep, convert_to_renderpass2.cpp, used by core_
validation for added RenderPass2 checks.

Also had to move glslang fwd for shader validation layer so updated
from confusing upstream branch of glslang that's integrated with shaderc
to the independent Khronos glslang repo.

Bug:  angleproject:2937 
Bug: chromium:845519
Change-Id: Ifee911f061dc426404c09499e49c2f029b174d03
Reviewed-on: https://chromium-review.googlesource.com/c/1355508
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Tobin Ehlis <tobine@google.com>

[modify] https://crrev.com/317a9ebdb019b02fda925072a5572265826676e9/third_party/vulkan-validation-layers/BUILD.gn
[modify] https://crrev.com/317a9ebdb019b02fda925072a5572265826676e9/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ac0f3e4c889024247e08e21e6c2c6ec943ea2db9

commit ac0f3e4c889024247e08e21e6c2c6ec943ea2db9
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Nov 30 17:47:54 2018

Roll src/third_party/angle b51eb09fb5bb..317a9ebdb019 (1 commits)

https://chromium.googlesource.com/angle/angle.git/+log/b51eb09fb5bb..317a9ebdb019


git log b51eb09fb5bb..317a9ebdb019 --date=short --no-merges --format='%ad %ae %s'
2018-11-30 tobine@google.com Vulkan: Roll VK deps forward


Created with:
  gclient setdep -r src/third_party/angle@317a9ebdb019

The AutoRoll server is located here: https://autoroll.skia.org/r/angle-chromium-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:845519
TBR=fjhenigman@chromium.org

Change-Id: I6c7a2da32e78a857475f89b720cd3d05e04b1b70
Reviewed-on: https://chromium-review.googlesource.com/c/1356632
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#612688}
[modify] https://crrev.com/ac0f3e4c889024247e08e21e6c2c6ec943ea2db9/DEPS

Alrighty, the fix has finally rolled in. Can anyone cc'ed on this issue confirm later on if this issue appears to be fixed?

I looked at a few bots to check and discovered that the vulkan roll regressed something:  issue 910699 

@thakis, thanks for filing & fixing the issue. Can you check if the slow build times with core_validation.o is fixed with the new roll?

These files look better now.

But third_party/angle/third_party/vulkan-validation-layers/angle/vulkan/object_tracker.cpp takes over a minute on several toolchains on 

https://chromium-build-stats.appspot.com/ninja_log/2018/12/13/swarm2173-c4/ninja_log.swarm2173-c4.chrome-bot.20181213-094237.5396.gz/trace.html

And unique_objects_generator.py also takes a long time to run.

Cool, thanks for the trace. object_tracker.cpp is probably because of const std::unordered_map<std::string, void*> name_to_funcptr_map. I think we could change that to const char *. Cody can you look into using the same technique you did for the other map?

There's also a similar or identical maps in thread_check.h and in unique_object_wrappers.h that are showing up in the profile. Not sure if those are from generated code. But we could fix all those instances too.

unique_objects_generator we could run as part of the roll process. This would take a bit more custom work. We have a similar problem filed under issue angleproject:2449 so I think it would make more sense to include it in there.

As an update, we think it is the python script taking the most time.  For instance, the largest bubble I see is named "win_clang_nacl_win64/gen/third_party/angle/third_party/vulkan-validation-layers/angle/vulkan/unique_objects_wrappers.h" and is taking 120,730 ms to finish.

thakis@, do you know if the runtime in your trace in in comment #64 is from a python script or from the same issue with unordered_map object generation?

I see long tasks for win_clang_nacl_win64/gen/third_party/angle/third_party/vulkan-validation-layers/angle/vulkan/unique_objects_wrappers.h, win_clang_nacl_win64/gen/third_party/angle/third_party/vulkan-validation-layers/angle/vulkan/object_tracker.cpp,  	
gen/third_party/angle/third_party/vulkan-validation-layers/angle/vulkan/unique_objects_wrappers.h and gen/third_party/angle/third_party/vulkan-validation-layers/angle/vulkan/object_tracker.cpp . Is there a way to know if this is from the python code generation or from the compilation? Not sure how you figure out that unique_objects_generator.py was taking a long time in comment 65.

I think the entries are named for the edge output, so compiles are labeled .o. So I think this being the python script sounds right.

Thanks. That's good to know. Blocking this on the ANGLE issue to prebake the generated files.