+some ninja people

The limit is 32 separators after canonicalization (i.e. removing "foo/.."). If we hit that, something else is broken. We shouldn't have paths that deep.

The compiler error in that log is:

[1016/2303] CXX obj/third_party/WebKit/Source/core/layout/svg/svg_layout/svg_layout_jumbo_1.obj
FAILED: obj/third_party/WebKit/Source/core/layout/svg/svg_layout/svg_layout_jumbo_1.obj 
ninja -t msvc -e environment.x86 -- e:\b\s\w\ir\cache\goma_client/gomacc.exe "e:\b\s\w\ir\cache\win_toolchain\vs_files\f53e4598951162bad6330f7a167486c7ae5db1e5\vc\bin\amd64_x86/cl.exe" /nologo /showIncludes /FC @obj/third_party/WebKit/Source/core/layout/svg/svg_layout/svg_layout_jumbo_1.obj.rsp /c gen/third_party/WebKit/Source/core/layout/svg/svg_layout_jumbo_1.cc /Foobj/third_party/WebKit/Source/core/layout/svg/svg_layout/svg_layout_jumbo_1.obj /Fd"obj/third_party/WebKit/Source/core/layout/svg/svg_layout_cc.pdb"
svg_layout_jumbo_1.cc

[snip]

Note: including file: e:\b\s\w\ir\cache\builder\win\src\out\release\gen\third_party\webkit\source\core\layout\svg\../../../../../../../../../third_party/WebKit/Source/core/layout/svg/SVGTextQuery.cpp
Note: including file:  ../../third_party/WebKit/Source\core/layout/svg/SVGTextQuery.h
too many path components[1017/2303] ACTION //chrome/browser/resources/settings:unpak(//build/toolchain/win:x86)

Which is 32 directories exactly. However, obviously, that's not canonicalized. Maybe that's more of a problem with the 
which makes me think that either canonicalization isn't working, or there's a problem in a script somewhere.

The fact that we're building "jumbo" things here makes me think that this is a tryjob from bratell's "jumbo" build work, and so maybe the jumbo script also needs to be canonicalizing things.

However, I'm not sure how to go from this build log back to a matching Gerrit issue to check that.

SwarmBucket hasn't implemented LogDog back-references yet; I'm told it's on the backburner. Currently, the way to get back is:

1) Notice that the log name is built around a BuildBucket ID: 8974709182564061120
2) Use the BuildBucket API to resolve the build: https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.get?id=8974709182564061120&_h=1&
3) Look at the "url" field: https://luci-milo.appspot.com/swarming/task/3736d8cd43fa9910

Yup, that's a jumbo build test patch, so good news is that (for the moment) no committed changes should hit this problem.

ah nice, that explains why I haven't seen this more than a handful of times.

Are there any recent references? I did a small change that sometimes shortens the path by one which just squeezed past the limit, but I have no further *good* workaround for now. Not as long as the generated files are in out/gen/<tree> and the source files are not there.

If the updated ninja is deployed this week I might be able to close this.

See the other bug; we souldn't run into this with current ninja either. Rather than blindly raising limits, we should understand why this happens and fix that.

RE #7, the scenario where we encountered is on SwarmBucket, an alternative build environment that we are intending to move builds to. This environment nests starting directories a few levels deeper than the current BuildBot environment:

BuildBot would be: E:\b\c\builder\win
Start: e:\b\s\w\ir\cache\builder\win

This isn't something that's going to change, but it does explain why this issue was encountered here and not in the man tryjob builds.

That said, you're riding this close to the delimiter limit regardless. Not sure what a good way to fix it on your end is.

RE #8, can't it be both? It looks like a PR to raise the limit to 64 landed over a year ago. Do we just need to bump the Ninja version in Chromium to pick it up, or are there other implications?

> This isn't something that's going to chang

Why not?

> limit to 64 landed over a year ago

No, it landed a month or two ago. The pull request has just been around longer.

We just need to bump the ninja version to get that change, but if we need more than 32 directories for chromium builds we're doing something else wrong.

> Why not?

The SwarmBucket build environment isn't going to fundamentally change its pathing scheme. The path choice is already deliberate, with each nested directory delineating ownership and expected accesses and behaviors. Rearranging it to shave off path delimiters to mollify a tool that limits them also seems like the wrong focus for this problem. SwarmBucket consumes 3 more delimiters than BuildBucket, for a total of 7. I don't think this is an unreasonable thing to do.

> We just need to bump the ninja version to get that change, but if we need more than 32 directories for chromium builds we're doing something else wrong.

I don't disagree. In SwarmBucket, Chromium has 26 delimiters to work with, which seems like a sufficiently large number. The correct solution here is for the build to not require so many back references, so in that sense we are doing something wrong.

On the other hand, the build environment consumes resources, and path delimiters are apparently semi-precious resources. Our directory scheme wasn't aware of this, but I don't think it's being oppressive or abusive in its choice to use 7 delimiters. Since a version of Ninja exists that will double the available path delimiter resources across all builds, it seems like we should deploy it in order to double our capacity and, in turn, reduce the infrastructure resource overhead.

> Rearranging it to shave off path delimiters to mollify a tool

As discussed upthread, the tool will mollify itself soon; I don't want this discussion to be about that. I also agree that most of the things going wrong are inside the chrome build layout, not on your new system. But if your system's base directory is e:\b\s\w\ir\cache\builder\win compared to the current E:\b\c\builder\win (which also already seems like at most 1 dir more than necessary), then I think that new system is also a regression in directory complexity (which is a kind of proxy for overall complexity).

The tooling is built as a series of layers, and ideally we'd use cool things like chroots or Docker containers to define a new root for each layer so they can operate independently. The rub is, of course, that these are not available consistently on most of the platforms that we need to support.

Directory complexity is very cheap, as far as regressions go.

> The tooling is built as a series of layers

That's what I was getting at with directory depth being a kind of proxy for overall complexity -- from a distance, you might have more layers than necessary for what you're building (build infra for chrome).

The LUCI team doesn't think so. It's actually fairly analogous to the logical layering responsibilities used by BuildBot environments, with the notable exception that we have drawn explicit functional boundaries between components with different perspectives and responsibilities instead of smashing them together into a single interdependent code space.

This seems to be going off-topic from the actual bug, which we both agree needs to be solved by having this specific build require fewer delimiters. If you'd like to discuss fundamental LUCI design, please start a discussion thread on luci-eng@.

So we have 
8 steps path elements for the build machine (e:\b\s\w\ir\cache\builder\win)
1 for the checkout (src)
3 for the gen folder in the build result (out/release/gen)
6 for build internal directory structure (third_party/webkit/source/core/layout/svg)
9 to get from gen to static source (../../../../../../../../..)
7 for source internal directory structure (third_party/WebKit/Source/core/layout/svg/SVGTextQuery.cpp)

We get a multiplier of 3 on the "source internal dir structure" which is the killer, though the 8 initial steps seems a bit deep. Surely there are not 8 levels of ownership?

@thakis, should something have collapsed the back references? These come from an #include "../../../.. ... " statement in generated source, probably through the header list parsed from compiler output.

Re comment 16: it's 32 dirs _after canonicalization_, see the other bug. The 9 ../ are subtracted, not added. Which is why I'm saying we don't really understand what's happening yet.

we can workaround one path element in the "build machine" (add conditional logic to https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_checkout/api.py?q=chromium_checkout/api.py&sq=package:chromium&l=40)
but that sounds like a hack around a problem that shouldn't exist

what's the rationale behind the number 64? What's the impact if we make it 128?

We store a bitfield for / vs \ (for diags) for each path. That used to be a 32-bit int, and that really should be enough for chrome. We don't yet understand why we run into this limit with jumbo builds, so it might even be enough and there might just be some bug in the jumbo build stuff somewhere, or a bad interaction between something that does and ninja.

Some other projects out there claimed they need 64, so we made that an uint64_t. But 32 should be enough for chrome. If it isn't, something else is broken, as said above.

I don't think it's anything new or specific to jumbo except that Blink (where I started jumbo) has deeper paths than for instance ui/resources and other places that generate similar structures. We've probably been closer to the 32 element limit than we've known.

Generated mojo files have quite deep paths as well but they rely on -I to find headers instead or ../../..

I'd suggest finding an example where this happens and diagnosing why. All examples so far were with paths with well below 32 dirs after canonicalization.

I guess the check happens on a path that hasn't been through canonicalization. Is all this inside the ninja binary?

Not a blocker for swarmbucket anymore since ninja was updated to allow longer paths (now 64 instead of 32 as limit).

(more importantly, ninja now has the limit on post-canonicalizaiton paths instead of pre-canonicalization paths. The 32 limit would be fine for us now too.)