I had the same idea a while ago and looked at it some. I think it's not super easy to implement because to me it looked like all the TU-level things get translated into .ll-file-level-things and the .ll module system takes care of things of that -- so this might require changing .ll semantics. (Injecting some implicit internal named namespace around each TU doesn't quite work either I believe.)

However, we did talk to some upstream clang folks and it sounds like there's definite an open mind for making this use case work better. Richard Smith suggested that the "API" could be "clang file1.cc file2.cc file3.cc" instead of pragmas + includes, and we could fairly easily make it so that that at least shares file caches and whatnot. (Saving the reparsing is more work, but sharing caches could already help a lot.)

Jens (jl@opera) tried implementing a clang plugin with pragmas, but ran into trouble with the identifier lookup finding all conflicting matches in each anonymous namespace.

I have been playing with the "inject a namespace for each source file's anonymous namespace".  The first part is quite a small patch (assign a canonical name to each source file's anonymous namespace), but I'm still figuring out how the identifier lookup code works, so I'm not yet sure if this method will work.

ccache and icecc don't support multiple source files on the command line- these would also need to be fixed if we try that method.  I suspect other tools might also assume that only one source file is specified on the command line.

Should we/I start a thread on cfe-dev?

I'd say a cfe-dev thread can wait until we have some better understanding of realistic options (read: some hacky prototype), but I tend to be too conservative with starting threads.

It's a big ugly, but on the other hand it is a much cleaner way of doing jumbo/unity builds than fixing the conflicts by hand. And, jumbo/unity builds are an idea that is not going to go away - lots of game companies use them, for instance, and those companies would love to see options that make them easier to maintain.

Yeah, I should've said that in comment 2 too: In general, I think it'd be really cool if we could add better support for unity builds to the compiler, given that there's demand for this feature (in the industry, not just chrome) and that currently setting up unity builds is pretty gnarly. So in general I'm very in favor of this idea. I'm just saying it might be difficult to do, but that doesn't mean we shouldn't try to do it :-)

I mentioned this to Richard a while back, and he seemed to think it would be easy, so maybe we just need to pick his brain a bit :-)

This seems like something the compiler could potentially do really well.

I like the sound of that!

I don't think anyone else has had any useful progress the last week. In the end either end up with a copy of everything in which case you don't get the performance win, or just a fraction of it, or you get complicated lookups. The "complicated lookups" is the part where any assistance is helpful.

As mentioned above, I've been working on a plugin for clang (+ a minimal patch to clang to support what the plugin does) that helps us out with jumbo compilation. One working version of it can be found here:

  https://github.com/jensl/llvm-project-20170507/commit/a00d5ce3f20bf1c7a41145be8b7a3a478df9935f

The plugin adds `#pragma jumbo` that enables the following special handling of includes into the main file (assuming that all such includes are includes of "source files" rather than traditional headers):

 - Anonymous namespace declarations in the directly included source file are hidden after the included source file has ended. Anonymous namespace declarations in included headers are left alone. The hiding is done by setting a flag in clang's NamespaceDecl object, and checking this flag in a single place in the name lookup code.

 - Macros defined in the directly included source file are automatically undefined again. Again, macros defined in included header files are left alone.

In addition to this, I modified the name mangling of anonymous namespaces. This is required to actually support "conflicting" names in the anonymous namespace, since otherwise they mangle to the same symbol name, which doesn't work. This is not controlled by the plugin, but could be, or could be controlled by a separate argument to clang. (Currently it's unconditional in the patch.) This modification affects how tools will demangle the symbol name; the anonymous namespaces will look like a real namespace named `__anonymous_<number>`.