No cache hit build mainly come from nacl compiling.

e.g.
0 Task:46873	38.699709s	../../ppapi/proxy/ppapi_messages.cc gomacc_pid=15912 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
1 Task:4304	38.328264s	../../native_client/src/shared/platform/refcount_base.cc gomacc_pid=7820 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
2 Task:4298	34.281342s	../../native_client/src/shared/platform/nacl_log.c gomacc_pid=13012 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
3 Task:4306	34.031281s	../../native_client/src/shared/platform/posix/nacl_error.c gomacc_pid=10824 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
4 Task:4305	33.870142s	../../native_client/src/untrusted/nacl/sys_private.c gomacc_pid=6280 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
5 Task:4299	33.500067s	../../native_client/src/shared/gio/gprintf.c gomacc_pid=16048 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
6 Task:4307	33.250068s	../../native_client/src/shared/platform/posix/nacl_threads.c gomacc_pid=15508 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
7 Task:4302	33.13978s	../../native_client/src/shared/platform/posix/nacl_thread_id.c gomacc_pid=1848 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
8 Task:4303	33.109431s	../../native_client/src/untrusted/irt/irt_cond.c gomacc_pid=9964 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
9 Task:4300	32.968859s	../../native_client/src/shared/platform/posix/lock.c gomacc_pid=12060 build_dir=C:\b\swarming\w\ir\cache\builder\src\out\Release.2	goma success
from
https://chromium-build-stats.appspot.com/compiler_proxy_log/2018/11/19/swarm2173-c4/compiler_proxy.exe.SWARM2173-C4.chrome-bot.log.INFO.20181119-111555.408.gz

nacl compiler embeds cwd in obj file, and I don't know how to remove that without changing nacl compiler.

Re: #3
Is that because -g is set for NaCl compile?  Or is it come from something else?

In addition to cache hit, I found very high deps cache miss and very high compiler_info miss.  Do you change the build output dir per build?

Made CL.
https://chromium-review.googlesource.com/c/chromium/src/+/1343472

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/31713864d2dc4dd7580e90aba522eeaf24345cba

commit 31713864d2dc4dd7580e90aba522eeaf24345cba
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Tue Nov 20 14:50:12 2018

Use fdebug-compilation-dir for nacl

Luckily, nacl compiler has fdebug-compilation-dir and it makes nacl compiled result deterministic.
https://chromium.googlesource.com/native_client/pnacl-clang/+/cfbfff6970d1fc876a6ed3f04289e3086c9a9129/include/clang/Driver/CC1Options.td#135

Bug: 906741
Change-Id: I4b8ea54d20ba4cc58c07cc941a14584df33426c8
Reviewed-on: https://chromium-review.googlesource.com/c/1343472
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609697}
[modify] https://crrev.com/31713864d2dc4dd7580e90aba522eeaf24345cba/build/config/compiler/BUILD.gn

Please reopen this bug or file a new issue, if you see low cache hit rate on the builder again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eb5fc54df5fc7343b059a7afec023c50715cb688

commit eb5fc54df5fc7343b059a7afec023c50715cb688
Author: Nico Weber <thakis@chromium.org>
Date: Tue Nov 27 02:39:04 2018

Disable symbols for nacl-compiler-built object files.

The non-pnacl gcc-based nacl compilers embed absolute paths in their debug info.
This is bad for goma usage and build determinism.  Since I've never
seen a stack in nacl-built code and neither the code nor the toolchain
are changing much anymore, just disable symbols for nacl-built code.
(The same code when built with a non-gcc nacl toolchain of course still
has symbols, including the nacl-clang-built IRT where we archive the symbols for
production builds.)

Bug: 899438,906741,429358
Change-Id: I3062a26f25281759cc240f1967ab670f28e562a6
Reviewed-on: https://chromium-review.googlesource.com/c/1351492
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Derek Schuff <dschuff@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610998}
[modify] https://crrev.com/eb5fc54df5fc7343b059a7afec023c50715cb688/build/config/compiler/BUILD.gn

I believe "Use fdebug-compilation-dir for nacl" in comment 7 was a no-op. The if is protected by strip_absolute_paths_from_debug_symbols, but that's set to

strip_absolute_paths_from_debug_symbols = (is_linux || (is_win && use_lld)) && use_goma

For nacl compiles, is_linux is false and is_nacl is true. Adding `|| is_nacl` in the parens makes it so that the flag is at least passed, and that does help a bit (so we should land that change) in that it removes the absolute path to the current directory. However, if you look at the .o file there's still an absolute path to the output .o file in there:



thakis@thakis:~/src/chrome/src$ cat test.cc
#include <stdio.h>
int foo = 0;
int main() {
  fprintf(stderr, "p %p\n", &foo);
}
thakis@thakis:~/src/chrome/src$ native_client/toolchain/linux_x86/pnacl_newlib/bin/pnacl-clang -c test.cc -g1 --pnacl-allow-translate -arch x86-32-nonsfi --pnacl-bias=x86-32-nonsfi --target=i686-unknown-nacl  
thakis@thakis:~/src/chrome/src$ xxd test.o | grep src
000001c0: 2f68 6f6d 652f 7468 616b 6973 2f73 7263  /home/thakis/src
000001d0: 2f63 6872 6f6d 652f 7372 6300 0111 0025  /chrome/src....%
00000630: 632f 6368 726f 6d65 2f73 7263 2f74 6573  c/chrome/src/tes
thakis@thakis:~/src/chrome/src$ 
thakis@thakis:~/src/chrome/src$ 
thakis@thakis:~/src/chrome/src$ native_client/toolchain/linux_x86/pnacl_newlib/bin/pnacl-clang -c test.cc -g1 --pnacl-allow-translate -arch x86-32-nonsfi --pnacl-bias=x86-32-nonsfi --target=i686-unknown-nacl -Xclang -fdebug-compilation-dir -Xclang . -o test.o
thakis@thakis:~/src/chrome/src$ xxd test.o | grep src
00000600: 732f 7372 632f 6368 726f 6d65 2f73 7263  s/src/chrome/src



dschuff, do you happen where this absolute path to the output .o file comes from and if we can do anything about it?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e3b79481334b06df815c1b013c2a7d5aa2ef044

commit 1e3b79481334b06df815c1b013c2a7d5aa2ef044
Author: Nico Weber <thakis@chromium.org>
Date: Tue Dec 04 02:49:02 2018

Actually pass -fdebug-compilation-dir . to nacl clang compiles.

https://chromium-review.googlesource.com/c/chromium/src/+/1343472 tried to do
this, but it only updated one of two conditionals.

clang-based NaCl compilers used to encode the absolute path to each .o file
twice in each .o file. After this, it's still encoded one time, so this
doesn't really help with making nacl compiles deterministic, but it helps a bit.

Bug: 906741
Change-Id: Idfe83764a1691719f7aa93dcf97f6c7be4e4772e
Reviewed-on: https://chromium-review.googlesource.com/c/1359972
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#613406}
[modify] https://crrev.com/1e3b79481334b06df815c1b013c2a7d5aa2ef044/build/config/compiler/BUILD.gn

For the remaining absolute path, it comes from here:
thakis@thakis:~/src/chrome/src$ cat test.cc 
#include <stdio.h>
int foo = 0;
int main() {
  fprintf(stderr, "p %p\n", &foo);
}
thakis@thakis:~/src/chrome/src$ native_client/toolchain/linux_x86/pnacl_newlib/bin/pnacl-clang -c test.cc -g0 --pnacl-allow-translate -arch x86-32-nonsfi --pnacl-bias=x86-32-nonsfi --target=i686-unknown-nacl -Xclang -fdebug-compilation-dir -Xclang . -o test.o 
thakis@thakis:~/src/chrome/src$ objdump -x test.o | grep -A2 SYMBOL
SYMBOL TABLE:
00000000 l    df *ABS*	00000000 /usr/local/google/home/thakis/src/chrome/src/test.o---test.cc---.po
00000000 l     O .rodata.str1.1	00000006 .L.str


Compare to:

thakis@thakis:~/src/chrome/src$ third_party/llvm-build/Release+Asserts/bin/clang -c test.cc -g0 --target=i686-unknown-unknown -Xclang -fdebug-compilation-dir -Xclang . -o test.o 
thakis@thakis:~/src/chrome/src$ objdump -x test.o | grep -A2 SYMBOL
SYMBOL TABLE:
00000000 l    df *ABS*	00000000 test.cc
00000000 l     O .rodata.str1.1	00000006 .L.str

This removes that abs string:

$ git diff
diff --git a/pnacl/driver/driver_tools.py b/pnacl/driver/driver_tools.py
index 1bdb943c8..37e5fcdb0 100755
--- a/pnacl/driver/driver_tools.py
+++ b/pnacl/driver/driver_tools.py
@@ -497,8 +497,8 @@ def CheckPathLength(filename, exit_on_failure=True):
 # add parent directories. Rinse, repeat.
 class TempNameGen(object):
   def __init__(self, inputs, output):
-    inputs = [ pathtools.abspath(i) for i in inputs ]
-    output = pathtools.abspath(output)
+    inputs = inputs[:] [ pathtools.abspath(i) for i in inputs ]
+    #output = pathtools.abspath(output)
 
     self.TempBase = output + '---linked'
     self.OutputDir = pathtools.dirname(output)
@@ -569,7 +569,8 @@ class TempNameGen(object):
     return temp
 
   def TempNameForInput(self, input, imtype):
-    fullpath = pathtools.abspath(input)
+    #fullpath = pathtools.abspath(input)
+    fullpath = input
     # If input is already a temporary name, just change the extension
     if fullpath.startswith(self.TempBase):
       temp = self.TempBase + '.' + imtype


(called through pnacl-driver.py ; TempNameGen() call right after the DefaultOutputName() call)

With the patch in comment 14, nacl_helper_nonsfi becomes build-dir independent, but only if I build with GOMA_DISABLED=true.

That change or something like it seems like a reasonable one to make. It will have to wait until I get the NaCl buildbots resurrected after the switch to LUCI though.

The following revision refers to this bug:
  https://chromium.googlesource.com/native_client/src/native_client.git/+/ff15c51d20785fa51c0ef03ccb480b87299acbeb

commit ff15c51d20785fa51c0ef03ccb480b87299acbeb
Author: Nico Weber <thakis@chromium.org>
Date: Wed Jan 16 17:43:27 2019

Let TempNameGen not generate absolute paths.

This is important to make the build output independent of the
checkout directory.

Bug: chromium:429358,chromium:906741
Change-Id: Ide8abac96ffe08e4439a037459ba13e82473fae9
Reviewed-on: https://chromium-review.googlesource.com/c/1365356
Reviewed-by: Derek Schuff <dschuff@chromium.org>

[modify] https://crrev.com/ff15c51d20785fa51c0ef03ccb480b87299acbeb/pnacl/driver/driver_tools.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a62f1125c0d22c782bdec98b45b0f4aa0bedcaf4

commit a62f1125c0d22c782bdec98b45b0f4aa0bedcaf4
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Wed Jan 16 20:54:51 2019

Roll src/native_client 1ea07c56ac9b..ff15c51d2078 (1 commits)

https://chromium.googlesource.com/native_client/src/native_client.git/+log/1ea07c56ac9b..ff15c51d2078


git log 1ea07c56ac9b..ff15c51d2078 --date=short --no-merges --format='%ad %ae %s'
2019-01-16 thakis@chromium.org Let TempNameGen not generate absolute paths.


Created with:
  gclient setdep -r src/native_client@ff15c51d2078

The AutoRoll server is located here: https://autoroll.skia.org/r/nacl-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:429358,chromium:906741
TBR=mseaborn@chromium.org

Change-Id: I45e70f325df82fb1f849715afdb0d0f57148fb52
Reviewed-on: https://chromium-review.googlesource.com/c/1415092
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#623347}
[modify] https://crrev.com/a62f1125c0d22c782bdec98b45b0f4aa0bedcaf4/DEPS