I am investigating the reason why obj/third_party/brotli/bro/bit_cost.o differs in
https://luci-milo.appspot.com/buildbot/chromium.fyi/Linux%20deterministic%20(dbg)/4532

offset seems to slightly different between 1st build (goma compile), and 2nd (local compile).  I also checked the same thing also occur on my local machine.

@@ -1,4 +1,4 @@
-There are 27 section headers, starting at offset 0x4628:
+There are 27 section headers, starting at offset 0x6760:
 
 Section Headers:
   [Nr] Name              Type             Address           Offset
@@ -7,8 +7,8 @@
        0000000000000000  0000000000000000           0     0     0
   [ 1] .text             PROGBITS         0000000000000000  00000040
        0000000000002456  0000000000000000  AX       0     0     16
-  [ 2] .rela.text        RELA             0000000000000000  000056d8
-       0000000000000c18  0000000000000018          25     1     8
+  [ 2] .rela.text        RELA             0000000000000000  00004f18
+       0000000000000c18  0000000000000018   I      25     1     8
   [ 3] .rodata.cst16     PROGBITS         0000000000000000  000024a0
        0000000000000060  0000000000000010  AM       0     0     16
   [ 4] .rodata.cst8      PROGBITS         0000000000000000  00002500
@@ -21,41 +21,41 @@
        0000000000000059  0000000000000000           0     0     1
   [ 8] .debug_info       PROGBITS         0000000000000000  00002b2c
        00000000000002c1  0000000000000000           0     0     1
-  [ 9] .rela.debug_info  RELA             0000000000000000  000062f0
-       0000000000000450  0000000000000018          25     8     8
+  [ 9] .rela.debug_info  RELA             0000000000000000  00005b30
+       0000000000000450  0000000000000018   I      25     8     8
   [10] .debug_ranges     PROGBITS         0000000000000000  00002ded
        0000000000000000  0000000000000000           0     0     1
   [11] .debug_macinfo    PROGBITS         0000000000000000  00002ded
        0000000000000001  0000000000000000           0     0     1
   [12] .debug_addr       PROGBITS         0000000000000000  00002dee
        0000000000000218  0000000000000000           0     0     1
-  [13] .rela.debug_addr  RELA             0000000000000000  00006740
-       0000000000000648  0000000000000018          25    12     8
+  [13] .rela.debug_addr  RELA             0000000000000000  00005f80
+       0000000000000648  0000000000000018   I      25    12     8
   [14] .debug_pubnames   PROGBITS         0000000000000000  00003006
        000000000000013d  0000000000000000           0     0     1
-  [15] .rela.debug_pubna RELA             0000000000000000  00006d88
-       0000000000000018  0000000000000018          25    14     8
+  [15] .rela.debug_pubna RELA             0000000000000000  000065c8
+       0000000000000018  0000000000000018   I      25    14     8
   [16] .debug_pubtypes   PROGBITS         0000000000000000  00003143
        00000000000000ae  0000000000000000           0     0     1
-  [17] .rela.debug_pubty RELA             0000000000000000  00006da0
-       0000000000000018  0000000000000018          25    16     8
+  [17] .rela.debug_pubty RELA             0000000000000000  000065e0
+       0000000000000018  0000000000000018   I      25    16     8
   [18] .comment          PROGBITS         0000000000000000  000031f1
        0000000000000024  0000000000000001  MS       0     0     1
   [19] .note.GNU-stack   PROGBITS         0000000000000000  00003215
        0000000000000000  0000000000000000           0     0     1
   [20] .eh_frame         X86_64_UNWIND    0000000000000000  00003218
        0000000000000070  0000000000000000   A       0     0     8
-  [21] .rela.eh_frame    RELA             0000000000000000  00006db8
-       0000000000000048  0000000000000018          25    20     8
+  [21] .rela.eh_frame    RELA             0000000000000000  000065f8
+       0000000000000048  0000000000000018   I      25    20     8
   [22] .debug_line       PROGBITS         0000000000000000  00003288
-  [23] .rela.debug_line  RELA             0000000000000000  00006e00
-       0000000000000018  0000000000000018          25    22     8
-  [24] .shstrtab         STRTAB           0000000000000000  00004521
+  [23] .rela.debug_line  RELA             0000000000000000  00006640
+       0000000000000018  0000000000000018   I      25    22     8
+  [24] .shstrtab         STRTAB           0000000000000000  00006658
        0000000000000104  0000000000000000           0     0     1
-  [25] .symtab           SYMTAB           0000000000000000  00004ce8
+  [25] .symtab           SYMTAB           0000000000000000  00004528
        00000000000005e8  0000000000000018          26    58     8
-  [26] .strtab           STRTAB           0000000000000000  000052d0
+  [26] .strtab           STRTAB           0000000000000000  00004b10
        0000000000000405  0000000000000000           0     0     1
 Key to Flags:
   W (write), A (alloc), X (execute), M (merge), S (strings), l (large)

Note that this was caused by goma issue, and I have fixed this issue.
However, I found new one.
https://luci-milo.appspot.com/buildbot/chromium.fyi/Linux%20deterministic%20%28dbg%29/4902
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FLinux_deterministic__dbg_%2F4902%2F%2B%2Frecipes%2Fsteps%2Fcompare_build_artifacts%2F0%2Fstdout
Is this something intended?
  0x4a86e0  : 46656220303520323031372030353a30303a3030002e2e2f2e2e2f626173652f 'Feb 05 2017 05:00:00.../../base/'
              4d617220303520323031372030353a30303a3030002e2e2f2e2e2f626173652f 'Mar 05 2017 05:00:00.../../base/'
               + ^^                                                             ^^^

Ah, yes, it is changing once per month as implemented in
https://cs.chromium.org/chromium/src/build/write_build_date_header.py

so determinism is bound to be broken every month to embed the kill date but this is deemed an acceptable trade off.

To be clear, I consider this WAI. I'm keeping the bug open as it would be nice if this would cause a fluke every month.

Just FYI
"Builder Linux deterministic (dbg)" does not fails  at compare_build_artifacts since March 5th :)

https://luci-milo.appspot.com/buildbot/chromium.fyi/Linux%20deterministic%20%28dbg%29/

Should we close this bug, then? I don't really like that someone has to check a red build once per month so if there's another solution I'd personally prefer that.

yes.
(and I want luci-milo to show at which step failed.  failed at compare_build_artifacts should be non-deterministic build but failed at others are fine for this builder)