Both 62 and 63 release branches failed similarly, e.g. https://uberchromegw.corp.google.com/i/chromeos_release/builders/asuka-release%20release-R62-9901.B/builds/50

For 62 specifically the pre flight passed building the same version of Chrome it failed on hours before the release array failed.
Starting at 7 we see https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-chrome-pre-flight-branch%20release-R62-9901.B/builds/244 passed and at 10pm we see failure part way through building Chrome in https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-chrome-pre-flight-branch%20release-R62-9901.B/builds/245

Last night I was also not able to load up LogDog for these builds, maybe there was a wider infra outage that impacted our ability to use Goma?

 crbug.com/700169  looks suspiciously like this issue.

A new version of goma was released at 10:20pm last night:
https://b.corp.google.com/issues/67872697#comment7

A bunch of Chrome/Chromium PFQ informational builds have same failures.

For reference, compiler_proxy logs
Success:
http://chromium-build-stats.appspot.com/compiler_proxy_log/2017/10/30/cros-beefy14-c2/compiler_proxy.cros-beefy14-c2.chrome-bot.log.INFO.20171030-195038.1102.gz

Failure:
http://chromium-build-stats.appspot.com/compiler_proxy_log/2017/10/31/cros-beefy14-c2/compiler_proxy.cros-beefy14-c2.chrome-bot.log.INFO.20171031-035352.1103.gz

Currently waiting for someone from goma to be available; I've contacted the mailing list with the information I have.

Hopefully we can figure out what's going on together, and if not, we can figure out how to roll back so we're able to make release builds.

+backup oncall

For the sake of posterity, copying here what was said in a chat thread:

The goma release bug[1] mentions releasing new versions of the client and the server at the same time, so it's not clear to those of us on the outside whether it is safe to rollback the client without touching the server.

It's also not clear if rolling back the release CL[2] is appropriate / works, or if the right thing to do would be to release a new version which is identical to an older version.

For now, folks in MTV time just don't have enough information or insight to be able to fix this on our own without making things potentially worse, so we need to wait for Tokyo to come online to help.

[1] b/67872697
[2] cl/173992453

Maybe we should disable goma on branches in chromeos-chrome.ebuild? It still builds fine locally without it.

rollback goma client to version=141.
see it fixes the issue.

master-chromium-pfq is green again.
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/5104

Let's see if release builders will also follow.

The build on bots looks successfully done, so the priority is reduced to 1.

Thanks to hashimoto@, I could successfully reproduced this locally.
So, it looks like sandox issue, since

% emerge-$BOARD chromoeos-chrome

hits the same issue, but

% FEATURES="-usersandbox" emerge-$BOARD chromeos-chrome

does not. Also, I made sure;
- compiler_proxy.INFO does not have any logs about starting Task.
- It looks like running one error command manually (outside of sandbox) in chroot successfully works.

goma-team, any recent changes, which potentially hits sandbox?

sandbox has no logs when failure happens?

Re #15. I couldn't find. vapier@, could you help?

sandbox tends to write to stderr when it fails.  if that's not available, then logs can get lost.  also, the Gentoo ebuild sandbox is in-process by overriding C lib symbols, so it might be causing weird problems for programs that rely/assume certain behavior.

you can run `sandbox` to start an interactive shell and run commands inside of a new sandbox to test behavior.

in version 141 to 142, there are no major code change in gomacc.
deps update was boringssl, which gomacc doesn't depend on.

so, I'm wondering lld might generate binary that couldn't be handled by sandbox well.

I'll try to investigate the detailed reason...

I changed lld is only used for linux, but gomacc is also linked by lld for chromeos?

shinyak@,
feel free to reassign to me if you're not the right owner. This isn't causing a problem on the tree anymore, so I feel like it shouldn't be sheriff-owned, but I can help if you aren't the right person.

Re #20, we use linux one for chromeos, specifically on bots.

#20, #22

This is about BUILD.gn in goma. You added os == "linux", but it holds on chromeos, too?
As far as I know, chromeos goma client has the same build configuration as linux, but base glibc version etc. are different.

Ah, I see. Then lld may be related to this. I thought that gomacc is made by 'Goma ChromeOS GN' bot.

#24, it's right but I believe your written if-condition holds on 'Goma ChromeOS GN' bot, too.

#25
Sorry, this is wrong. Since Goma ChromeOS GN is not using clang, lld should not be used there X(

In conclusion, lld may be related to this issue or not?
Which gomacc caused this issue, gomacc built on ChromeOS builder or Linux builder?

I confirmed gomacc linked with lld caused this issue.

$ cros_sdk --goma_dir=~/goma --chrome_root=~/work/chrome

--- with lld

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ ~/goma/goma_ctl.py start
using /tmp/goma_shinyak as tmpdir
GOMA version dd02c2ad42f2c7f6d00176c9ffa3df6aabfe7e15@1509336204
448
waiting for compiler_proxy...
compiler proxy (pid=448) status: http://127.0.0.1:8088 ok

Now goma is ready!

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ ~/goma/gomacc port
8088

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ /usr/bin/sandbox
============================= Gentoo path sandbox ==============================
Detection of the support files.
Verification of the required files.
Setting up the required environment variables.
The protected environment has been started.
--------------------------------------------------------------------------------
Process being started in forked instance.

shinyak@shinyak ~/trunk/src/scripts $ ~/goma/gomacc port
Segmentation fault (core dumped)



--- without lld

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ /usr/bin/sandbox
============================= Gentoo path sandbox ==============================
Detection of the support files.
Verification of the required files.
Setting up the required environment variables.
The protected environment has been started.
--------------------------------------------------------------------------------
Process being started in forked instance.

shinyak@shinyak ~/trunk/src/scripts $ ~/goma/gomacc port
8088

So...

1. We will release the next version of goma without linking lld.
2. I'll prioritize chromeos goma canary to detect the failure earlier

+ruiu

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/goma/client/+/e7483c354c47b8299674ea770fce7de158b5fa14

commit e7483c354c47b8299674ea770fce7de158b5fa14
Author: Shinya Kawanaka <shinyak@google.com>
Date: Thu Nov 02 07:01:46 2017

is this related with https://bugs.chromium.org/p/chromium/issues/detail?id=772559 ?

No, I don't think that is related. So far I've narrowed the problem down to this:

$ cat exit.s
.text
.globl _start
_start:
movq $60, %rax
movq $0, %rdi
syscall

.globl exit
.globl __libc_start_main
$ make exit.o
as   -o exit.o exit.s
$ ld.lld exit.o -o ~/.subversion/exit   --hash-style=gnu  -dynamic-linker /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libc.so.6 

In the sandbox:

$ .subversion/exit
Segmentation fault (core dumped)

I needed the --hash-style=gnu flag as well as the references to at least two DSO symbols to reproduce the problem, so I suspect that either lld is creating an invalid GNU-style hash table in some cases or the sandbox isn't reading it correctly.

hmm, the sandbox ELF parser does walk the symbol table to try and find the end, and unfortunately that uses a heuristic because the ELF format provides no way of getting that info up front.  lld might produce binaries that the heuristic doesn't handle.

i guess i'll have to bite the bullet and implement GNU hash table parsing, and if that fails, add a sanity check to make sure we don't walk outside the bounds of the registered memory region.

https://gitweb.gentoo.org/proj/sandbox.git/tree/libsandbox/wrapper-funcs/__wrapper_exec.c

The comment in https://gitweb.gentoo.org/proj/sandbox.git/tree/libsandbox/wrapper-funcs/__wrapper_exec.c says that it uses the sysv hash to get the number of symbols if the hash table exists.

So a quick "fix" is to pass "-hash-style=both" to lld so that it generates both sysv and gnu style hashes in an output file.

We ran into a similar problem in the CFI runtime library, and our solution at the time was to bounds check symbol name references.

http://llvm-cs.pcc.me.uk/projects/compiler-rt/lib/cfi/cfi.cc#230

Although from looking at the GNU hash table format it does seem possible to determine the number of symbols in the symbol table without needing to hash symbol names.

Fundamentally it feels more like a problem of the ELF spec itself. It is perhaps too late to fix it, but if ELF had `DT_SYMSZ` (or whatever it is called) field in the .dynamic section, we wouldn't have had to infer it from a hash table.

yeah, if there was an explicit signal like DT_SYMSZ ("Size in bytes of the symbol table"), everything would be fine :/

adding a check to sandbox akin to the CFI library probably wouldn't be too bad

There seems not be the builders whose name are shown in #0?
https://luci-milo.appspot.com/search?q=chromeos

Can we think the issue obsoleted?

ah, yeah, I think this is obsolete.

now goma-client is not linking with lld