New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 5 users
Status: Available
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: ----



Sign in to add a comment
BuildPackages failing to build chromeos-chrome
Project Member Reported by teravest@chromium.org, Oct 31 Back to list

Builders failed on: 
- asuka-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/asuka-release/1631
- auron_paine-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/auron_paine-release/1631
- auron_yuna-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/auron_yuna-release/1630
- banjo-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/banjo-release/1615
- banon-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/banon-release/1631
- betty-arc64-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/betty-arc64-release/230
- betty-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/betty-release/668
- bob-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/bob-release/1014
- buddy-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/buddy-release/1614
- butterfly-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/butterfly-release/4200
- candy-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/candy-release/1630
- caroline-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/caroline-release/1173
- cave-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/cave-release/1633
- celes-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/celes-release/1630
- chell-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/chell-release/1599
- clapper-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/clapper-release/2054
- coral-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/coral-release/473
- cyan-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/cyan-release/1632
- daisy-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/daisy-release/1632
- daisy_skate-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/daisy_skate-release/1908
- daisy_spring-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/daisy_spring-release/2965
- edgar-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/edgar-release/1635
- elm-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/elm-release/1632
- enguarde-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/enguarde-release/1766
- eve-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/eve-release/1056
- expresso-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/expresso-release/1795
- falco-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/falco-release/2963
- falco_li-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/falco_li-release/1735
- fizz-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/fizz-release/692
- gandof-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/gandof-release/1622
- glimmer-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/glimmer-release/1913
- gnawty-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/gnawty-release/1630
- gru-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/gru-release/1666
- guado-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/guado-release/1635
- hana-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/hana-release/1173
- heli-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/heli-release/1634
- jecht-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/jecht-release/1631
- kahlee-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/kahlee-release/695
- kefka-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/kefka-release/1634
- kevin-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/kevin-release/1641
- kip-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/kip-release/1706
- lars-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/lars-release/1633
- leon-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/leon-release/2447
- link-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/link-release/6061
- lulu-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/lulu-release/1634
- lumpy-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/lumpy-release/5482
- mccloud-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/mccloud-release/1637
- monroe-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/monroe-release/2251
- nefario-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/nefario-release/332
- newbie-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/newbie-release/668
- ninja-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/ninja-release/1637
- novato-arc64-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/novato-arc64-release/230
- novato-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/novato-release/471
- nyan_big-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/nyan_big-release/1923
- nyan_blaze-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/nyan_blaze-release/1635
- nyan_kitty-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/nyan_kitty-release/1612
- oak-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/oak-release/1654
- orco-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/orco-release/1634
- panther-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/panther-release/2255
- parrot-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/parrot-release/4272
- parrot_ivb-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/parrot_ivb-release/1636
- peach_pi-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/peach_pi-release/2032
- peach_pit-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/peach_pit-release/3089
- peppy-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/peppy-release/3091
- poppy-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/poppy-release/887
- pyro-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/pyro-release/1150
- quawks-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/quawks-release/1917
- reef-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/reef-release/1623
- reef-uni-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/reef-uni-release/392
- reks-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/reks-release/1630
- relm-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/relm-release/1633
- rikku-release: 
  https://luci-milo.appspot.com/buildbot/chromeos/rikku-release/1636



 
Cc: d...@chromium.org ayatane@chromium.org
Both 62 and 63 release branches failed similarly, e.g. https://uberchromegw.corp.google.com/i/chromeos_release/builders/asuka-release%20release-R62-9901.B/builds/50

For 62 specifically the pre flight passed building the same version of Chrome it failed on hours before the release array failed.
Starting at 7 we see https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-chrome-pre-flight-branch%20release-R62-9901.B/builds/244 passed and at 10pm we see failure part way through building Chrome in https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-chrome-pre-flight-branch%20release-R62-9901.B/builds/245

Last night I was also not able to load up LogDog for these builds, maybe there was a wider infra outage that impacted our ability to use Goma?
Comment 2 Deleted
crbug.com/700169 looks suspiciously like this issue.

A new version of goma was released at 10:20pm last night:
https://b.corp.google.com/issues/67872697#comment7
Cc: malaykeshav@chromium.org warx@chromium.org
A bunch of Chrome/Chromium PFQ informational builds have same failures.
Comment 6 Deleted
Cc: aga...@chromium.org
Currently waiting for someone from goma to be available; I've contacted the mailing list with the information I have.

Hopefully we can figure out what's going on together, and if not, we can figure out how to roll back so we're able to make release builds.
Cc: iannucci@chromium.org
+backup oncall
For the sake of posterity, copying here what was said in a chat thread:

The goma release bug[1] mentions releasing new versions of the client and the server at the same time, so it's not clear to those of us on the outside whether it is safe to rollback the client without touching the server.

It's also not clear if rolling back the release CL[2] is appropriate / works, or if the right thing to do would be to release a new version which is identical to an older version.

For now, folks in MTV time just don't have enough information or insight to be able to fix this on our own without making things potentially worse, so we need to wait for Tokyo to come online to help.

[1] b/67872697
[2] cl/173992453
Cc: ukai@chromium.org estaab@chromium.org
Components: Infra>Goma
Cc: ihf@chromium.org
Maybe we should disable goma on branches in chromeos-chrome.ebuild? It still builds fine locally without it.
rollback goma client to version=141.
see it fixes the issue.
Cc: hidehiko@chromium.org
master-chromium-pfq is green again.
https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/5104

Let's see if release builders will also follow.
Labels: -Pri-0 Pri-1
The build on bots looks successfully done, so the priority is reduced to 1.

Thanks to hashimoto@, I could successfully reproduced this locally.
So, it looks like sandox issue, since

% emerge-$BOARD chromoeos-chrome

hits the same issue, but

% FEATURES="-usersandbox" emerge-$BOARD chromeos-chrome

does not. Also, I made sure;
- compiler_proxy.INFO does not have any logs about starting Task.
- It looks like running one error command manually (outside of sandbox) in chroot successfully works.

goma-team, any recent changes, which potentially hits sandbox?

sandbox has no logs when failure happens?

Cc: vapier@chromium.org
Re #15. I couldn't find. vapier@, could you help?
sandbox tends to write to stderr when it fails.  if that's not available, then logs can get lost.  also, the Gentoo ebuild sandbox is in-process by overriding C lib symbols, so it might be causing weird problems for programs that rely/assume certain behavior.

you can run `sandbox` to start an interactive shell and run commands inside of a new sandbox to test behavior.
in version 141 to 142, there are no major code change in gomacc.
deps update was boringssl, which gomacc doesn't depend on.

so, I'm wondering lld might generate binary that couldn't be handled by sandbox well.
I'll try to investigate the detailed reason...
I changed lld is only used for linux, but gomacc is also linked by lld for chromeos?
Cc: teravest@chromium.org
Owner: shinyak@chromium.org
shinyak@,
feel free to reassign to me if you're not the right owner. This isn't causing a problem on the tree anymore, so I feel like it shouldn't be sheriff-owned, but I can help if you aren't the right person.
Re #20, we use linux one for chromeos, specifically on bots.
#20, #22

This is about BUILD.gn in goma. You added os == "linux", but it holds on chromeos, too?
As far as I know, chromeos goma client has the same build configuration as linux, but base glibc version etc. are different.
Ah, I see. Then lld may be related to this. I thought that gomacc is made by 'Goma ChromeOS GN' bot.
#24, it's right but I believe your written if-condition holds on 'Goma ChromeOS GN' bot, too.
#25
Sorry, this is wrong. Since Goma ChromeOS GN is not using clang, lld should not be used there X(


In conclusion, lld may be related to this issue or not?
Which gomacc caused this issue, gomacc built on ChromeOS builder or Linux builder?
I confirmed gomacc linked with lld caused this issue.

$ cros_sdk --goma_dir=~/goma --chrome_root=~/work/chrome

--- with lld

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ ~/goma/goma_ctl.py start
using /tmp/goma_shinyak as tmpdir
GOMA version dd02c2ad42f2c7f6d00176c9ffa3df6aabfe7e15@1509336204
448
waiting for compiler_proxy...
compiler proxy (pid=448) status: http://127.0.0.1:8088 ok

Now goma is ready!

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ ~/goma/gomacc port
8088

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ /usr/bin/sandbox
============================= Gentoo path sandbox ==============================
Detection of the support files.
Verification of the required files.
Setting up the required environment variables.
The protected environment has been started.
--------------------------------------------------------------------------------
Process being started in forked instance.

shinyak@shinyak ~/trunk/src/scripts $ ~/goma/gomacc port
Segmentation fault (core dumped)



--- without lld

(cr) ((85217e1...)) shinyak@shinyak ~/trunk/src/scripts $ /usr/bin/sandbox
============================= Gentoo path sandbox ==============================
Detection of the support files.
Verification of the required files.
Setting up the required environment variables.
The protected environment has been started.
--------------------------------------------------------------------------------
Process being started in forked instance.

shinyak@shinyak ~/trunk/src/scripts $ ~/goma/gomacc port
8088




So...

1. We will release the next version of goma without linking lld.
2. I'll prioritize chromeos goma canary to detect the failure earlier

Cc: ruiu@google.com
+ruiu

Project Member Comment 31 by bugdroid1@chromium.org, Nov 2
The following revision refers to this bug:
  https://chrome-internal.googlesource.com/goma/client/+/e7483c354c47b8299674ea770fce7de158b5fa14

commit e7483c354c47b8299674ea770fce7de158b5fa14
Author: Shinya Kawanaka <shinyak@google.com>
Date: Thu Nov 02 07:01:46 2017

Cc: p...@chromium.org
is this related with https://bugs.chromium.org/p/chromium/issues/detail?id=772559 ?
No, I don't think that is related. So far I've narrowed the problem down to this:

$ cat exit.s
.text
.globl _start
_start:
movq $60, %rax
movq $0, %rdi
syscall

.globl exit
.globl __libc_start_main
$ make exit.o
as   -o exit.o exit.s
$ ld.lld exit.o -o ~/.subversion/exit   --hash-style=gnu  -dynamic-linker /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libc.so.6 

In the sandbox:

$ .subversion/exit
Segmentation fault (core dumped)

I needed the --hash-style=gnu flag as well as the references to at least two DSO symbols to reproduce the problem, so I suspect that either lld is creating an invalid GNU-style hash table in some cases or the sandbox isn't reading it correctly.
hmm, the sandbox ELF parser does walk the symbol table to try and find the end, and unfortunately that uses a heuristic because the ELF format provides no way of getting that info up front.  lld might produce binaries that the heuristic doesn't handle.

i guess i'll have to bite the bullet and implement GNU hash table parsing, and if that fails, add a sanity check to make sure we don't walk outside the bounds of the registered memory region.

https://gitweb.gentoo.org/proj/sandbox.git/tree/libsandbox/wrapper-funcs/__wrapper_exec.c
The comment in https://gitweb.gentoo.org/proj/sandbox.git/tree/libsandbox/wrapper-funcs/__wrapper_exec.c says that it uses the sysv hash to get the number of symbols if the hash table exists.

So a quick "fix" is to pass "-hash-style=both" to lld so that it generates both sysv and gnu style hashes in an output file.
We ran into a similar problem in the CFI runtime library, and our solution at the time was to bounds check symbol name references.

http://llvm-cs.pcc.me.uk/projects/compiler-rt/lib/cfi/cfi.cc#230

Although from looking at the GNU hash table format it does seem possible to determine the number of symbols in the symbol table without needing to hash symbol names.
Fundamentally it feels more like a problem of the ELF spec itself. It is perhaps too late to fix it, but if ELF had `DT_SYMSZ` (or whatever it is called) field in the .dynamic section, we wouldn't have had to infer it from a hash table.
Cc: ddavenp...@chromium.org
yeah, if there was an explicit signal like DT_SYMSZ ("Size in bytes of the symbol table"), everything would be fine :/

adding a check to sandbox akin to the CFI library probably wouldn't be too bad
Sign in to add a comment