This CL seems suspicious:
https://codereview.chromium.org/1964003002

It looks like simple chrome on the pfq bots still uses gyp, and there isn't a gyp way to build harfbuzz any more?

That CL shouldn't affect anything, and harfbuzz should still be configured via gyp just fine.

I forgot we also build simple chrome in the PFQ builders. We should probably deprecate that, but I think we wanted to get basic VMTests running from simple chrome first, iirc?

In the meanwhile we should migrate the simple chrome builds in the pfq to use GN.

Didn't David James say that the artifacts from the simple chrome build on the PFQ were used somewhere? If not, we should get rid of the step.

Re #4 - Ah, I do recall that discussion but I forget the conclusion. I will re-investigate.

Running:

cros --log-level debug chrome-sdk --board x86-generic --nogoma -- gclient runhooks

Runs fine locaaly with a ToT build. Running with -chroot pointing to my local checkout (just now synced to ToT) also runs fine.

Hmm.

I just rolled chromite locally and that didn't seem to matter.
I realize I haven't run build_packages for x86-generic in my chroot, doing that now.

This seems like a possible culprit, especially since this only affects 32 bit builds:

https://chromium-review.googlesource.com/#/c/342471/4/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

RE 5 I believe the conclusion was no, we don't use simple chrome artifacts from the pfq, but having simple chrome tested in the pfq is important to protect new chrome from breaking that workflow.

Ahah. This doesn't seem to have anything to do with gyp/gn, just whether or not we are using a clean checkout.

Running this from my regular chrome checkout:

$ rm -rf third_party/harfbuzz-ng/
$ gclient sync --force

Got:

gyp: /usr/local/google/home/stevenjb/Work/chrome/src/third_party/harfbuzz-ng/harfbuzz.gyp not found (cwd: /usr/local/google/home/stevenjb/Work/chrome)
Error: Command '/usr/bin/python src/build/gyp_chromium --running-as-hook' returned non-zero exit status 1 in /usr/local/google/home/stevenjb/Work/chrome

I lied. It is GYP releated. I forgot I had unset GYP_CHROMIUM_NO_ACTION=1. I get the same error running:

$ ./build/gyp_chromium

harfbuzz-ng is checked into the regular chromium repo; it's not pulled in via DEPS, so I'm not sure what you're seeing?

I just realized that. FWIW that is messed up.

what is messed up?

s/messed up/confusing/ (that we have code in third_party that is part of
the normal git repo and not pulled in through DEPS).

-Anyway-, continuing to poke sticks at things...

don't look at third_party/WebKit, then :)

Actually the large majority of things in third_party are checked in, because pulling them via repos makes things slower and more fragile. We only recommend using DEPS for things that see a high rate of change and that we need to roll frequently to stay close to tip-of-tree.

(Yes, and then there are directories not in third_party/ that are DPES controlled like v8/, but again, that is an entirely different topic. Returning to our usual programming...)

I will add that this seems like a compelling argument to test SimpleChrome separately from the PFQ builder:  I am not sure what if any correlation there is between the chromeos updates in this build and the SimpleChrome environment.

I see this getting passed to 'cros chrome-sdk':
--sdk-path /b/cbuild/internal_master/buildbot_archive/x86-generic-chromium-pfq/R52-8314.0.0-rc2

I -assume- that means it is using the chromeos environment for R52-8314.0.0-rc2, and not what is being used for the chroot build??

+davidjames@

Re #9: Aviv, we have these simple chrome builders on the right: https://uberchromegw.corp.google.com/i/chromium.chromiumos/console

I can't remember if they are informational or normal waterfall builders. We should make them important if they are currently FYI only. Building simple chrome should belong in the chromium waterfall rather than the PFQ.

Another possible culprit that shows up in the first failing pfq build: https://chromium-review.googlesource.com/#/c/344422/

Re #20: I thought that might be it but tested locally and couldn't reproduce the error.

OK, with a local ToT chroot build with BOARD=x96-generic, running:

$  cros --log-level debug chrome-sdk --board=x86-generic --nogoma --chroot /usr/local/google/home/stevenjb/Work/chromeos/chroot -- gclient runhooks

Shows an error (but not the harbuzz one):

Updating projects from gyp files...
Package libevdev-cros was not found in the pkg-config search path.
Perhaps you should add the directory containing `libevdev-cros.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libevdev-cros' found
gyp: Call to 'pkg-config --cflags libevdev-cros' returned exit status 1 while in /usr/local/google/home/stevenjb/Work/chrome/src/build/linux/system.gyp.
Error: Command '/usr/bin/python src/build/gyp_chromium --running-as-hook' returned non-zero exit status 1 in /usr/local/google/home/stevenjb/Work/chrome
12:31:13: DEBUG: Uploading command stats to 'https://chromiumos-build-stats.appspot.com/upload_command_stats'

The error I am seeing does not appear to be chrome specific, at least I updated my local chrome checkout to master@393045 (tag 52.0.2734.0, which was the last green run).

Steven, I think you're on the right path.
https://uberchromegw.corp.google.com/i/chromeos/builders/x86-generic-chromium-pfq/builds/8610/steps/SimpleChromeWorkflow/logs/stdio


Log lines:

Error: Command '/usr/bin/python src/build/gyp_chromium --running-as-hook' returned non-zero exit status 1 in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src
Updating projects from gyp files...
Package libevdev-cros was not found in the pkg-config search path.
Perhaps you should add the directory containing `libevdev-cros.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libevdev-cros' found
gyp: Call to 'pkg-config --cflags libevdev-cros' returned exit status 1 while in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src/src/build/linux/system.gyp.
Package harfbuzz was not found in the pkg-config search path.
Perhaps you should add the directory containing `harfbuzz.pc'
to the PKG_CONFIG_PATH environment variable
No package 'harfbuzz' found
gyp: Call to 'pkg-config --cflags harfbuzz' returned exit status 1 while in /b/cbuild/internal_master/.cache/distfiles/target/chrome-src/src/third_party/harfbuzz-ng/harfbuzz.gyp.
Hook '/usr/bin/python src/build/gyp_chromium --running-as-hook' took 10.61 secs

So libevdev-cros and system.gyp are somehow implicated.

I'm unfortunately still unable to reproduce this :( I'm at TOT cros, with chrome at master@393045. I delete out_x86-generic directory before invoking the same gclient runhooks command, and it succeeds. I don't know what I'm missing

Grrr. simple chrome is parsing the partial-metadata.json file generated from the ebuild somehow, hindering my efforts to partially reproduce what is going on in SimpleChromeWorkflow:

https://uberchromegw.corp.google.com/i/chromeos/builders/x86-generic-chromium-pfq/builds/8610/steps/SimpleChromeWorkflow/logs/stdio.

Will keep trying, but I have to prep for an interview soon.

+hashimoto@ in case he has any thoughts

It is worth noting that the PFQ builder is using R52-8315.0.0:
https://uberchromegw.corp.google.com/i/chromeos/builders/x86-generic-chromium-pfq/builds/8610

But it is not at all clear what the SimpleChrome builders on the chromium tree are using:
https://build.chromium.org/p/chromium.chromiumos/builders/ChromiumOS%20x86-generic%20Compile/builds/19106

If it is the same as the default for devs, it is  R52-8278.0.0

The simple chrome builders are also using gn, which is why I believe they are green. 

The 8610 simple chrome build is using 8315.0.0. There are two steps that print the target_sysroot (steps, and generate_build_files). Look for target_sysroot in these logs:
https://build.chromium.org/p/chromium.chromiumos/builders/ChromiumOS%20x86-generic%20Compile/builds/19106/steps/steps/logs/stdio
https://build.chromium.org/p/chromium.chromiumos/builders/ChromiumOS%20x86-generic%20Compile/builds/19106/steps/generate_build_files/logs/stdio

Sorry, in the last comment, I meant the 19106 simple chrome build

Oh, good point about GN in comment #28.

OK, I just verified that this CL ('use debug fission for arm and x86'):

https://chromium-review.googlesource.com/342471

Is causing the following error when running:

$ cros --log-level debug chrome-sdk --board=x86-generic --nogoma --chroot /usr/local/google/home/stevenjb/Work/chromeos/chroot -- gclient runhooks

...

Updating projects from gyp files...
Package libevdev-cros was not found in the pkg-config search path.
Perhaps you should add the directory containing `libevdev-cros.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libevdev-cros' found
gyp: Call to 'pkg-config --cflags libevdev-cros' returned exit status 1 while in /usr/local/google/home/stevenjb/Work/chrome/src/build/linux/system.gyp.
Error: Command '/usr/bin/python src/build/gyp_chromium --running-as-hook' returned non-zero exit status 1 in /usr/local/google/home/stevenjb/Work/chrome


This is not too surprising since it is only affecting x86 / arm, but I have no clue why it is affecting libevdev-cros. I will try to isolate which part of the change is causing the problem next.

Not sure why it's calling 'pkg-config --cflags harfbuzz'.
It should use pkg-config-wrapper when using a sysroot.
(https://code.google.com/p/chromium/codesearch#chromium/src/build/common.gypi&l=1020)

I tried to figure out why it's happening, but couldn't reproduced with my local checkout w/ x86-generic.
The environment_chromeos-base_chromeos-chrome.tar.xz artifact from the PFQ builder looks fine as its GYP_DEFINES contains system_libdir=lib, and cros_chrome_sdk.py is configuring sysroot and pkg-config correctly.
Also I looked at the contents of the sysroot artifact, and it looked fine as /usr/lib/pkgconfig contains libevdev-cros.pc and harfbuzz.pc.

So 'harfbuzz' was something of a red herring.

The change at fault is indeed this:
https://chromium-review.googlesource.com/#/c/342471/4/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

It modifies BUILD_DEFINES to have:

release_extra_cflags=-gsplit-dwarf -g

This later gets split into two separate options, 'release_extra_cflags=-gsplit-dwarf' and '-g'. This causes problems in cros_chrome_sdk.py.

I am working on a fix.

re #33: Can we just revert this CL right now?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/23a7281c069a2336e65ba8aeceba396b3592454a

commit 23a7281c069a2336e65ba8aeceba396b3592454a
Author: Yunlian Jiang <yunlian@google.com>
Date: Mon May 16 17:37:41 2016

chromeos-chrome: do not pass both '-g' and '-gsplit-dwarf' at the same time

Currently cros_chrome_sdk.py could not handle multiple options
in release_extra_cflags, so remove '-g' as a workaround.

BUG= chromium:611784 
TEST=The debug info is still there.

Change-Id: Ic1cb6e2c533684954bc3c4a169099d56248d3ff3
Reviewed-on: https://chromium-review.googlesource.com/344811
Commit-Queue: Luis Lozano <llozano@chromium.org>
Tested-by: Luis Lozano <llozano@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[modify] https://crrev.com/23a7281c069a2336e65ba8aeceba396b3592454a/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/93bc764c6de0c6fdc5604d3fce1da8eabb89314b

commit 93bc764c6de0c6fdc5604d3fce1da8eabb89314b
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Mon May 16 17:58:32 2016

DictToGypDefines - use double quotes to support gyp flags with spaces

BUG= chromium:611784 
TEST=Simple chrome works with args like:
     'release_extra_cflags=-gdwarf-split -g'

Change-Id: I69b6cc9276a3c9c6928cdd9c7b587c32d59afb6b
Reviewed-on: https://chromium-review.googlesource.com/344733
Commit-Ready: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Ryo Hashimoto <hashimoto@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>

[modify] https://crrev.com/93bc764c6de0c6fdc5604d3fce1da8eabb89314b/lib/chrome_util.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/34d10c212b3c82049acebadc0b3c99bcc8cd5b78

commit 34d10c212b3c82049acebadc0b3c99bcc8cd5b78
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Mon May 16 16:05:19 2016

chromeos-chrome ebuild: Support multiple extra cflags

BUG= chromium:611784 
TEST=Simple chrome works with gyp

Change-Id: I9312a3e2a9ea4f8aa9c3a9a39b837b9f7f61a41d
Reviewed-on: https://chromium-review.googlesource.com/344578
Commit-Ready: Steven Bennetts <stevenjb@chromium.org>
Tested-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/34d10c212b3c82049acebadc0b3c99bcc8cd5b78/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild