Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 2 users
Status: Fixed
Owner: ----
Closed: Apr 28
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment
PFQ builders fail TestSimpleChromeWorkflow with "Could not run pkg-config."
Project Member Reported by mnissler@chromium.org, Apr 28 Back to list
Sample: https://uberchromegw.corp.google.com/i/chromeos/builders/arm-generic-chromium-pfq/builds/3398/steps/TestSimpleChromeWorkflow/logs/stdio

Relevant output:

20:15:39: INFO: RunCommand: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpKvNXVl/chrome-sdk-cache23pTml/cache chrome-sdk --board arm-generic --cwd /b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/cbuild/repository/buildbot_archive/arm-generic-chromium-pfq/R60-9501.0.0-rc2 --chrome-src /b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --nogoma -- bash -c '/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/buildtools/linux64/gn gen "/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release" --args="$GN_ARGS"' in /b/cbuild/repository
20:15:39: DEBUG: Configured cache_dir to '/tmp/cbuildbot-tmpKvNXVl/chrome-sdk-cache23pTml/cache'
ERROR at //build/config/linux/pkg_config.gni:101:17: Script returned non-zero exit code.
    pkgresult = exec_script(pkg_config_script, args, "value")
                ^----------
Current dir: /b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release/
Command: python -- /b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/build/config/linux/pkg-config.py -p x86_64-pc-linux-gnu-pkg-config nss -v -lssl3
Returned 1.
stderr:

Could not run pkg-config.

See //third_party/nss/BUILD.gn:15:3: whence it was called.
  pkg_config("system_nss_no_ssl_config") {
  ^---------------------------------------
See //crypto/BUILD.gn:225:25: which caused the file to be included.
    public_configs += [ "//third_party/nss:system_nss_no_ssl_config" ]
                        ^-------------------------------------------

@@@STEP_FAILURE@@@
20:15:41: ERROR: Traceback (most recent call last):
  File "/b/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 628, in Run
    self.PerformStage()
  File "/b/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 281, in PerformStage
    self._BuildChrome(sdk_cmd)
  File "/b/cbuild/repository/chromite/cbuildbot/stages/chrome_stages.py", line 255, in _BuildChrome
    sdk_cmd.Run(['bash', '-c', gn_gen_cmd])
  File "/b/cbuild/repository/chromite/cbuildbot/commands.py", line 2784, in Run
    cros_build_lib.RunCommand(cros_cmd, cwd=self.cwd, **run_args)
  File "/b/cbuild/repository/chromite/lib/cros_build_lib.py", line 625, in RunCommand
    raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: cros --log-level debug --cache-dir /tmp/cbuildbot-tmpKvNXVl/chrome-sdk-cache23pTml/cache chrome-sdk --board arm-generic --cwd /b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --sdk-path /b/cbuild/repository/buildbot_archive/arm-generic-chromium-pfq/R60-9501.0.0-rc2 --chrome-src /b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src --nogoma -- bash -c '/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/buildtools/linux64/gn gen "/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release" --args="$GN_ARGS"'
cmd=['cros', '--log-level', 'debug', '--cache-dir', '/tmp/cbuildbot-tmpKvNXVl/chrome-sdk-cache23pTml/cache', 'chrome-sdk', '--board', u'arm-generic', '--cwd', '/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--sdk-path', u'/b/cbuild/repository/buildbot_archive/arm-generic-chromium-pfq/R60-9501.0.0-rc2', '--chrome-src', '/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src', '--nogoma', '--', 'bash', '-c', u'/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/buildtools/linux64/gn gen "/b/cbuild/repository/.cache/distfiles/target-master/chrome-src/src/out_arm-generic/Release" --args="$GN_ARGS"'], cwd=/b/cbuild/repository

 
Cc: richard....@arm.com vapier@chromium.org
This change seems like the smoking gun: https://chromium-review.googlesource.com/c/483342/

It's possible that the PFQ builders aren't set up correctly?

I'll revert above CL speculatively and see whether it helps.
Project Member Comment 2 by bugdroid1@chromium.org, Apr 28
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/00b062e732629c252673b4cba77a9206edf5cbce

commit 00b062e732629c252673b4cba77a9206edf5cbce
Author: Mattias Nissler <mnissler@chromium.org>
Date: Fri Apr 28 10:14:39 2017

Revert "chromeos-chrome: add host_pkg_config for cross-compilation"

This reverts commit c27bea7ccb346760ee278b2d42ab3cd6188467c5.

Reason for revert: Likely the culprit for TestSimpleChromeWorkflow breakage in Chrome PFQ builders, see  http://crbug.com/716399 .

Original change's description:
> chromeos-chrome: add host_pkg_config for cross-compilation
> 
> This new build variable ensures that host-only compilation helpers
> aren't incorrectly linked against libraries in the current board
> sysroot.
> 
> BUG=chromium:710841
> TEST=cbuildbot kevin-release
> 
> Change-Id: Ifc667fd55cfaa00e2721c7894684909b6bd08b46
> Reviewed-on: https://chromium-review.googlesource.com/483342
> Commit-Ready: Richard Townsend <richard.townsend@arm.com>
> Tested-by: Richard Townsend <richard.townsend@arm.com>
> Reviewed-by: Mike Frysinger <vapier@chromium.org>
> 

TBR=vapier@chromium.org,richard.townsend@arm.com
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=chromium:710841,  chromium:716399 

Change-Id: If877ffdca42ce697957d7a438e4675a9cf7541ee
Reviewed-on: https://chromium-review.googlesource.com/489948
Reviewed-by: Mattias Nissler <mnissler@chromium.org>
Tested-by: Mattias Nissler <mnissler@chromium.org>

[modify] https://crrev.com/00b062e732629c252673b4cba77a9206edf5cbce/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

This was the right thing to do, I'm not sure why x86_64-pc-linux-gnu-pkg-config might not be available in this environment, or why it passed the commit queue.
Cc: chadversary@chromium.org
Labels: Hotlist-CrOS-Sheriffing
Status: Fixed
The revert fixed the TestSimpleChromeWorkflow step.
hrm, simple chrome.  the build pkg-config that is available inside the chroot is not guaranteed to be available outside.  we prob need to update the chromite/cli/cros/cros_chrome_sdk.py script first to set this new knob, and then reland that chromeos-chrome ebuild change.
OK, so I wasn't able to replicate the precise build, but I was able to the error via the cros shell. It seems to relate from the use of $(tc-getBUILD_PKG_CONFIG), which seems to return 'x86_64-pc-linux-gnu-pkg-config', which isn't provided in the arm-generic toolchain inside the cros shell. Patchset #2 of https://chromium-review.googlesource.com/483342 would probably correct this, but it's not a very clean solution.


Oops, guess vapier beat me to it. I'll add a follow up task inside https://bugs.chromium.org/p/chromium/issues/detail?id=710841
OK, so I still haven't been able to replicate the problem in the simple chrome environment. Would it be possible to get more of the build log so I can assess exactly what it's doing?
FWIW, attached is the full output from the build step. I'm not sure whether it's an actual problem or some build config issue. Do you ensure that pkg-config is installed in the build root?
pkg_config_build_error.txt
27.0 KB View Download
we don't ship a pkg-config for your active build system

just update cros_chrome_sdk.py as i mentioned earlier to set the variable and clobber whatever the ebuild has set up
So replicating the build steps as closely as possible and modifying the simple chrome environment a bit to simulate the unreverted ebuild, I think I have managed to come up with a patch for this. All it involves is adding 'gn_args["host_pkg_config"] = "pkg-config"' in cli/cros/cros_chrome_sdk.py at around line 703. If I can get it signed off internally, I'll be able to push it quite soon.
The current fix is available at https://chromium-review.googlesource.com/#/c/496446/. I don't quite understand yet the relationship between the ebuild's configuration and simple chrome's environment, but adding it into the the SDK script does seem to override the pkg_config setting in the environment's GN_ARGS.
Project Member Comment 14 by bugdroid1@chromium.org, May 8
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4e87bfa4ee18aa9d1799ea350eea3a0a406470b9

commit 4e87bfa4ee18aa9d1799ea350eea3a0a406470b9
Author: Richard Townsend <Richard.Townsend@arm.com>
Date: Mon May 08 19:43:43 2017

Add host_pkg_config for simple chrome builds.

host_pkg_config is used to make sure that compiled_actions targets
find the right libraries/include headers required for compilation on the
host machine for use elsewhere in the build process. It's related with
crbug.com/710841 and  crbug.com/710006 , which both block  crbug.com/595493 

BUG= chromium:716399 
TEST=Built with simple-chrome environment

Change-Id: I2daaf5d7f89d7f5229946e0f19fe44ece530e714
Reviewed-on: https://chromium-review.googlesource.com/496446
Commit-Ready: Richard Townsend <richard.townsend@arm.com>
Tested-by: Richard Townsend <richard.townsend@arm.com>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/4e87bfa4ee18aa9d1799ea350eea3a0a406470b9/cli/cros/cros_chrome_sdk.py

Labels: VerifyIn-61
Sign in to add a comment