tryserver.chromium.mac hermetic failure [difference between actual environment and expected environment] |
|||||||
Issue descriptionhttps://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.mac%2Fmac_chromium_compile_dbg_ng%2F388964%2F%2B%2Frecipes%2Fsteps%2Fanalyze%2F0%2Fstdout """ /b/c/b/mac/src/buildtools/mac/gn gen //out/Debug --check -> returned 1 ERROR at //build/config/mac/mac_sdk.gni:61:5: Script returned non-zero exit code. exec_script("//build/mac/find_sdk.py", find_sdk_args, "list lines") ^---------- Current dir: /b/c/b/mac/src/out/Debug/ Command: python -- /b/c/b/mac/src/build/mac/find_sdk.py --print_sdk_path 10.10 Returned 1. stderr: Traceback (most recent call last): File "/b/c/b/mac/src/build/mac/find_sdk.py", line 89, in <module> print main() File "/b/c/b/mac/src/build/mac/find_sdk.py", line 57, in main sdks = [re.findall('^MacOSX(10\.\d+)\.sdk$', s) for s in os.listdir(sdk_dir)] OSError: [Errno 2] No such file or directory: '/Library/Developer/CommandLineTools/Platforms/MacOSX.platform/Developer/SDKs' See //build/config/mac/base_rules.gni:11:3: whence it was imported. import("//build/config/mac/mac_sdk.gni") ^-------------------------------------- See //base/BUILD.gn:28:1: whence it was imported. import("//build/config/sysroot.gni") ^---------------------------------- See //BUILD.gn:78:5: which caused the file to be included. "//base:base_unittests", ^---------------------- GN gen failed: 1 """
,
Apr 5 2017
Looking at the logdog output, it seems like we're not picking up the FORCE_MAC_TOOLCHAIN environment variable.
,
Apr 5 2017
The current way that we are propagating this flag: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium/config.py?type=cs&q=FORCE_MAC_TOOLCHAIN+file:build/+-file:json&l=194 On the base config, we set: """ if c.TARGET_PLATFORM == 'mac': c.env.FORCE_MAC_TOOLCHAIN = 1 """ This then propagates to get_env() """ if self.c.env.FORCE_MAC_TOOLCHAIN: ret['FORCE_MAC_TOOLCHAIN'] = self.c.env.FORCE_MAC_TOOLCHAIN """ https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium/api.py?type=cs&q=FORCE_MAC_TOOLCHAIN+file:build/+-file:json&l=76 All the expected JSON files for tryserver.chromium.mac/mac_chromium_compile_dbg_ng imply that we should be picking up the environment variable FORCE_MAC_TOOLCHAIN. https://cs.chromium.org/search/?q=FORCE_MAC_TOOLCHAIN+file:build/+file:tryserver.chromium.mac&type=cs So I think the problem we're seeing is that the environment on the VM doesn't match the expected environment. https://cs.chromium.org/chromium/build/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_mac_mac_chromium_dbg_ng.json?q=FORCE_MAC_TOOLCHAIN+file:build/+file:tryserver.chromium.mac&dr=C&l=320 dpranke@, I think this is an infra issue. If you agree, can you find someone to take this bug?
,
Apr 5 2017
,
Apr 5 2017
,
Apr 6 2017
Issue 708847 has been merged into this issue.
,
Apr 6 2017
The analyze command invokes mb.py directly, rather than going through the run_mb() wrapper in the chromium module, and it looks like the code is missing the check for the env var. The analyze step in the JSON files skip the actual call out to MB, and so you don't see the env and can't tell whether it would've had the env var set or not, so the expected JSON files are being misleading here.
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/ece7e5aa69872d3551a64f9400024745b774a5c0 commit ece7e5aa69872d3551a64f9400024745b774a5c0 Author: Dirk Pranke <dpranke@chromium.org> Date: Thu Apr 06 02:23:12 2017 Fix recipe invocation of mb analyze on mac to check FORCE_MAC_TOOLCHAIN. The filter recipe_module code that runs the "analyze" logic calls out to mb.py directly to call analyze, rather than call into the chromium module's run_mb() method. As a result, there is code duplication, and there was a line in the chromium method that wasn't in the filter method, and we were incorrectly not exporting the FORCE_MAC_TOOLCHAIN variable. R=erikchen@chromium.org TBR=phajdan.jr@chromium.org NOTREECHECKS=true BUG= 708657 Change-Id: Id842967d32c935aa7750a9209e7f672e93d4bb19 Reviewed-on: https://chromium-review.googlesource.com/469076 Commit-Queue: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> [modify] https://crrev.com/ece7e5aa69872d3551a64f9400024745b774a5c0/scripts/slave/recipes/chromium_trybot.expected/use_v8_patch_on_blink_trybot.json [modify] https://crrev.com/ece7e5aa69872d3551a64f9400024745b774a5c0/scripts/slave/recipes/chromium_trybot.expected/use_skia_patch_on_blink_trybot.json [modify] https://crrev.com/ece7e5aa69872d3551a64f9400024745b774a5c0/scripts/slave/recipe_modules/filter/api.py
,
Apr 6 2017
This is hopefully fixed now.
,
Apr 6 2017
A successful analyze call: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.mac%2Fmac_chromium_compile_dbg_ng%2F389835%2F%2B%2Frecipes%2Fsteps%2Fanalyze%2F0%2Fstdout
,
Apr 6 2017
Issue 708893 has been merged into this issue. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by erikc...@chromium.org
, Apr 5 2017I sshed into the device. Setting DEVELOPER_DIR makes the script pass. Unsetting it shows the same error: """ chrome-bot@vm1089-m4:(Mac 10.12.2):/b/c/b/mac/src$ export DEVELOPER_DIR=/b/c/b/mac/src/build/mac_files/Xcode.app/ chrome-bot@vm1089-m4:(Mac 10.12.2):/b/c/b/mac/src$ /b/c/b/mac/src/build/mac/find_sdk.py 10.10 10.10 chrome-bot@vm1089-m4:(Mac 10.12.2):/b/c/b/mac/src$ unset DEVELOPER_DIR chrome-bot@vm1089-m4:(Mac 10.12.2):/b/c/b/mac/src$ /b/c/b/mac/src/build/mac/find_sdk.py 10.10 Traceback (most recent call last): File "/b/c/b/mac/src/build/mac/find_sdk.py", line 89, in <module> print main() File "/b/c/b/mac/src/build/mac/find_sdk.py", line 57, in main sdks = [re.findall('^MacOSX(10\.\d+)\.sdk$', s) for s in os.listdir(sdk_dir)] OSError: [Errno 2] No such file or directory: '/Library/Developer/CommandLineTools/Platforms/MacOSX.platform/Developer/SDKs' """