Here's the output of running sdk_info.py directly:

λ python build/config/mac/sdk_info.py macosx
machine_os_build="17F77"
sdk_build="17E189"
sdk_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk"
sdk_platform_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform"
sdk_version="10.13"
xcode_build="9F2000"
xcode_version="0941"
xcode_version_int=941

Seems like a mac sdk detection problem specific to the gn build setup? I assume you'd see the same when running gn in chromium on your machine?

What's the output of `env`?

λ env
rvm_bin_path=/Users/michaelreinstein/.rvm/bin
TERM_PROGRAM=Apple_Terminal
GEM_HOME=/Users/michaelreinstein/.rvm/gems/ruby-2.2.1
CCACHE_SLOPPINESS=time_macros
SHELL=/bin/bash
TERM=xterm-256color
IRBRC=/Users/michaelreinstein/.rvm/rubies/ruby-2.2.1/.irbrc
TMPDIR=/var/folders/_5/_ng5y8rx60b72fdbq7vhh0m00000gn/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.rEKqInesNC/Render
TERM_PROGRAM_VERSION=404
OLDPWD=/Users/michaelreinstein/wwwroot/xid
MY_RUBY_HOME=/Users/michaelreinstein/.rvm/rubies/ruby-2.2.1
TERM_SESSION_ID=346F0BD1-C5DF-41D9-98F3-BDBB1EC6D5C7
USER=michaelreinstein
_system_type=Darwin
rvm_path=/Users/michaelreinstein/.rvm
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.BYnOCseQ85/Listeners
CCACHE_CPP2=yes
rvm_prefix=/Users/michaelreinstein
PATH=/Users/michaelreinstein/.cargo/bin:/Users/michaelreinstein/wwwroot/v8/third_party/llvm-build/Release+Asserts/bin:/usr/local/sbin:/Users/michaelreinstein/.rvm/gems/ruby-2.2.1/bin:/Users/michaelreinstein/.rvm/gems/ruby-2.2.1@global/bin:/Users/michaelreinstein/.rvm/rubies/ruby-2.2.1/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/michaelreinstein/.rvm/bin:/Users/michaelreinstein/wwwroot/depot_tools
PWD=/Users/michaelreinstein/wwwroot/v8
LANG=en_US.UTF-8
_system_arch=x86_64
XPC_FLAGS=0x0
PS1=λ 
_system_version=10.13
CXX=ccache clang++ -Qunused-arguments
XPC_SERVICE_NAME=0
rvm_version=1.26.11 (latest)
SHLVL=1
HOME=/Users/michaelreinstein
LOGNAME=michaelreinstein
GEM_PATH=/Users/michaelreinstein/.rvm/gems/ruby-2.2.1:/Users/michaelreinstein/.rvm/gems/ruby-2.2.1@global
CC=ccache clang -Qunused-arguments
DISPLAY=/private/tmp/com.apple.launchd.Lm3LlMuAMR/org.macosforge.xquartz:0
RUBY_VERSION=ruby-2.2.1
_system_name=OSX
_=/usr/bin/env

Looks pretty sane to me. This should work, not sure what's wrong. You'll have to debug things yourself :-/

Alright, well thanks for asserting some things. Even just hearing things don't look obviously wrong is helpful. :)

I've narrowed down the commands that run to reproduce this error:

λ gn gen out.gn/x64.release --check
ERROR at dynamically parsed input that //build/config/mac/mac_sdk.gni:84:19 loaded :1:1: Invalid token.
machine_os_build="17F77"
^
I have no idea what this is.
See //build/toolchain/mac/BUILD.gn:14:1: whence it was imported.
import("//build/config/mac/mac_sdk.gni")
^--------------------------------------
See //BUILD.gn:646:1: which caused the file to be included.
action("js2c") {
^---------------



Here is line 84 of mac_sdk.gni:84:

_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")


script_name is //build/config/mac/sdk_info.py and sdk_info_args is [ "macosx" ]

So this line is calling  python build/config/mac/sdk_info.py macosx which prints this to stdout:

machine_os_build="17F77"
sdk_build="17E189"
sdk_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk"
sdk_platform_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform"
sdk_version="10.13"
xcode_build="9F2000"
xcode_version="0941"
xcode_version_int=941

exec_script() with the 3rd argument set to "scope" is supposed to evaluate this string as gn code and then make it available to the return object (_mac_sdk_result in this case.)

what I don't understand is the stuff that sdk_info.py prints out looks like it should be valid gn code, but it is failing in the tokenizing step (this line I think https://chromium.googlesource.com/chromium/src/+/master/tools/gn/tokenizer.cc#405 )

after doing some digging it appears that gn is being run from buildtools/mac/gn  Is it possible this binary needs to be rebuilt with a more up-to-date version?

Any help would be greatly appreciated. 

What's buildtools/mac/gn --version? The binary should autoupdate when you run `gclient sync` (which you did run).

λ buildtools/mac/gn --version
UNKNOWN

(cd buildtools; git log --oneline | head -1)

?

λ (cd buildtools; git log --oneline | head -1)
0dd5c6f Merge "Revert "Remove libcpp_is_static""

If I read

https://gn.googlesource.com/gn/+log/master

and

https://bugs.chromium.org/p/chromium/issues/detail?id=858852#c3

right, then the UNKNOWN means you have the latest revision (we moved gn around which broke --version, and the binary in buildtools doesn't have the fix for that yet; the flag worked fine before the move).


Since  bug 859536  isn't fixed yet: To debug more, you can build a local gn with `ninja -C out/builddir`...oh wait, you only have a v8 checkout, not a gn checkout.

Well, then you can clone https://gn.googlesource.com/gn/ and build that and add some debug printfs to see what's going wrong. (Everything you say in comment 7 sounds right to me.)

If you just want to build v8 and don't mind some local lingering hacks, you can alternatively remove the machine_os_build print line from sdk_info.py and replace

https://cs.chromium.org/chromium/src/build/config/mac/mac_sdk.gni?q=_mac_sdk_result&sq=package:chromium&g=0&l=87

with just

machine_os_build="17F77"

right in the gn file and see if that helps any.

Thanks for the info!


I've modified mac_sdk.gni as such:

#_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")

xcode_version = "0941" #_mac_sdk_result.xcode_version
xcode_build = "9F2000" #_mac_sdk_result.xcode_build
machine_os_build = "17F77" #_mac_sdk_result.machine_os_build

basically just hardcoding the 3 used settings, and then it works:

λ tools/dev/v8gen.py x64.release -vv
################################################################################
/usr/local/opt/python@2/bin/python2.7 -u tools/mb/mb.py gen -f infra/mb/mb_config.pyl -m developer_default -b x64.release out.gn/x64.release
  
  Writing """\
  is_debug = false
  target_cpu = "x64"
  """ to /Users/michaelreinstein/wwwroot/v8/out.gn/x64.release/args.gn.
  
  /Users/michaelreinstein/wwwroot/v8/buildtools/mac/gn gen out.gn/x64.release --check
  Done. Made 122 targets from 79 files in 311ms

so it definitely is just this one issue with exec_script() not handling being able to parse this into a scope. 


> Since  bug 859536  isn't fixed yet ... you can clone https://gn.googlesource.com/gn/ 
> and build that and add some debug printfs to see what's going wrong.

So it sounds like what might be happening is the gn binary that v8 is using resides in buildtools/mac/gn, and this might not be updated to the newest due to changes to how gn gets included with the project. Is this an accurate re-statement of the problem? I'll try building gn from the link you provided and see if that version solves the problem.


> If you just want to build v8 and don't mind some local lingering hacks,

unfortunately I probably can't have local lingering hacks. I'm ultimately trying to build v8 to include in deno which pulls in v8 as an unmodified dependency (https://github.com/ry/deno)

If the manually built version of gn that I'm about to build works, what's the resolution? 

If the manually-built version of gn at 4cf8d1531 works, then I'd be really confused since my current mental model is that v8 uses the same gn binary as chromium (as of https://chromium.googlesource.com/v8/v8/+/5d4dbb6ee98b18868807830c358c05dcf8f6d219 , which updated buildtools), and that's gn built at that revision.


Hrm, I suppose the standalone gn build just uses whatever clang is installed on the system building gn instead of the hermetic clang the chromium and v8 builds use. Maybe scottmg's Xcode's clang had some compiler bug causing the weirdness you see, and your local clang is different or something?

But let's see if you can still repro with the self-built gn first.

It fails to build:

λ git clone https://gn.googlesource.com/gn
λ cd gn/
λ git checkout 4cf8d1531
Note: checking out '4cf8d1531'.
HEAD is now at 4cf8d153 Make Mac explicitly target 10.9
λ python build/gen.py
λ ninja -C out
ninja: Entering directory `out'
[3/54] LINK gn
FAILED: gn 
ccache clang++ -Qunused-arguments -Wl,-S -flto -mmacosx-version-min=10.9 -o gn tools/gn/gn_main.o -framework AppKit -framework CoreFoundation -framework Foundation -framework Security base.a gn_lib.a
ld: could not process llvm bitcode object file, because /Users/michaelreinstein/wwwroot/v8/third_party/llvm-build/Release+Asserts/lib/libLTO.dylib could not be loaded file 'tools/gn/gn_main.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[12/54] CXX tools/gn/functions_unittest.o
ninja: build stopped: subcommand failed.

if I add the --debug flag as in:

λ python build/gen.py --debug and then run the remaining commands it works though.

The -flto is fromhttps://gn.googlesource.com/gn/+/1a883b0163f400af82a140df2166559779b51b70 ; you remove 'flto' in build/gen.py in the gn repo (from both cflags and ldflags) to work around that.

Alternatively you can remove third_party/llvm-build/Release+Asserts/bin from your PATH to use local xcode clang instead of chromium clang.

The changes you mentioned do indeed allow the build of gn to proceed:

modified the lines in gn/build/gen.py:

cflags.extend(['-O3', '-DNDEBUG'])
#cflags.extend(['-O3', '-flto', '-DNDEBUG'])

ldflags.append('-Wl,-S' if is_mac else '-Wl,-strip-all')
#ldflags.append('-flto')


then ran:

λ rm -rf out/
λ python build/gen.py
λ ninja -C out
ninja: Entering directory `out'
[236/236] LINK gn_unittests

then I copy this binary into the v8 project:
cp out/gn ~/wwwroot/v8/buildtools/mac/

I still get the same error when attempting to run gn on the v8 project

λ tools/dev/v8gen.py x64.release -vv

this still fails with the same error:

ERROR at dynamically parsed input that //build/config/mac/mac_sdk.gni:85:19 loaded :1:1: Invalid token.
  machine_os_build="17F77"

That's good news, now you can add debug printfs :-)

You don't need to copy over the gn binaries, you should be able to run `../../path/to/gn gen out/foobar` while in v8/src to get the same effect.

If I wanted to run my gn build through gdb is this sufficient to get all the debug symbols in place?

λ python build/gen.py --debug
λ ninja -C out

thank you very much for your help so far by the way!

I think that should be enough, but I haven't tried it.

(If you run `ninja -C out gn` then it builds just gn and not gn_unittests, which will be a bit faster.)

cool, I'll work on stepping this through gdb, or if that fails I'll resort to the good ole' fashioned printfs.

> then it builds just gn and not gn_unittests, which will be a bit faster

thanks! 

I think I've figured out what's happening. After running my debug symbol laden copy of gn through gdb, I've found that the input file that gets passed to the tokenizer does has an 8 byte escape sequence at the beginning: \E[?1034h


If I hexdump the output of the python command this can be readily seen:

λ python build/config/mac/sdk_info.py macosx | hexdump -C
00000000  1b 5b 3f 31 30 33 34 68  6d 61 63 68 69 6e 65 5f  |.[?1034hmachine_|
00000010  6f 73 5f 62 75 69 6c 64  3d 22 31 37 46 37 37 22  |os_build="17F77"|
00000020  0a 73 64 6b 5f 62 75 69  6c 64 3d 22 31 37 45 31  |.sdk_build="17E1|
00000030  38 39 22 0a 73 64 6b 5f  70 61 74 68 3d 22 2f 41  |89".sdk_path="/A|
00000040  70 70 6c 69 63 61 74 69  6f 6e 73 2f 58 63 6f 64  |pplications/Xcod|
00000050  65 2e 61 70 70 2f 43 6f  6e 74 65 6e 74 73 2f 44  |e.app/Contents/D|
00000060  65 76 65 6c 6f 70 65 72  2f 50 6c 61 74 66 6f 72  |eveloper/Platfor|
00000070  6d 73 2f 4d 61 63 4f 53  58 2e 70 6c 61 74 66 6f  |ms/MacOSX.platfo|
00000080  72 6d 2f 44 65 76 65 6c  6f 70 65 72 2f 53 44 4b  |rm/Developer/SDK|
00000090  73 2f 4d 61 63 4f 53 58  31 30 2e 31 33 2e 73 64  |s/MacOSX10.13.sd|
000000a0  6b 22 0a 73 64 6b 5f 70  6c 61 74 66 6f 72 6d 5f  |k".sdk_platform_|
000000b0  70 61 74 68 3d 22 2f 41  70 70 6c 69 63 61 74 69  |path="/Applicati|
000000c0  6f 6e 73 2f 58 63 6f 64  65 2e 61 70 70 2f 43 6f  |ons/Xcode.app/Co|
000000d0  6e 74 65 6e 74 73 2f 44  65 76 65 6c 6f 70 65 72  |ntents/Developer|
000000e0  2f 50 6c 61 74 66 6f 72  6d 73 2f 4d 61 63 4f 53  |/Platforms/MacOS|
000000f0  58 2e 70 6c 61 74 66 6f  72 6d 22 0a 73 64 6b 5f  |X.platform".sdk_|
00000100  76 65 72 73 69 6f 6e 3d  22 31 30 2e 31 33 22 0a  |version="10.13".|
00000110  78 63 6f 64 65 5f 62 75  69 6c 64 3d 22 39 46 32  |xcode_build="9F2|
00000120  30 30 30 22 0a 78 63 6f  64 65 5f 76 65 72 73 69  |000".xcode_versi|
00000130  6f 6e 3d 22 30 39 34 31  22 0a 78 63 6f 64 65 5f  |on="0941".xcode_|
00000140  76 65 72 73 69 6f 6e 5f  69 6e 74 3d 39 34 31 0a  |version_int=941.|
00000150


So somehow invoking python is causing this escape sequence to be included in the output.  Now the question seems to be, what is the right way to detect this and prevent python from doing it?

seems to be the relevant python bug: https://bugs.python.org/issue19884

I'd be happy to submit a patch or help with making this more robust for users as this issue was particularly annoying given that it involves invisible control characters.

just let me know how I can help

This seems to be independent of V8. Should we move the issue to Chromium tracker?

I guess it's a question of which project //build/config/mac/mac_sdk.gni is a part of. It contains this line, which is the culprit:

_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")

This invokes python and attempts to parse the results as a gn scope and return it. If this line is run with python < v3.6 and TERM=xterm this will inject the hidden escape sequence and break the build. given that:

* v8 relies on python 2.x
* having TERM=xterm in the environement is a very common environment setting for macosx

This seems like it's worth guarding against in v8, right? Or perhaps I'm misunderstanding the relationship between the chromium and v8 repos?

V8 inherits that from Chromium. It lives in src/build. So I'll move to different tracker.

Moved issue v8:7905 to now be issue chromium:860176.

> V8 inherits that from Chromium. It lives in src/build. 
> So I'll move to different tracker.

Sounds good, thanks!

Maybe also make the description more specific.

Wow, nicely done figuring this out, that sounds gnarly.

I don't see this problem on my mac though. Do you still see it if you use system python instead of the one in /usr/local/opt? (sorry, I missed that you weren't using regular default system python)

If I read https://bugs.python.org/issue19884 right:

- this is only an issue on mac if python uses gnu readline instead of editline, which I think means it doesn't affect system python

- the fix has been merged to the 2.7 branch too, so a recent python 2.7 should be fine too

So one has to be reasonably unlucky to hit this error case.

It's possible that I've misread the python bug, but my current understanding from reading this comment is that the patch is only live in python >= 3.4

https://bugs.python.org/issue19884#msg273764

> which I think means it doesn't affect system python

Hmm... that's interesting! Hadn't considered that.


> So one has to be reasonably unlucky to hit this error case.

I guess I'm just one of the lucky ones. :)

Despite this being a potentially rare bug, it's pretty nasty. I would think this bug could be somewhat easy to work around in the invocation..something along the lines of this pseudocode:

output = `python -c 'import readline' |less`

if(output starts with ESC[?1034h(END)) {
  currentTerm = $TERM

  export TERM=
  invoke script as per normal
  export TERM=currentTerm
} else {
  invoke script as per normal
}

Hm, it's fundamentally a "I mucked with my environment and now things don't work" kind of bug. If there's a simple workaround, we can do it, but generally the number of potential bugs like that is endless, and it's the first time I've heard about this specific bug.

> I mucked with my environment and now things don't work

I didn't muck with anything. Some other sdk has likely done this "helpfully" on my behalf. 


> generally the number of potential bugs like that is endless, 
> and it's the first time I've heard about this specific bug

Sure, I get that this class of problems is bottomless. I'm not demanding anything be done. Ideally this could be fixed but if not I understand the issue hasn't been encountered before and is low priority. I will say though that in my experience one dimension of software quality is how thoroughly it surfaces and works around quirks in the operating environment. :) 

Having this issue searchable in the chromium issue tracker archives might be sufficient for other people that run into this.

Anyway I won't take up more of your time, thanks very much for all of the help!

Setting defect without priority to Pri-2.