@sbc, @dpranke, can you please help with this? We dont understand why this only happens for AMD64 and it seems only when we build with GCC. Started happening yesterday. Any changes you have made in this area?

I'm not sure I'm the best person to debug this, but for reference $GN_ARGS in this build is set to:

GN_ARGS = is_debug=false  use_v4l2_codec=true use_v4lplugin=false use_ozone=true use_evdev_gestures=true use_xkbcommon=true linux_use_bundled_binutils=false use_debug_fission=false enable_remoting=true enable_nacl=true icu_use_data_file=true use_cras=true use_system_harfbuzz=true use_cups=true is_asan=false is_clang=false cros_host_is_clang=false clang_use_chrome_plugins=false ozone_auto_platforms=false ozone_platform_gbm=true use_system_minigbm=true is_chrome_branded=true is_official_build=true internal_gles2_conform_tests=true symbol_level=2 target_sysroot="/build/samus" system_libdir="lib64" pkg_config="/build/samus/build/bin/pkg-config" target_os="chromeos" ozone_platform="gbm" target_cpu="x64" cros_target_ar="x86_64-cros-linux-gnu-ar" cros_target_cc="x86_64-cros-linux-gnu-gcc -B/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.25.51-gold" cros_target_cxx="x86_64-cros-linux-gnu-g++ -B/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.25.51-gold" host_toolchain="//build/toolchain/cros:host" custom_toolchain="//build/toolchain/cros:target" v8_snapshot_toolchain="//build/toolchain/cros:v8_snapshot" cros_target_ld="x86_64-cros-linux-gnu-g++ -B/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/binutils-bin/2.25.51-gold" cros_target_extra_cflags="-pipe -pipe -march=corei7 -fauto-profile=/build/samus/tmp/portage/chromeos-base/chromeos-chrome-57.0.2946.0_rc-r1/work/afdo/chromeos-chrome-amd64-57.0.2946.0_rc-r1.afdo -Wno-error" cros_target_extra_cppflags="" cros_target_extra_cxxflags="-pipe -pipe -pipe -march=corei7 -D__google_stl_debug_vector=1 -fauto-profile=/build/samus/tmp/portage/chromeos-base/chromeos-chrome-57.0.2946.0_rc-r1/work/afdo/chromeos-chrome-amd64-57.0.2946.0_rc-r1.afdo -Wno-error" cros_target_extra_ldflags="-Wl,-O1 -Wl,-O2 -Wl,--as-needed" cros_host_cc="x86_64-pc-linux-gnu-gcc" cros_host_cxx="x86_64-pc-linux-gnu-g++" cros_host_ar="x86_64-pc-linux-gnu-ar" cros_host_ld="x86_64-pc-linux-gnu-g++" cros_host_extra_cflags="" cros_host_extra_cxxflags="" cros_host_extra_cppflags="" cros_host_extra_ldflags="" cros_v8_snapshot_cc="x86_64-pc-linux-gnu-gcc" cros_v8_snapshot_cxx="x86_64-pc-linux-gnu-g++" cros_v8_snapshot_ar="x86_64-pc-linux-gnu-ar" cros_v8_snapshot_ld="x86_64-pc-linux-gnu-g++" cros_v8_snapshot_extra_cflags="" cros_v8_snapshot_extra_cxxflags="" cros_v8_snapshot_extra_cppflags="" cros_v8_snapshot_extra_ldflags=""

@sbc, How can I verify the values for GN_ARGS, Since it it not printed when the build failed (https://uberchromegw.corp.google.com/i/chromeos/builders/amd64-llvm-toolchain/builds/89/steps/SimpleChromeWorkflow/logs/stdio ).

I looked at the stdio of the very first "steps" which I think is the sum of all the steps... and found it in there.  Not sure which substep its part of.

I have no idea how this failure could be intermittent.

I tried this locally and it worked fine:

gn gen out/cros --args='target_os="chromeos" host_toolchain="//build/toolchain/cros:host" custom_toolchain="//build/toolchain/cros:target" cros_v8_snapshot_extra_ldflags=""'

Any idea how to find out exactly which chromium revision is being built here?

the buildbot log states what version is used if you check out the summary:
SyncChrome SyncChrome
tag 57.0.2946.0

Regarding #4, Those values are used in chroot. Not in SimpleChrome.

The builder builds chrome twice,

First: Within chroot (Note, it is using gcc as cros_host_cc). I believe this is where the above values came from.

Second: SimpleChrome build. This is where it failed.

Can you find the gn args from the failing build.

If custom_toolchain="//build/toolchain/cros:target" is not specified then most of the cros_* options are not available because `build/toolchain/cros_toolchain.gni` doesn't get included.

Unfortunately, The failing build log for SimpleChrome does not output GN_ARGS values. 

I ran the buildbot build locally and it seems to be going fine. 
I am trying to get access to the builder to check if there is a builder issue and if I can reproduce it there.

From my local build, custom_toolchain = "//build/toolchain/cros:target"
Also attaching the GN_ARGS values.

Deleted: local_samus.log 4.5 KB

local_samus.log 4.5 KB View Download

I do not see anything obviously wrong in the build files, and none of that stuff has changed recently. There was a roll of GN on Dec 5th, but that wouldn't explain why things started failing only three days later.

If you continue to see this on the builder but run out of other ideas for how to figure out what's going on, and/or if you are able to reproduce it locally, let me know.

Build passed today so maybe it was some server flakiness.

Some builders still failed in SimpleChrome flow. I will try accessing the trybot to check.

Seems to happen (intermittently?) on x86-generic-full as well:

https://build.chromium.org/p/chromiumos/builders/x86-generic-full/builds/19332

https://build.chromium.org/p/chromiumos/builders/x86-generic-full/builds/19328

Tried reproducing on buildbot cros-beefy272-c2 where it failed today for amd64-llvm-next-toolchain. 

Ran the build steps 20 times in a loop for SimpleChrome (post sysroot generation) a few times.
a) gn Gen only
b) 3 steps - True (This only fetches files?), runhooks and gn Gen.

But no luck.

There are a few steps done in parallel with sysroot generation.
Is there a timing issue somewhere? Maybe some files are not generated before gn Gen is executed?

Deleted: run_all_steps_20.sh 1.7 KB

run_all_steps_20.sh 1.7 KB View Download

Deleted: run_gn_only_20.sh 890 bytes

run_gn_only_20.sh 890 bytes View Download

Attaching log for today's fail on amd64-llvm-next build.

Deleted: amd64_failed.log 80.3 KB

amd64_failed.log 80.3 KB View Download

I also verified that gn_args are same for both passing and failing gn Gen Runs (attached).
dpranke@ Can you take a look or  help in reproducing the issue?

Deleted: gn_args.txt 3.8 KB

gn_args.txt 3.8 KB View Download

We've been seeing a few other similar flakes lately (see my comments on  bug 674213 ). 

Is this something that is failing consistently at this point?

@dpranke, Can you provide a patch that reverts the change you referred on  bug 674213 . We can try to apply the patch to our waterfall builders locally and see if the flakiness in GN goes away.
Alternately, Is there some logging mechanism in GN that we can turn on e.g.  print the files it processed etc.?

Failed today again for 2 builders: arm-llvm-toolchain and x86-llvm-next-toolchain

https://uberchromegw.corp.google.com/i/chromeos/builders/arm-llvm-toolchain/builds/95/steps/SimpleChromeWorkflow/logs/stdio

https://uberchromegw.corp.google.com/i/chromeos/builders/x86-llvm-next-toolchain/builds/95/steps/SimpleChromeWorkflow/logs/stdio

Hm, those appear to be two different problems:

First appears to be a compile flag conflict:

armv7a-cros-linux-gnueabi-g++ -B/tmp/cbuildbot-tmp2B3216/chrome-sdk-cacheK14JGi/cache/chrome-sdk/tarballs/veyron_jaq+__b__cbuild__internal_master__buildbot_archive__arm-llvm-toolchain__R57-9089.0.0-rc1+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -MMD -MF obj/base/base_static/switches.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_ASH=1 -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DENABLE_WAYLAND_SERVER=1 -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DENABLE_MEDIA_ROUTER=1 -DFIELDTRIAL_TESTING_ENABLED -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -DOS_CHROMEOS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -fno-strict-aliasing -funwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -march=armv7-a -mfloat-abi=hard -mtune=generic-armv7-a -pthread -mfpu=neon -mthumb -Wall -Werror -Wno-psabi -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-missing-field-initializers -Wno-unused-parameter -g2 -gsplit-dwarf --sysroot=../../../../../../../../../../tmp/cbuildbot-tmp2B3216/chrome-sdk-cacheK14JGi/cache/chrome-sdk/tarballs/veyron_jaq+__b__cbuild__internal_master__buildbot_archive__arm-llvm-toolchain__R57-9089.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz -fvisibility=hidden -O2 -fno-ident -fdata-sections -ffunction-sections -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -Wno-narrowing -Wno-literal-suffix -fno-rtti -fno-exceptions -pipe -mcpu=cortex-a12 -mtune=cortex-a12 -mfpu=neon -mfloat-abi=hard -D__google_stl_debug_vector=1 -c ../../base/task_scheduler/switches.cc -o obj/base/base_static/switches.o
../../base/task_scheduler/switches.cc:1:0: error: switch -mcpu=cortex-a12 conflicts with -march=armv7-a switch [-Werror]



The second appears to be a GN configuration error:

10:44:47: DEBUG: RunCommand: /tmp/cbuildbot-tmpzuOCeX/chrome-sdk-cache2fsQdF/cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls
ERROR at the command-line "--args":36:34: Build argument has no effect.
cros_v8_snapshot_extra_ldflags = ""
                                 ^-
The variable "cros_v8_snapshot_extra_ldflags" was set as a build argument
but never appeared in a declare_args() block in any buildfile.

To view all possible args, run "gn args --list <builddir>"
ERROR No targets.
I could not find any targets to write, so I'm doing nothing.
ERROR 

regarding #24, I opened https://bugs.chromium.org/p/chromium/issues/detail?id=674696 for the first issue.

Second issue is the flakiness in GN that we are discussing.

Sorry for the lack of an update; we're seeing similar issues on a number of different builders, so I'm just going to revert the change and roll chrome forward and we can see if this goes away. Does that sound good?

There's another bug for the GN flakiness, I'm going to make this depend on that one for now ...

Okay, the GN change has been reverted; can you try this again?

I got sidetracked on other issues and forgot to update this, but I saw a similar error when trying to build on a branch, where the GN I was using was newer but the branch did not have the GN changes.

I think we are going to have to be careful about introducing backward-incompatible changes to GN, since devs like myself work on multiple branches with the same depot_tools.

I don't know if the solution is versioning or using warnings instead of errors for a few releases, but requiring devs to sync depot_tools with a branch would be a real headache.

All toolchain waterfall builders did not have this GN error in last 2 days.

That's good! Let's give it a few more days to see ...

@stephenjb - in chromium builds, the version of GN that the builds use is pinned via DEPS in the repo itself, so we don't have a compatibility problem. Is the CrOS build just pulling GN from head somewhere? 

I was speaking only about building as a developer. Or are you saying that the 'gn' in depot_tools is actually a wrapper that runs a version of gn associated with the checkout? If that is the case, that was broken when I last attempted to build the 56 branch (middle of last week sometime).

As for CrOS builds, it may be trickier. There is Simple Chrome, and there is building in the chroot environment. I believe the chroot has it's own depot_tools, but maybe that doesn't matter.

Yes, it's a wrapper, and yes, it's broken for branch builds (bug 658104).

OK, cool. Thanks.

Sorry, still happening!

https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-release/builds/728

23:10:02: DEBUG: Uploading command stats to 'https://chromiumos-build-stats.appspot.com/upload_command_stats'
23:12:08: INFO: RunCommand: cros --log-level debug --cache-dir /tmp/cbuildbot-tmp1V88_r/chrome-sdk-cache6V5Vdl/cache chrome-sdk --board cyan --cwd /b/cbuild/internal_master/.cache/distfiles/target-master/chrome-src-internal/src --sdk-path /b/cbuild/internal_master/buildbot_archive/cyan-release/R57-9146.0.0 --chrome-src /b/cbuild/internal_master/.cache/distfiles/target-master/chrome-src-internal/src --nogoma -- bash -c '/b/cbuild/internal_master/.cache/distfiles/target-master/chrome-src-internal/src/buildtools/linux64/gn gen "/b/cbuild/internal_master/.cache/distfiles/target-master/chrome-src-internal/src/out_cyan/Release" --args="$GN_ARGS"' in /b/cbuild/internal_master
/b/cbuild/internal_master/chromite/third_party/google/protobuf/__init__.py:37: UserWarning: Module simplejson was already imported from /b/build/third_party/simplejson/__init__.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
  __import__('pkg_resources').declare_namespace(__name__)
23:12:10: DEBUG: Configured cache_dir to '/tmp/cbuildbot-tmp1V88_r/chrome-sdk-cache6V5Vdl/cache'
23:12:11: DEBUG: RunCommand: /tmp/cbuildbot-tmp1V88_r/chrome-sdk-cache6V5Vdl/cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls
ERROR at the command-line "--args":36:34: Build argument has no effect.
cros_v8_snapshot_extra_ldflags = ""
                                 ^-
The variable "cros_v8_snapshot_extra_ldflags" was set as a build argument
but never appeared in a declare_args() block in any buildfile.

To view all possible args, run "gn args --list <builddir>"
ERROR No targets.
I could not find any targets to write, so I'm doing nothing.
ERROR 

What version of GN is running in that build? Can you confirm that either you're running the GN binary pulled in via Chromium's src/buildtools and have this change:

https://chromium.googlesource.com/chromium/src/+/1791c73e9c10c356a1121f85f685de401e27a511

or that you have a version of GN that has this fix:

https://codereview.chromium.org/2616523002/

? Either way, `gn --version` should report something > 439366.

Ningning, can you help with this?  Thanks!

Checked the version on cyan builder.

chrome-bot@build152-m2:(Linux 12.04):/b/cbuild/internal_master/chroot/usr$ bin/gn --version
439377

Looks like not the version we want (439366) ?

well, it's newer, so, in theory that'd be good enough.

However, in practice, my change didn't fix things anyway :).

Once  bug 674213  is fixed, this problem should go away. brettw@ owns that bug, so I'm not sure if there's anything more to do here. How do you want to track this? Make brettw@ the owner? Some CrOS team member? 

This is hopefully fixed now. I'll close this; please reopen if it occurs again and we're sure we have the fix (gn --version returns 443805 or higher).

Closing. Please reopen it if its not fixed. Thanks!