The current build page for this bot shows almost total failure:

https://build.chromium.org/p/tryserver.chromium.android/builders/android_arm64_dbg_recipe

-Sheriff +Trooper

emea trooper fly-by-comment:
  milo has more specific time when this started failing:
    https://luci-milo.appspot.com/buildbot/tryserver.chromium.android/android_arm64_dbg_recipe/?cursor=CnoKDgoGbnVtYmVyEgQIkIESEmRqC3N-bHVjaS1taWxvclULEg1idWlsZGJvdEJ1aWxkIkJbInRyeXNlcnZlci5jaHJvbWl1bS5hbmRyb2lkIiwiYW5kcm9pZF9hcm02NF9kYmdfcmVjaXBlIiwiMjk1MDU2Il0MGAAgAQ&limit=200 
    (as of now, it's somewhere around ~400 builds ago)


2017-06-23 4:04 AM (CEST)		Failed	#294943	(dbeam@chromium.org)	failed compile (with patch) 
failed compile (without patch) 
failed Failure reason
2017-06-23 4:06 AM (CEST)		Failed	#294942	(chaopeng@chromium.org)	failed compile (with patch) 
failed compile (without patch) 
failed Failure reason
2017-06-23 4:34 AM (CEST)		Success	#294941	(yhirano@chromium.org)	build successful
2017-06-23 4:24 AM (CEST)		Success	#294940	(slangley@chromium.org)	build successful
2017-06-23 4:18 AM (CEST)		Failed	#294939	(patricialor@chromium.org)	failed compile (with patch) 
failed Failure reason
2017-06-23 4:29 AM (CEST)		Success	#294938	(agrieve@chromium.org)	build successful

It's a CQ builder, so raising the priority accordingly. Andrii said he'll take the bot off the CQ.. thx!

 Issue 736264  has been merged into this issue.

removing builder from CQ: https://chromium-review.googlesource.com/c/544884/

The issue seems more widespread and not android only.
On this CL https://chromium-review.googlesource.com/c/544482/ I just saw:

win_chromium_compile_dbg_ng [1]: 
ninja: error: unknown target 'cloud_print/virtual_driver/win/port_monitor:port_monitor_dll(build/toolchain/win:x64)'

linux_chromium_rel_ng [2]:
ninja: error: unknown target 'components/nacl/loader:nacl_helper_nonsfi_copy(build/toolchain/nacl:newlib_pnacl_nonsfi)'



[1] https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.win%2Fwin_chromium_compile_dbg_ng%2F440713%2F%2B%2Frecipes%2Fsteps%2Fcompile__without_patch_%2F0%2Fstdout
[2] https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_rel_ng%2F486362%2F%2B%2Frecipes%2Fsteps%2Fcompile__without_patch_%2F0%2Fstdout

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0a2e609cdc6ec3678f8340915b0ebc18c61b6e30

commit 0a2e609cdc6ec3678f8340915b0ebc18c61b6e30
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 23 09:37:58 2017

Temporary remove android_arm64_dbg_recipe from CQ.

TBR=jochen@chromium.org,mgiuca@chromium.org

Bug:  736215 
Change-Id: I10fe5c623fdedd51ab0039132a07b8a413a06978
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/544885
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#481837}
[modify] https://crrev.com/0a2e609cdc6ec3678f8340915b0ebc18c61b6e30/infra/config/cq.cfg

Thanks to machenbach@, the culprit found: https://codereview.chromium.org/2953643004

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/62cb083b0f12f5ebc6c3fe3d324c039feea39948

commit 62cb083b0f12f5ebc6c3fe3d324c039feea39948
Author: tandrii <tandrii@chromium.org>
Date: Fri Jun 23 09:47:02 2017

Revert of Roll buildtools 7f2cacbbe2..38477c2e10 (patchset #1 id:1 of https://codereview.chromium.org/2953643004/ )

Reason for revert:
Probably broke incremental builds on trybots.

BUG= 736215 

Original issue's description:
> Roll buildtools 7f2cacbbe2..38477c2e10
>
>   In order to roll GN 7c9bd5f020..bedb4b202b (r465654:r481731) and pick up
>   the following changes:
>
>   a104997ecbfd Allow overriding script executable in .gn file
>   18697c83b41f Move if, else, true, false into a keyword category
>   d4a5e98235da [tracing] Switch to new heap dump format.
>   8480d86659b8 tools/gn: Implement Windows SDK version command line switch
>   3e0d5228e24d Use ContainsValue() instead of std::find() in tools/
>   e4a28698a612 Fix GN bootstrap
>   118577369265 [spelling] existance to existence
>   21886543a0cb Fix a bug in `gn analyze` for host-only file mods.
>   f8bf5083fb27 Fix gn get_label_info comment
>   05f56d412142 allocator: remove ENABLE_MEMORY_TASK_PROFILER and use only USE_ALLOCATOR_SHIM
>   e329e9ec19db Fix GN bootstrap
>   2928b5dc30a1 Add information on the requirements for a new Chromium port.
>   73228cd43d2e allocator: rename use_experimental_allocator_shim to use_allocator_shim
>   797723cd4453 Fix GN bootstrap
>   84fa8b0864ae Replace sanitizers:deps with exe_and_shlib_deps (Chromium repo only)
>   d77555bcaf03 GN: Fix single-file compilations in VS2017 projects
>   cbbd560a5c57 gn: get rid of GetRealPath() function
>   cf2a4bf9cb35 gn: convert WrapUnique calls to MakeUnique where possible
>   dfb837a97c2e gn: fix bootstrap on windows
>   777e1cd5ec01 Fix gn bootstrap after https://codereview.chromium.org/2872503003
>   25e109413ae5 gn: Document gen's --check flag.
>   145252018a78 Add a GUID to base::SharedMemoryHandle.
>   b154afcbc07f memory-infra: Move dump level check to observer and rename session state
>   9b718c790912 Fix GN bootstrap
>   643d6bc90fea gn: fix bootstrap.py after https://codereview.chromium.org/2846893003
>   dd1ae15f741a gn docs: Add a missing closing paren.
>   0088ee5017f4 GN: aix port along with linux_s390x, linux_ppc64 and linux_ppc64le support.
>   4a4589fc42e3 memory-infra: Start disentangling tracing from memory-infra
>   7fd3f6718896 Add curly brackets to list of characters that gn needs to escape
>   ca2ec763405c Improvements to uses of base::SmallMap
>   ddb3babce43a Use $root:default as a "default" rule
>
> TBR=brettw@chromium.org
>
> Review-Url: https://codereview.chromium.org/2953643004
> Cr-Commit-Position: refs/heads/master@{#481774}
> Committed: https://chromium.googlesource.com/chromium/src/+/3391d2ce4059ebe801dd8310d13d8dc59293600e

TBR=brettw@chromium.org,dpranke@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Review-Url: https://codereview.chromium.org/2956623002
Cr-Commit-Position: refs/heads/master@{#481839}

[modify] https://crrev.com/62cb083b0f12f5ebc6c3fe3d324c039feea39948/DEPS

https://codereview.chromium.org/2953643004 DEPS roll of buildtools which contains roll of gn did pass through CQ. So, it's likely incremental build which got borked, and that's why we saw semi-flaky failures.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e12d437dc7f395d72995b548c9dacf21b0b1526e

commit e12d437dc7f395d72995b548c9dacf21b0b1526e
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 23 09:51:50 2017

Revert "Temporary remove android_arm64_dbg_recipe from CQ."

This reverts commit 0a2e609cdc6ec3678f8340915b0ebc18c61b6e30.

Reason for revert: typo && not useful any more.

Original change's description:
> Temporary remove android_arm64_dbg_recipe from CQ.
> 
> TBR=jochen@chromium.org,mgiuca@chromium.org
> 
> Bug:  736215 
> Change-Id: I10fe5c623fdedd51ab0039132a07b8a413a06978
> No-Try: True
> Reviewed-on: https://chromium-review.googlesource.com/544885
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#481837}

TBR=mgiuca@chromium.org,tandrii@chromium.org,jochen@chromium.org

Change-Id: Ie58bf3ed01bc35899ec73a63dd1db90ec797becd
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  736215 
Reviewed-on: https://chromium-review.googlesource.com/544826
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#481840}
[modify] https://crrev.com/e12d437dc7f395d72995b548c9dacf21b0b1526e/infra/config/cq.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a84d39ad4042857ced02e1b1585b7d29aa8f6c09

commit a84d39ad4042857ced02e1b1585b7d29aa8f6c09
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Jun 23 10:04:13 2017

Landmine to ensure GN roll revert has effect everywhere.

TBR=machenbach@chromium.org

Bug:  736215 
Change-Id: I30b49f478c8c6ba9a6507092924a6f05a5f6ea8e
No-Try: True
Reviewed-on: https://chromium-review.googlesource.com/544867
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#481841}
[modify] https://crrev.com/a84d39ad4042857ced02e1b1585b7d29aa8f6c09/build/get_landmines.py

So, landmin landed and there are no failures yet...

First landmined builds finished green. I don't see any failed compile steps any more due to missing target. So, assigning to Dirk to figure out what went wrong in Gn world. 

Dirk, among many CLs above, the only relevant revert is this:
https://codereview.chromium.org/2956623002

I don't think we needed the landmine. I'm pretty sure that the my fix in 

https://chromium.googlesource.com/chromium/src/+/21886543a0cb

was the problem, and it just would've always broken any `analyze` call on a bot that was trying to build "all". I'm going to revert that CL and then re-land a new GN roll.

I'm closing this bug since the revert was the sufficient to fix things.

Can we make landmines harder to land so that they get used less often? Or can we come up with a new type of landmine that only applies on the bots? Landmines are a very big and disruptive hammer and they tend to behave poorly on developer machines (they are currently failing on my machine because I have Visual Studio open after debugging a chrome binary that I built, they blow away .sln and .vcxproj files, etc.)

I guess I should open a separate bug for this rant...

Yup, that's probably a separate bug for the rant :).

I don't want to cast any blame on tandrii@ while he was trying to do the right thing, but we probably want some combination of (a) you shouldn't land a landmine unless you are positive it is needed, and (b) we should think about whether there is some way to undo the landmine.

Filed  bug 736939  for the separate rant / request to make them undoable :).

I've also filed  bug 736941  in order to fix the root cause (analyze being bypassed during a GN roll).

Sorry for the disruption. The landmine suggestion is on me.

But a simple comment in the end of the file saying "Don't do this!" would have indeed made us think twice and wait longer.

https://cs.chromium.org/chromium/src/build/get_landmines.py?dr=C&l=26

  # DO NOT add landmines as part of a regular CL. Landmines are a last-effort
  # bandaid fix if a CL that got landed has a build dependency bug and all bots
  # need to be cleaned up. If you're writing a new CL that causes build
  # dependency problems, fix the dependency problems instead of adding a
  # landmine.


I guess it could be at the end of the file instead.

I think the real issue here is that adding "TBR=anything" to a CL description completely skips OWNERS checks. It should only assume lg's from people listed in the TBR line and still do an owners computation. All build/owners would've known not to do this.

Is this supposed to be fixed?
I'm now consistently getting 
ninja: Entering directory `E:\b\c\b\win\src\out\Release'
ninja: error: unknown target 'cloud_print/virtual_driver/win/port_monitor:port_monitor_dll(build/toolchain/win:x64)'
step returned non-zero exit code: 1

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.win%2Fwin_chromium_rel_ng%2F477339%2F%2B%2Frecipes%2Fsteps%2Fcompile__without_patch_%2F0%2Fstdout

Reopening as it appears to be back.

If you run `gn --version`, what do you get?

The change that caused the error you're seeing was reverted, and it should be gone.

How do I run 'gn --version' on the bot?

ah, sorry, I missed that this was on the bot. Looking further.

I'm not actually sure what's going on here, and I'll have to look into it further. AFAIK, this isn't broadly broken but I'll keep an eye on things.

It seems pretty consistently broken on win_chromium_rel_ng and win_chromium_compile_dbg_ng for me...

Agreed. See this change as an example
https://chromium-review.googlesource.com/c/550595/

Due to the urgency of this change, I just git cl land'ed it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dcd2a9c3b60b39d13cf676170c44fd21c2dd9b4a

commit dcd2a9c3b60b39d13cf676170c44fd21c2dd9b4a
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Jun 28 01:28:17 2017

Revert "migrate more builders to the src side"

This reverts commit 233794b894e0f0c51ad93bccf02097cf5e7169ae.

Reason for revert: I think this is causing build failures because of the change to the "all" target on the windows bots. I am still investigating this; see  crbug.com/736215 .

Original change's description:
> migrate more builders to the src side
> 
> This is part of the GYP_GONE work to remove chromium_builder_tests
> target and to specify compilation targets in the src side
> rather than on the build side.
> 
> As from a manual inspection these added all the builders that
> were referrencing 'chromium_builder_tests' target.
> 
> BUG=None
> 
> Change-Id: I7093840926505f9ed109b53f8297f7d24fb55b12
> Reviewed-on: https://chromium-review.googlesource.com/543046
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Commit-Queue: Thiago Farina <tfarina@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#481288}

TBR=tfarina@chromium.org,dpranke@chromium.org
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true

Bug:  736215 
Change-Id: I73e62b26f81d859f7c4bb85452e61da4582275c5
Reviewed-on: https://chromium-review.googlesource.com/551187
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#482842}
[modify] https://crrev.com/dcd2a9c3b60b39d13cf676170c44fd21c2dd9b4a/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/dcd2a9c3b60b39d13cf676170c44fd21c2dd9b4a/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/dcd2a9c3b60b39d13cf676170c44fd21c2dd9b4a/testing/buildbot/chromium.win.json

My current theory is that https://chromium-review.googlesource.com/c/543046/ is the source of the current problem (or part of the problem). That CL added "all" as an additional compile target to the bots; before that CL we would only try to build the affected tests.

My theory is that GN's analyze was broken for "all" on windows even ignoring the change in comment #18.

The win clang bots have been building all for a long time, so it's at least mostly working

good point. the mystery deepens ...

win_clang is a 64-bit builder; I think the problem with the failing target -- cloud_print/virtual_driver/win/port_monitor:port_monitor_dll(build/toolchain/win:x64) -- only exists when doing 32-bit builds.

From my testing in https://codereview.chromium.org/2957213003/, I think the problem is worked-around for now. In patchset #1, you can see a build where I changed one of the files robliao changed, and the try job succeeds. In patchset #3, where I essentially re-applied tfarina's change, the try job fails.

I don't understand yet why analyze is failing this way. The obvious options are (a) my revert didn't work right, and we're still getting the broken binary on windows, or (b) a different bug has been there all along and "all" would've been broken with analyze on an x86 builder.

So far, in addition to my tests above, I haven't seen any other compile failures due to this since my revert landed. I will check again in another hour or two to confirm, but likely otherwise pick this up again tomorrow.

https://build.chromium.org/p/chromium.fyi/console?category=win%20clang has 32bit bots building all. Looks like they turned red on the buildtools roll and haven't recovered yet.

You're right, but there was a skia roll that happened at about the same time and was picked up in the same builds; all the compile failures look like they're link failures due to missing skia symbols.

Building the actual "all" target wouldn't have the same problem the trybots have, which is that analyze returns the literal string listed in comment #39, but you can't pass that string to ninja, you need the output_name instead of the gn label.

We also build "all" on the /p/chromium/builders/Win builder, and that's been green the whole time.

My coment from #40 still stands, I think the workaround worked. It looks like there's actually a separate bug in the way `analyze` works that is showing up only on windows and has been there all along (no pun intended).

I'll fix that case the same time I fix bug 667989.

(to follow up on 42: you're right, i analyzed this poorly yesterday (was on a phone). i followed up on the skia roll today, and the skia folks fixed our 32-bit clang bots again)

I can't land my CL because of this: https://chromium-review.googlesource.com/c/chromium/src/+/721689

As you can see android_arm64_dbg_recipe always fails, and it always fails with the same  error (for both with and without patch compilations):

ninja: error: unknown target 'android_webview:libwebviewchromium(build/toolchain/android:android_clang_arm)'

Also seeing the same problem: https://chromium-review.googlesource.com/c/chromium/src/+/738536, from looking at the bot it seems that all builds are failing except for those that don't actually compile anything (although I can't verify this since logs aren't loading for me for some reason).

We're also seeing it here: https://chromium-review.googlesource.com/c/chromium/src/+/738775

Friendly ping. My 2 CLs are bitrotting because of this.

jbudorick@, can you help resolving this?

If you can, I suggest you simply bypass the error for now by using NOTRY=true.

Fixing this is, I think slightly complicated and I don't know that I'll be able to get to it in the next day or two.

#57: what do you have in mind for a fix?

From my end, this appears to be in gn one way or another, either in what it's emitting from analyze (too many intermediate targets?) or in the .ninja files it writes. I'm not very familiar w/ gn's implementation, though -- would you say that's a fair assessment?

hrm, we almost want something like an opaque group that analyze doesn't peer through w/ Analyzer::Filter...

the problem is that analyze isn't producing the right output target names for things in the non-default toolchain.

Those targets are supposed to be filtered out, but that logic isn't working right, either. 

Either or both of these things should be relatively easy to fix, and I've cleared out some of the other stuff I was working on, so I should be able to look at this later today.

dtrainor: I believe this is what's making your CL https://chromium-review.googlesource.com/c/chromium/src/+/693320 constantly fail.

It looks like there are a couple of different problems here.

The first is that if your change affects a target that is built with one of the alternate toolchains, then GN returns the label of that toolchain from analyze (e.g., `//chrome/android:monochrome(..android_clang_arm)` rather than `//chrome/android:monochrome`; however, (a) MB doesn't know how to deal with translating the toolchain part of the label back into the matching ninja target, partially because (b) GN doesn't generate phony targets for targets in alternate toolchains.

This shows up most commonly in the android monochrome builds (which build lots of things for both arm32 and arm64), but will also show up in the x86 windows build (where we build some things for both x86 and x64), and can potentially show up even for things in other toolchains.

The right long-term fix is probably to make GN return the output paths for the targets in `compile_targets`, rather than the labels; however, that'll be an interface change so we might want to use a flag to control this.

A related issue (which we haven't hit yet, probably because we're hitting the above issue first), is that if you pass in `all` as a compile target, it's possible that your change may affect so many compile targets that we'll ultimately make the command line we pass to ninja too long. Probably the right fix for this is to change ninja to take an @rsp file as input. 

For the moment, I'm going to change MB to detect these situations and at least handle them conservatively so that the try jobs don't fail; however, I'll leave this bug open until we can fix the underlying issues as well. 

I filed https://github.com/ninja-build/ninja/issues/1355 for the enhancement request for Ninja.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/45165076d5eb711bfe6e0a1847e2fbd3c72d7c2b

commit 45165076d5eb711bfe6e0a1847e2fbd3c72d7c2b
Author: Dirk Pranke <dpranke@chromium.org>
Date: Wed Nov 08 04:57:49 2017

Make MB handle non-default toolchains and lots of results better.

If someone uploads a patch that causes analyze to return either
way too many files as a result, or return targets that are
built using toolchains other than the default, bad things can
happen.

This CL modifies MB to detect those situations and handle them
more robustly, until we can roll fixes into GN and Ninja that
will deal with this situation better.

R=jbudorick@chromium.org, brettw@chromium.org
BUG= 736215 

Change-Id: I9e678136d151b6636fc1581c92d7e0b0626a9342
Reviewed-on: https://chromium-review.googlesource.com/757563
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#514756}
[modify] https://crrev.com/45165076d5eb711bfe6e0a1847e2fbd3c72d7c2b/tools/mb/mb.py
[modify] https://crrev.com/45165076d5eb711bfe6e0a1847e2fbd3c72d7c2b/tools/mb/mb_unittest.py

I've filed bug 782716 to improve GN's behavior here.

The "bots fail to build" problem should be fixed now, so I'm going to close this to avoid confusion.