This looks like WAI WONTFIX  bcz of "with patch". Sergiy, why is compile with patch reported, and why to troopers?

Why makes you think that compile with patch being flaky is WAI?

Troopers bcz compile flakiness is often infrastructure-related. If you believe this is a sheriff issue, please follow instructions from the description and move this issue to the sheriff queue.

My logic was "with patch" => could be caused by bad patch -> hence flaky by design. But surely flakiness reports this IFFF there is a successful build with the same patchset. 

Back to trooper queue, not Pri0, not for emea trooper.

Yes. That's the case. Each flake means that there is passing and failing build from the same builder on the same patchset. Besides we require at least 3 pairs of passing/failing builds to file a bug, which should reduce number of false positives.

Detected 3 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

These don't look like infra failures to me. I'm bouncing this over to the Sheriff queue. There's at least a couple of different causes of flakes, including a flake building the v8 snapshot, and a flake building assets on ios.

Detected 16 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

This seems to be a bug in the sheriff-o-matic tool.

Detected 8 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 14 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Can you explain what you meant in comment 11, Rouslan? Why is this a sheriff-o-matic bug?

Erik: "Compile with patch" is a step in try-bots. Try-bots are used for work in progress, which is not ready to land.

Sheriff-o-matic should concern itself only with build-bots. Build-bots don't have a "compile with patch" step. Therefore, Sheriff-o-matic should not look into try-bot failures. 

Is my understanding correct here?

The flakiness pipeline looks at tryjob results between identical patchsets and files bugs when the results are different (i.e. something is flaky). So the tooling is working as intended by looking at tryjobs.

I'm going to put this back in the sheriff queue since this looks to be an actual flake.

Detected 4 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

Detected 3 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 16 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

There are a few different things showing up in the list of flakes, but ios-device-xcode-clang dominates the list, so scoping to that.

Those failures look like this, or similar:

[23901/23907] COMPILE_XCASSETS Chromium.app/Assets.car
FAILED: Chromium.app/Assets.car 
export DEVELOPER_DIR=/b/build/slave/ios-device-xcode-clang/build/src/build/ios_files/Xcode.app;  rm -f Chromium.app/Assets.car && TOOL_VERSION=1498868417 python ../../build/toolchain/mac/compile_xcassets.py -p iphoneos -t 9.0 -T com.apple.product-type.application -o Chromium.app/Assets.car gen/third_party/material_design_icons/ic_arrow_back.xcassets gen/third_party/material_design_icons/ic_chevron_right.xcassets gen/third_party/material_design_icons/ic_close.xcassets gen/third_party/material_design_icons/ic_info.xcassets gen/third_party/material_design_icons/ic_search.xcassets gen/ios/chrome/browser/ui/qr_scanner/qr_scanner_torch_off.xcassets gen/ios/chrome/browser/ui/qr_scanner/qr_scanner_torch_on.xcassets gen/ios/chrome/browser/ui/activity_services/activity_services_print.xcassets gen/ios/chrome/browser/ui/first_run/checkbox.xcassets gen/ios/chrome/browser/ui/first_run/checkbox_checked.xcassets gen/ios/chrome/browser/ui/downloads/error_icon.xcassets gen/ios/chrome/browser/ui/downloads/file_icon_body.xcassets gen/ios/chrome/browser/ui/downloads/file_icon_fold.xcassets gen/ios/chrome/browser/ui/downloads/file_icon_fold_complete.xcassets gen/ios/chrome/app/resources/launchscreen_app_logo.xcassets gen/ios/chrome/app/resources/launchscreen_brand_name.xcassets
2017-07-02 23:38:05.973 ibtoold[20617:38149] CoreSimulator is attempting to unload a stale CoreSimulatorService job.  Existing job (com.apple.CoreSimulator.CoreSimulatorService.179.1.E8ttyeDeVgWK) is from an older version and is being removed to prevent problems.
/* com.apple.actool.compilation-results */
/b/build/slave/ios-device-xcode-clang/build/src/out/Release-iphoneos/Chromium.app/Assets.car
ninja: build stopped: subcommand failed.
step returned non-zero exit code: 1

The "attempting to unload a stale CoreSimulatorService job" suggests that machine is doing something else at the same time, and https://stackoverflow.com/questions/39972105/how-can-i-fix-this-warning-coresimulator-is-attempting-to-unload-a-stale-coresi comes up in a search. Do these machines have multiple versions of Xcode installed?

There's discussion above about whether this is a real flake and the flakiness detection is working, and presumably it is and it does. So, who would be able to investigate this issue?

Detected 18 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Detected 18 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

sdefresne: could this issue be related to what you're doing in  issue 738359 ?

Indirectly related yes.

The issue is that actool is talking to a system service provided by Xcode and it is really picky on which version of Xcode is used. If the version mismatch (which likely happen due to hermetic build), then the build will fail but the service will be restarted with the correct version so the next build pass.

I've filled https://bugs.chromium.org/p/chromium/issues/detail?id=739163 to try to find a fix for this (one possibility is to just kill the service before running the command to ensure it has the proper version).

Assigning this to sdefresne@ for work on  issue 739163  :)

From comment #3 in  issue 739163 :

Are there specific builders this is hitting, that are shared between mac and ios builds?  Can we stop sharing these builders for the time being as a short term fix, especially if Xcode 9 really fixes this.

In fact this started before my change to iOS build. I think we should stop sharing builder bots between iOS and macOS as we'll always have those kind of failures.

why does the mix of ios and mac matter? Don't they both normally use the same hermetic toolchain?

They do not, unfortunately.

They don't? Do they use different hermetic toolchains? Or is one still not hermetic?

The issue is that compile_xcasset.py is a bit too conservative in what it accept (this is because we want to treat warning as errors which actool does not allow us to do) and that actool sometimes randomly spew warning messages when some service installed by Xcode.

When building with hermetic build, each hermetic build comes with its own version of the service, and sometimes cause actool to spew those warning messages. The script interpret this as error and bails out (conservatively). If only one version of Xcode were to be used, the particular message would be much less likely.

In  issue 739163  I'm going to make the tool more lenient in what it accept hoping that we'll not missing any real errors.

#33: they use different hermetic toolchain but they are not really hermetic as Xcode like to spwan system services (daemons) and autostart them from command-line tools.

I see, thanks.

What are the odds that this is not only causing compile flakes but is also causing the compile time to be much worse? (See bug 739556).

Detected 35 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

This would just cause a compile failure.  Do the bots do anything funky like do a second build on this kind of failure?

Yes, when the build fails, they try to build without patch (to check whether the build failure was introduced by the patch). I think the CQ also automatically retry failed bots to test whether the failure was due to a flake. So this mean the build could be tried up to 4 times per patches.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b5820db5ef26d58bcdfe1ce6dcfb77130a8f879a

commit b5820db5ef26d58bcdfe1ce6dcfb77130a8f879a
Author: Sylvain Defresne <sdefresne@chromium.org>
Date: Thu Jul 06 20:24:51 2017

Make build/toolchain/mac/compile_xcassets.py less strict.

The script build/toolchain/mac/compile_xcassets.py tries to turn
actool warnings into errors and to discard unrelevant messages
but was too strict causing compilation failures when actool was
just printing informational messages.

Change the script to be less strict and only keep the warnings or
error matching the '.*\.xcassets:.* (error|warning): .*' pattern.

Add unit test that the new function to filter actool output works
as intended.

Bug:  739163 , 736068 
Change-Id: I7a7e2c0765159e50809516c067703b6d59d91a8c
Reviewed-on: https://chromium-review.googlesource.com/561008
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Justin Cohen <justincohen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#484728}
[modify] https://crrev.com/b5820db5ef26d58bcdfe1ce6dcfb77130a8f879a/build/toolchain/mac/compile_xcassets.py
[add] https://crrev.com/b5820db5ef26d58bcdfe1ce6dcfb77130a8f879a/build/toolchain/mac/compile_xcassets_unittests.py

Detected 7 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Lowering priority as I don't see any flakes on iOS since my CL landed.

Removing from the sheriff queue since this is being addressed already.

Detected 3 new flakes for test/step "compile (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRjb21waWxlICh3aXRoIHBhdGNoKQw. This message was posted automatically by the chromium-try-flakes app.

Unassigning since all recently occurring flakes are not on ios bots.

"compile (with patch)" appears to be no longer flaky on ios-device-xcode-clang, but the step still flakes on some other builders.