Or, create python script per feature.
- upload ninja log
- upload compiler_proxy info
- upload gomastats
- upload counter
...

Since goma_utils.py is also used by resource, I guess it could also be an option.

Since some code in goma_utils deprecated due to deprecation of builder specific things, I will work on this after that.
However, to avoid the issue like below again, we need to do this.
https://chromium-review.googlesource.com/c/chromium/tools/build/+/951845

 Issue 819471  has been merged into this issue.

Detected 4 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

I checked win-msvc-rel builder, and it seems to working fine now.  Maybe temporary breakage but not continuous breakage?

Detected 4 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

 Issue 821709  has been merged into this issue.

Detected 24 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Let me change the goal to #1.

Detected 10 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

Summary of #12.
a. missing google.cloud.bigquery: 7
b. missing goma_utils: 3

For a. I do not see tendency on builders: (but no mac?)
ChromeOS 2, Android 2, Win 1, Linux 2
ccompute: 6, golo: 1
step name: "compile (with patch)"

All of b. missing goma_utils are caused on win-msvc-rel builder.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

Oh flakes, this isn't worth troopers attention.

FYI, https://chromium-review.googlesource.com/c/chromium/tools/build/+/970042 to dump paths on failure from next time.

Detected 4 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

I love playing whack a mole with tryflakes...

Detected 5 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).

This keeps flaking. So assigning to yyanagisawa@ and raising pri.

Detected 15 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Hmm, win-msvc-rel builder seems not like upload_goma_logs.py to import other modules in slave X(

Detected 8 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 40 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 8 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 21 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Does anybody know how 'compiler_proxy_info' in ninja.log json used?

I thought chromium_build_stats uses it but I could only find:
https://chromium.googlesource.com/infra/infra/+/master/appengine/chromium_build_stats/gopath/src/ninjalog/ninjalog.go#107

and, the field seems not be used by anybody.
If nobody uses this, we can factor out UploadNinjaLogs from goma_util.py.

This does not need to be upload_goma_logs.py.
https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/chromium/api.py#485

it was planned to associate ninja log and compiler proxy log, e.g. merge detailed trace from compiler proxy log into ninja trace etc.

I heard: it's used by human (us?) to understand how this ninja_log is generated.

However, usually I download ninja_log from build page or https://chromium-build-stats.appspot.com/ninja_log/, so I have never used the information.

Thanks for the comment.
It might not be used now but I guess it sounds like we would like to have the way to associate ninja.log and corresponding compiler_proxy.INFO.

Then, I come up with two ideas:
1. put build ID to both files metadata.
2. put compiler_proxy's path in ninja.log json or metadata. factor out upload ninja log and path to compiler_proxy.INFO is given via argument.

For 1. we can provide ninja log upload as independent program for both with and without goma.
since ninja log has build ID, it might be easy for us to find out actual build from ninja.log.
I guess ninja.log and compiler_proxy.INFO are usually uploaded to the same directory, and files in the directory is not so much, this may work.

For 2. finding corresponding compiler_proxy.INFO should be easy.
unlike 1, upload ninja log code might be yet another python module. I feel it slightly far from what I wanted to do with this issue.

Detected 21 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 20 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 10 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 10 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 10 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 8 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "postprocess_for_goma". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHwsSBUZsYWtlIhRwb3N0cHJvY2Vzc19mb3JfZ29tYQw. This message was posted automatically by the chromium-try-flakes app.

All the recent flakes I saw had to do with cloudtail. Consistently they looked like https://ci.chromium.org/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel/42438, where the stop cloudtail call fails.

This looks like just a bug in the cloudtail stuff. There's a goma script to control this, so I don't know if there's anything a trooper could really do about this?

https://chromium.googlesource.com/chromium/tools/build/+log/ac0764ecc2deecd77d6eaa17bbdafe741151b90e/scripts/slave/recipe_modules/goma/resources/cloudtail_utils.py shows that ukai@ made the last changes to this, so assigning to them to look at this for now.

yyanagisawa.  why you unassigned from owner?

sample failure

https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/mac_chromium_rel_ng/98481
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8940553139763140304/+/steps/compile__with_patch_/0/stdout

network error continued for a long time
but marked as red in compile step

maybe, it should be different bug than "split features of upload_goma_logs.py or goma_utils.py and make python script self-contained." ?

anyway, no alert fired for "network error continued for a long time" ?
below threshold?

hmm, seems no falkiness reports since May 21.
closing.