"remote_run_result" is flaky |
||||||
Issue description"remote_run_result" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label. We have detected 195 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFyZW1vdGVfcnVuX3Jlc3VsdAw. The chromium-try-flakes app is able to file bugs for individual tests when the test launcher is uploading results to the Test Results Server. If recent flakes above are caused by failing tests and you would like to have them filed as invidual bugs, please read more at https://goo.gl/QJKXV4. This flaky test/step was previously tracked in issue 723175 .
,
May 30 2017
A lot of bots are purple due to this.
,
May 30 2017
Most of them got green for the next run. Lowering to P1.
,
May 30 2017
Detected 7 new flakes for test/step "remote_run_result". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyHAsSBUZsYWtlIhFyZW1vdGVfcnVuX3Jlc3VsdAw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
,
May 30 2017
Issue 727556 has been merged into this issue.
,
May 30 2017
Issue 727555 has been merged into this issue.
,
May 30 2017
,
May 30 2017
re #3: Since 30 minutes ago they seem to be failing consistently. Most of my trybot build are failing with purple due to this. It seems to be affecting linux and android trybots. Here is a list according to my failing CL: android_arm64_dbg_recipe android_clang_dbg_recipe android_compile_dbg android_cronet android_n5x_swarming_rel cast_shell_android cast_shell_linux chromeos_amd64-generic_chromium_compile_only_ng chromeos_daisy_chromium_compile_only_ng chromium_presubmit linux_android_rel_ng linux_chromium_asan_rel_ng linux_chromium_chromeos_ozone_rel_ng linux_chromium_chromeos_rel_ng linux_chromium_compile_dbg_ng linux_chromium_headless_rel linux_chromium_rel_ng linux_chromium_tsan_rel_ng
,
May 31 2017
Looks like a lot of update_scripts exceptions like this: """ ________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress --verbose https://chromium.googlesource.com/external/boto.git /b/build/third_party/gsutil/_gclient_boto_m41FBJ' in '/b' [0:00:07] Cloning into '/b/build/third_party/gsutil/_gclient_boto_m41FBJ'... [0:00:07] POST git-upload-pack (197 bytes) [0:00:07] remote: Daily bandwidth rate limit exceeded for 182615506979@project.gserviceaccount.com """ Then a later exception in "remote_run_result" "IOError: [Errno 2] No such file or directory: '/b/rr/tmpN0xw3s/recipe_result.json'" Seems like the update_scripts part is the failing step, not remote_run. Probably same issue as yesterday's GoB quota-related outage: https://o.corp.google.com/#Outage:397695 Though these builders seem to be running okay now. Will keep an eye on it for a bit though. Quota apparently resets at 1pm MTV time.
,
Jun 1 2017
These were probably all due to the gob quota issue causing update_scripts to fail on git requests. We've reset the daily gob quota, requested a quota increase and are looking into ways to reduce our overall gob usage. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by guidou@chromium.org
, May 29 2017Labels: -Sheriff-Chromium