"update_scripts" is flaky |
|||||||||
Issue description"update_scripts" is flaky. This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is not infrastructure-related (e.g. flaky test), please add Sheriff-Chromium label and change issue status to Untriaged. When done, please remove the issue from Trooper Bug Queue by removing the Infra-Troopers label. We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs This flaky test/step was previously tracked in issue 606486 .
,
May 2 2016
,
May 2 2016
Seems to be that these small pool of vm's have become a builder dumping ground. There's too many assigned. I've deleted the win & win.chrome build directories on vm{181,185}-m4 which are the current full slaves so as to provide temporary relief.
Marking this back to untriaged.
win_chrome_official
win_chromium_compile_rel_ng
win_chromium_dbg_ng
win_chromium_syzyasan_rel
win_chromium_variable
win_chromium_variable_archive
win_chromium_variable_chrome
win_chromium_variable_gn
win_chromium_variable_webkit_builder
win_chromium_variable_webkit_layout
win_chromium_variable_webkit_win7_builder
win_chromium_variable_webkit_win7_builder_x64
win_nacl_sdk
win_nacl_sdk_build
win_optional_gpu_tests_rel
,
May 2 2016
vm187-m4 and vm192-m4 will be in the same situation soon if nothing is done. There are (at least) two pools in this situation: master.tryserver.chromium.mac:ios_cq_dbg master.tryserver.chromium.win:win_optional_chromium An idea to prevent this would be to add a presubmit check that would reject pools with more than, say, five builders.
,
May 3 2016
,
May 3 2016
Issue 608554 has been merged into this issue.
,
May 3 2016
You didn't hear this from me but I can increase the size of the disk on these to provide some breathing room until a permanent fix to this builder pool problem is fixed.
,
May 3 2016
The problem will come back in no time, and on other pools as well. We have to fix the issue for real. Filed http://crbug.com/608565 to track that.
,
May 3 2016
pschmidt: if you have time to upgrade disks, feel free to chip away at http://crbug.com/605604
,
May 3 2016
Upgraded disks on vm{181,185..187,192}-m4. This should provide some head room until 608565 is addressed.
,
May 3 2016
Issue 608855 has been merged into this issue.
,
May 3 2016
I'm seeing several bots running out of space for similar reasons; vm673-m4, vm900-m4, vm898-m4, vm1026-m4, vm676-m4, vm675-m4, build86-a4, build87-a4, vm899-m4
,
May 3 2016
smut, can you fix the pool for tryserver.chromium.mac? Are you the best person to do that?
,
May 3 2016
Once the blocking bug is resolved, anyone will be able to rm -rf /b/build/slave/ios_* to free up disk space on the iOS try bot VMs.
,
May 3 2016
I take it we can silence the alerts on vm1026-m4 in the meantime?
,
May 9 2016
Assigning to pgervais@ since he seems to have the most insight here. Please re-add troopers if there is anything trooper can do to help.
,
May 25 2016
Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).
,
Jun 2 2016
Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 6 2016
,
Jun 9 2016
Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Trooper Bug Queue (unless already there).
,
Jun 9 2016
Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 9 2016
Detected 3 new flakes for test/step "update_scripts". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyGQsSBUZsYWtlIg51cGRhdGVfc2NyaXB0cww. This message was posted automatically by the chromium-try-flakes app.
,
Jun 14 2016
All recent flakes don't seem to be disk space issues. It doesn't seem to prefer one OS over another and they all fail due to no-output timeouts. Not sure what would cause update_scripts to hang (some git lock file hanging around?) but it happens so rarely that I really don't think it's worth spending time investigating. |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 by smut@chromium.org
, May 2 2016Components: -Tests>Flaky Infra>Labs
Labels: OS-Windows