New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 817468 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug

Blocked on:
issue 819284

Blocking:
issue 805475



Sign in to add a comment

Mac10.9 Tests (dbg) are failing due to insufficient capacity

Project Member Reported by chengx@chromium.org, Feb 28 2018

Issue description

Many tests are failing with error message "There wasn't enough capacity to run your test some shards did not complete: 0"

 
Blockedon: -805475
Owner: shenghua...@chromium.org
Reassigning to @shenghuazhang to rename the builders to 10.13 and make sure all the tests are no longer running on 10.9, per comment https://bugs.chromium.org/p/chromium/issues/detail?id=805475#c77 .
Blocking: 805475
Issue 817141 has been merged into this issue.
Issue 817229 has been merged into this issue.
Issue 817360 has been merged into this issue.
Issue 817401 has been merged into this issue.
Issue 817741 has been merged into this issue.
Issue 817922 has been merged into this issue.
A tester on chromium.memory also targets mac10.9 swarming bots:
https://ci.chromium.org/buildbot/chromium.memory/Mac%20ASan%2064%20Tests%20%281%29/

Issue 818016 has been merged into this issue.
Status: Started (was: Assigned)
Working on the CL to upgrade os version to 10.13 in chromium.mac: https://chromium-review.googlesource.com/c/chromium/tools/build/+/942302

Project Member

Comment 7 by bugdroid1@chromium.org, Mar 2 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/589eaef11cc56076d6fbb16b34519267d78cd31d

commit 589eaef11cc56076d6fbb16b34519267d78cd31d
Author: Shenghua Zhang <shenghuazhang@chromium.org>
Date: Fri Mar 02 02:51:51 2018

[Mac os] Migrate 10.9 os to 10.13 in chromium.mac

Swarming mac 10.9 bots were upgraded to 10.13. This cl changes the
chromium.mac 10.9 tests builders os config to 10.13.

Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: infra
Bug:805475,  817468 
Change-Id: Ia7b50e12ff294b1dfd2d4373f4f4f99b9345e952
Reviewed-on: https://chromium-review.googlesource.com/942302
Commit-Queue: Shenghua Zhang <shenghuazhang@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Sergey Berezin <sergeyberezin@chromium.org>

[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/chromium.expected/dynamic_swarmed_gtest_mac_gpu.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac_GC_Stress.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipe_modules/chromium_tests/chromium_mac.py
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/remove_culprits_for_flaky_failures.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_mac64_rel.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipe_modules/swarming/api.py
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/findit_tests_multiple_culprits.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_mac64_dbg.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/flake.expected/use_build_parameter_for_tests.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/use_build_parameter_for_tests_non_json_buildbucket.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/chromium.expected/dynamic_swarmed_isolated_script_test_mac_gpu.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/findit_consecutive_culprits.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/findit_culprit_in_last_sub_range.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipe_modules/swarming/examples/full.expected/basic_luci.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipe_modules/swarming/examples/full.expected/basic.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/swarming_tests.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/findit_steps_multiple_culprits.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac___debug.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac64___debug.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipe_modules/swarming/examples/full.expected/service_account.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_mac64_asan_rel.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_mac_dbg.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/use_abbreviated_revision_in_step_name.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_tryserver_v8_v8_mac_gc_stress_dbg.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/findit_culprit_in_first_sub_range.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac64.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/use_build_parameter_for_tests.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipe_modules/swarming/examples/full.expected/named_caches.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac64_ASAN.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/chromium.py
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/flake.expected/flakiness_swarming_tests.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/use_analyze_set_to_False_for_non_linear_try_job.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac64_with_test_spec.json
[modify] https://crrev.com/589eaef11cc56076d6fbb16b34519267d78cd31d/scripts/slave/recipes/findit/chromium/test.expected/findit_culprit_in_middle_sub_range.json

Issue 818038 has been merged into this issue.
Both 'Mac10.9 Tests (dbg)' and 'Mac ASan 64 Tests (1)' run on mac 10.13 swarming bots now, not purple after the #7 cl landed:
https://ci.chromium.org/buildbot/chromium.mac/Mac10.9%20Tests%20%28dbg%29/
https://ci.chromium.org/buildbot/chromium.memory/Mac%20ASan%2064%20Tests%20%281%29/

Next step would be renaming the builders to 10.13.
Project Member

Comment 10 by bugdroid1@chromium.org, Mar 3 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/ec63b7bf2186c1c3117bc4cbd5ebdbf37f8b1d27

commit ec63b7bf2186c1c3117bc4cbd5ebdbf37f8b1d27
Author: Shenghua Zhang <shenghuazhang@google.com>
Date: Sat Mar 03 02:23:22 2018

Components: Infra>Client>Chrome
OK, after renaming 10.9 builders and changing configs, the new upgraded 10.13 bots are green and run tests on 10.13 testers successfully during the weekend:
https://ci.chromium.org/p/chromium/g/chromium.mac/console
https://ci.chromium.org/buildbot/chromium.memory/Mac%20ASan%2064%20Tests%20%281%29/
Cc: dpranke@chromium.org st...@chromium.org
From swarming tasks list http://shortn/_7stAJjfDW2, there are still a few tasks running on Mac 10.9 chrome pool:
1) 'Check/Mac-10.9/xxx/V8 Mac64 ASAN'. Seems like V8 builder 'V8 Mac64 ASAN' is targeting to swarming os 'Mac-10.9'. I am going to upgrade it.
2) 'findit/ref_task_id/xxx'. Not sure which findit builder it belongs to. And it only happened on March/05 for 4 tasks. 

@stgao hi Shuotao, can you confirm where those 'findit' tasks running on Mac10.9 came from? Is it safe for us to upgrade all Mac 10.9 machines without breaking them?
Cc: lijeffrey@chromium.org wylieb@chromium.org
Those 'findit/ref_task_id/xxx' tasks are triggered directly by Findit because it looks back in the older builds which were still on Mac 10.9, e.g. before the upgrade of "Mac ASan 64 Tests (1)". Swarming task 3c1429de19d04310 referred to 3bfd5ef5d3312010 and copied the dimensions.
It would be great if we have 2 Swarming bots still on Mac 10.9.

lijeffrey@ is finishing up a new analysis pipeline, and we will switch to sticking to the dimensions of the latest build instead.
Change on Findit side is tracked by  bug 819284 
Blockedon: 819284
Thanks for the confirmation and filing the pipeline bug. Currently there are 20 Mac10.9 swarming bots, and it would probably be a safer capacity to leave 5 bots for findit tasks. Marking the findit bug as a blocker for the rest 10.9 bots.

Here is the cl to upgrade V8 builders os version https://chromium-review.googlesource.com/c/chromium/tools/build/+/949606, which needs @machenbach or @sergiyb LGTM. I would request labs to help upgrade another 15 bots after it landed.
Project Member

Comment 16 by bugdroid1@chromium.org, Mar 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/7ea35f7c90c609acfd6db5ed5a51d7ccb7698ad5

commit 7ea35f7c90c609acfd6db5ed5a51d7ccb7698ad5
Author: Shenghua Zhang <shenghuazhang@chromium.org>
Date: Wed Mar 07 09:53:44 2018

[Mac os] Upgrade mac os to 10.13 for v8 builder 'V8 Mac64 ASAN'

Mac os 10.9 machines would be gone and upgraded to 10.13. This CL
changes the builder 'V8 Mac64 ASAN' swarming os dimension to '10.13'
from '10.9'.

Bug:  817468 
Change-Id: I8d6da3e03e276684ef41ad6941a5bb558648e20f
Reviewed-on: https://chromium-review.googlesource.com/949606
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Michael Achenbach <machenbach@chromium.org>

[modify] https://crrev.com/7ea35f7c90c609acfd6db5ed5a51d7ccb7698ad5/scripts/slave/recipe_modules/v8/builders.py
[modify] https://crrev.com/7ea35f7c90c609acfd6db5ed5a51d7ccb7698ad5/scripts/slave/recipes/v8.expected/full_client_v8_V8_Mac64_ASAN.json

After #16 CL landed, there is no more 'V8 Mac64 ASAN' task running on Mac os 10.9 bots from Mar 7: http://shortn/_cKvaZqGLly
Status: Fixed (was: Started)
This bug as description is fixed. Close the bug.

Sign in to add a comment