New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 878527 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 21
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

'Not enough available machines exist in in swarming' is confusing

Project Member Reported by crouleau@chromium.org, Aug 28

Issue description

See for example this log: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus6_WebView_Perf%2F2484%2F%2B%2Frecipes%2Fsteps%2Ftest_pre_run%2F0%2Fsteps%2Fs__trigger__performance_webview_test_suite_on_Android_device_Nexus_6%2F0%2Fstdout

Healthy bots: ['build203-b7--device1', 'build203-b7--device3', 'build203-b7--device2', 'build203-b7--device5', 'build203-b7--device4', 'build203-b7--device7', 'build203-b7--device6', 'build202-b7--device4', 'build202-b7--device5', 'build202-b7--device6', 'build202-b7--device7', 'build202-b7--device1', 'build202-b7--device2', 'build202-b7--device3']
Dead Bots: ['build204-b7--device1']
Traceback (most recent call last):
  File "/b/c/b/Android_Nexus6_WebView_Perf/src/testing/trigger_scripts/perf_device_trigger.py", line 301, in <module>
    sys.exit(main())
  File "/b/c/b/Android_Nexus6_WebView_Perf/src/testing/trigger_scripts/perf_device_trigger.py", line 298, in main
    return triggerer.trigger_tasks(args, remaining)
  File "/b/c/b/Android_Nexus6_WebView_Perf/src/testing/trigger_scripts/base_test_triggerer.py", line 246, in trigger_tasks
    selected_configs = self.select_config_indices(args, verbose)
  File "/b/c/b/Android_Nexus6_WebView_Perf/src/testing/trigger_scripts/perf_device_trigger.py", line 122, in select_config_indices
    return self._select_config_indices_with_soft_affinity(args, verbose)
  File "/b/c/b/Android_Nexus6_WebView_Perf/src/testing/trigger_scripts/perf_device_trigger.py", line 131, in _select_config_indices_with_soft_affinity
    raise ValueError('Not enough available machines exist in in swarming'
ValueError: Not enough available machines exist in in swarmingpool. Contact labs to rack in more hardware
step returned non-zero exit code: 1

When I read that log it doesn't make any sense because there is a long list of Healthy bots, but actually for whatever reason we cannot run against them. These logs need to explain a little more. Like if we can't run because of device affinity or a maintenance mutex, then we should print that out so that sheriffs can figure this out.

Relevant code: https://cs.chromium.org/chromium/src/testing/trigger_scripts/perf_device_trigger.py?q=trigger_scripts/perf_device_trigger.py&sq=package:chromium&dr

See potentially related issue 878332.


 
Cc: mar...@chromium.org kbr@chromium.org
Emily is on leave right now, but Marc-Antoine, Ned, and Ken reviewed some of the code for that.
Owner: nednguyen@chromium.org
Status: Assigned (was: Available)
Cc: jbudorick@chromium.org
Components: -Speed Speed>Benchmarks>Waterfall
Note that with task fallback, you could simplify the code a bit now.
Owner: bradhall@chromium.org
Reassign to Brad to triage/backlog
Project Member

Comment 6 by bugdroid1@chromium.org, Nov 21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e0735301cdf44187f35957a15c5b55fa809427cd

commit e0735301cdf44187f35957a15c5b55fa809427cd
Author: Brad Hall <bradhall@google.com>
Date: Wed Nov 21 19:37:26 2018

Make the "not enough bots" error message more explicit

Bug:  878527 
Change-Id: I2e527d39337232a657c34aa51c021d82b5f8a02d
Reviewed-on: https://chromium-review.googlesource.com/c/1347054
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Brad Hall <bradhall@google.com>
Cr-Commit-Position: refs/heads/master@{#610168}
[modify] https://crrev.com/e0735301cdf44187f35957a15c5b55fa809427cd/testing/trigger_scripts/perf_device_trigger.py

Status: Fixed (was: Assigned)

Sign in to add a comment