New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 617292 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Jun 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

balance-pool broken

Project Member Reported by dgarr...@chromium.org, Jun 3 2016

Issue description

dgarrett$balance-pool --all-boards cq
ERROR:root:No label information was logged before 2016-06-03 14:21:05.
Traceback (most recent call last):
  File "/usr/local/bin/balance-pool", line 513, in <module>
    main(sys.argv)
  File "/usr/local/bin/balance-pool", line 504, in main
    labels=[_POOL_PREFIX + arguments.pool])
  File "/usr/local/google/home/dgarrett/sand/clean/src/third_party/autotest/files/site_utils/host_label_utils.py", line 45, in get_all_boards
    for labels in host_labels.values()]
AttributeError: 'NoneType' object has no attribute 'values'

 
Looks like there's an error getting any labels at all. From a tiny script that just does the following:

host_label_utils.get_host_labels(labels=['servo'])

I get the output:             
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.25.66.62
DEBUG:urllib3.connectionpool:"HEAD /cautotest HTTP/1.1" 200 0
INFO:elasticsearch:HEAD http://172.25.66.62:9800/cautotest [status:200 request:0.105s]
DEBUG:elasticsearch:> None
DEBUG:elasticsearch:< 
No handlers could be found for logger "elasticsearch.trace"
DEBUG:urllib3.util.retry:Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0)
DEBUG:urllib3.connectionpool:"GET /cautotest/_search?timeout=30 HTTP/1.1" 200 123
INFO:elasticsearch:GET http://172.25.66.62:9800/cautotest/_search?timeout=30 [status:200 request:0.069s]
DEBUG:elasticsearch:> {"sort": [{"time_index": "desc"}], "query": {"bool": {"must": [{"term": {"_type": "host_labels_time_index"}}, {"range": {"time_index": {"lte": 1464988416.230998}}}]}}, "size": 1, "fields": ["time_index"]}
DEBUG:elasticsearch:< {"took":31,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
ERROR:root:No label information was logged before 2016-06-03 14:13:36.
None

Cc: jrbarnette@chromium.org
Talked with Richard and decided trying to debug ES problems wasn't worth it.  The right move would to remove the call to get all boards using ES to using lab_inventory to get all boards and guess what! My cl has that change so I'll be committing soon and this should be a non-issue then.
Project Member

Comment 4 by bugdroid1@chromium.org, Jun 4 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cf0ad2ba0e9df5ca6473f50829235b7cc94163d5

commit cf0ad2ba0e9df5ca6473f50829235b7cc94163d5
Author: Kevin Cheng <kevcheng@chromium.org>
Date: Tue Apr 19 21:51:39 2016

[autotest] Update run-pool-inventory to balance_pool.

Balance out the pool before we send out the notification eMail to the
deputy.

BUG= chromium:605747 , chromium:607200 , chromium:617292 
TEST=locally with dry-run/debug options and unittest

Change-Id: I47cd45fe495e1d46a1a49e97922868de74c5fd02
Reviewed-on: https://chromium-review.googlesource.com/339674
Commit-Ready: Kevin Cheng <kevcheng@chromium.org>
Tested-by: Kevin Cheng <kevcheng@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/cf0ad2ba0e9df5ca6473f50829235b7cc94163d5/site_utils/balance_pools.py
[modify] https://crrev.com/cf0ad2ba0e9df5ca6473f50829235b7cc94163d5/site_utils/lab_inventory_unittest.py
[modify] https://crrev.com/cf0ad2ba0e9df5ca6473f50829235b7cc94163d5/contrib/run-pool-inventory
[modify] https://crrev.com/cf0ad2ba0e9df5ca6473f50829235b7cc94163d5/site_utils/lab_inventory.py

Status: Fixed (was: Untriaged)
Closing... please feel free to reopen if its not fixed.
Status: Verified (was: Fixed)

Sign in to add a comment