New issue
Advanced search Search tips

Issue 852564 link

Starred by 1 user
Status: Fixed
Owner:
pprabhu@chromium.org
Closed: Jun 2018
Cc:
ayatane@chromium.org
pprabhu@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: ----



Sign in to add a comment

Host label detection hits AFE instead of updating HostInfo

Project Member Reported by xixuan@chromium.org, Jun 13 2018 
https://chrome-swarming.appspot.com/task?id=3e13c11ee4978310&refresh=10&show_raw=1

In /usr/local/autotest/results/swarming-3e13c11ee4978311/chromeos4-row7-rack7-host7/provision_chromeos4-row7-rack7-host7/debug/autoserv.DEBUG of pprabhu-skylab-drone-3.hot:

/13 14:02:57.281 DEBUG|             utils:0283| [stdout] Detected label: hw_video_acc_h264
06/13 14:02:57.282 DEBUG|             utils:0283| [stdout] Detected label: hw_video_acc_vp8
06/13 14:02:57.282 DEBUG|             utils:0283| [stdout] Detected label: hw_video_acc_enc_h264
06/13 14:02:57.282 DEBUG|             utils:0283| [stdout] Detected label: hw_video_acc_enc_vp8
06/13 14:02:57.284 INFO |        base_label:0256| adding labels: ['internal_display', 'webcam', u'touchscreen', u'sku:blaze_cpu_nyan_4GB', u'phase:MP', 'servo', 'ec:cros', u'variant:blaze', 'cts_abi_arm', 'power:battery', 'hw_video_acc_vp8', 'os:cros', 'audio_loopback_dongle', u'touchpad', 'board:nyan_blaze', 'storage:mmc', 'hw_video_acc_enc_h264', 'model:nyan_blaze', 'hw_video_acc_h264', 'hw_video_acc_enc_vp8', 'bluetooth']
06/13 14:02:57.492 DEBUG|        retry_util:0202| ending retries with error: <class 'autotest_lib.frontend.afe.json_rpc.proxy.JSONRPCException'>(DoesNotExist: The following hosts do not exist: chromeos4-row7-rack7-host7
Traceback (most recent call last):
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 109, in dispatchRequest
    results['result'] = self.invokeServiceEndpoint(meth, args)
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 147, in invokeServiceEndpoint
    return meth(*args)
  File "/usr/local/autotest/frontend/afe/rpc_handler.py", line 270, in new_fn
    return f(*args, **keyword_args)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 1172, in replacement
    return func(**kwargs)
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 498, in host_add_labels
    _create_label_everywhere(label, [id])
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 234, in _create_label_everywhere
    host_objs = models.Host.smart_get_bulk(hosts)
  File "/usr/local/autotest/frontend/afe/model_logic.py", line 855, in smart_get_bulk
    ', '.join(invalid_inputs)))
DoesNotExist: The following hosts do not exist: chromeos4-row7-rack7-host7
)
06/13 14:02:57.492 ERROR|           control:0064| Exception while updating labels.


Is it bad?

Comment 1 by pprabhu@chromium.org, Jun 14 2018

Labels: Hotlist-Skylab
Owner: pprabhu@chromium.org
Status: Assigned (was: Untriaged)
This is another instance of a test going to AFE directly instead of updating the host info.

Comment 2 by pprabhu@chromium.org, Jun 14 2018 

The client side exception is more useful for debugging:

Traceback (most recent call last):
  File "/usr/local/autotest/results/swarming-3e13c11ee4978311/chromeos4-row7-rack7-host7/provision_chromeos4-row7-rack7-host7/control.srv", line 62, in provision_machine
    host.update_labels()
  File "/usr/local/autotest/server/hosts/cros_host.py", line 2137, in update_labels
    self.labels.update_labels(self)
  File "/usr/local/autotest/server/hosts/base_label.py", line 257, in update_labels
    afe.run('host_add_labels', id=host.hostname, labels=labels_to_add)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 131, in run
    self, call, **dargs)
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 244, in GenericRetry
    return _run()
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 177, in _Wrapper
    ret = func(*args, **kwargs)
  File "/usr/local/autotest/site-packages/chromite/lib/retry_util.py", line 243, in _run
    return functor(*args, **kwargs)
  File "/usr/local/autotest/server/cros/dynamic_suite/frontend_wrappers.py", line 94, in _run
    return super(RetryingAFE, self).run(call, **dargs)
  File "/usr/local/autotest/server/frontend.py", line 108, in run
    result = utils.strip_unicode(rpc_call(**dargs))
  File "/usr/local/autotest/frontend/afe/json_rpc/proxy.py", line 143, in __call__
    raise BuildException(resp['error'])
JSONRPCException: DoesNotExist: The following hosts do not exist: chromeos4-row7-rack7-host7

Comment 4 by pprabhu@chromium.org, Jun 14 2018

Summary: Host label detection hits AFE instead of updating HostInfo (was: Provision succeeds (based on status.log), but there're traceback of errors in debug log.)
Project Member

Comment 5 by bugdroid1@chromium.org, Jun 16 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c0377d5c7e1bf03c45c781418c021d889d5518e7

commit c0377d5c7e1bf03c45c781418c021d889d5518e7
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Sat Jun 16 05:16:33 2018

[autoest] Use HostInfoStore to update detected host labels.

BUG= chromium:852564 
TEST=Manually check that label detection updates only the known labels
     for a host in AFE and a host in skylab.

Change-Id: I2c8dfb77ae9c889df2e0188c1531a6b1b8c73775
Reviewed-on: https://chromium-review.googlesource.com/1101646
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/c0377d5c7e1bf03c45c781418c021d889d5518e7/server/hosts/base_label.py
[modify] https://crrev.com/c0377d5c7e1bf03c45c781418c021d889d5518e7/server/hosts/base_label_unittest.py

Comment 6 by pprabhu@chromium.org, Jun 16 2018

Status: Fixed (was: Started)
►

Sign in to add a comment