New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 891757 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
OOO
Closed: Oct 3
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug



Sign in to add a comment

cros-full-0003 heartbeat is 0

Project Member Reported by gu...@chromium.org, Oct 3

Issue description

from shard-client.log
09:29:32 ERROR| Heartbeat failed. JSONRPCException: IntegrityError: (1062, "Duplicate entry '244525630-6482' for key 'host_queue_ent
ries_job_id_and_host_id'")
Traceback (most recent call last):
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 109, in dispatchRequest
    results['result'] = self.invokeServiceEndpoint(meth, args)
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 147, in invokeServiceEndpoint
    return meth(*args)
  File "/usr/local/autotest/frontend/afe/rpc_handler.py", line 270, in new_fn
    return f(*args, **keyword_args)
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 2084, in shard_heartbeat
    rpc_utils.persist_records_sent_from_shard(shard_obj, jobs, hqes)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 1032, in persist_records_sent_from_shard
    job_ids_sent=job_ids_persisted)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 1003, in _persist_records_with_type_sent_from_shard
    current_record.update_from_serialized(serialized_record)
  File "/usr/local/autotest/site-packages/django/db/transaction.py", line 223, in inner
    return func(*args, **kwargs)
  File "/usr/local/autotest/frontend/afe/model_logic.py", line 1157, in update_from_serialized
    self._deserialize_local(local)
  File "/usr/local/autotest/frontend/afe/model_logic.py", line 1050, in _deserialize_local
    super(type(self), self).save()
  File "/usr/local/autotest/site-packages/django/db/models/base.py", line 546, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/autotest/site-packages/django/db/models/base.py", line 626, in save_base
    rows = manager.using(using).filter(pk=pk_val)._update(values)
  File "/usr/local/autotest/site-packages/django/db/models/query.py", line 591, in _update
    return query.get_compiler(self.db).execute_sql(None)
  File "/usr/local/autotest/site-packages/django/db/models/sql/compiler.py", line 1014, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/local/autotest/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/autotest/site-packages/django/db/backends/mysql/base.py", line 122, in execute
    six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])
  File "/usr/local/autotest/site-packages/django/db/backends/mysql/base.py", line 120, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/autotest/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/autotest/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (1062, "Duplicate entry '244525630-6482' for key 'host_queue_entries_job_id_and_host_id'")

This is very similar to issue 880991, but I checked the db of master and shard, they were match. I also restarted the shard client, but didn't help.
 
One fact is, since yesterday, cros-full-0003 has low heart beat rate. In order to resolve that, I moved board relm from it to 0041. This is also match with issue 880991.
I followed the hack in crbug.com/880991#c10 and c11, seems shard client worked again.
Status: Fixed (was: Untriaged)

Sign in to add a comment