New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 793070 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 792985
Owner: ----
Closed: Dec 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

DB error after provisioning (Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT)

Project Member Reported by ihf@chromium.org, Dec 7 2017

Issue description

Starting with R65-10193.0.0 I see a lot of DB failures affecting CTS provisionings as below. The problem happens immediately after provisioning.

https://stainless.corp.google.com/search?col=build&exclude_non_release=true&exclude_cts=false&suite=%5Earc%5C-cts%24&row=test&view=matrix&first_date=20171201&last_date=20171207
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/160964472-chromeos-test/chromeos6-row2-rack24-host18/debug/

2/07 10:05:08.025 INFO |        dev_server:1496| Name: /home/chromeos-test/images/kevin-release/R65-10194.0.0/update.gz Accessed: 2017-12-07 05:54:16.631393 Modified: 2017-12-07 05:53:26.394425 Size: 969226083 bytes.
12/07 10:05:08.026 DEBUG|provision_AutoUpda:0115| Installing image
12/07 10:05:08.027 DEBUG|         host_info:0229| Committing HostInfo to store ShadowingStore[AfeStore[chromeos6-row2-rack24-host18], FileStore[/usr/local/autotest/results/hosts/chromeos6-row2-rack24-host18/2715537-provision/20170712100348/host_info_store/store_acf08fc4-6e2c-4a1b-a90d-4ae1565736a1]]
12/07 10:05:08.028 DEBUG|         host_info:0229| Committing HostInfo to store AfeStore[chromeos6-row2-rack24-host18]
12/07 10:05:08.192 DEBUG|         afe_store:0083| removing labels: ['cros-version:kevin-release/R64-10176.7.0']
12/07 10:05:08.393 DEBUG|        retry_util:0202| ending retries with error: <class 'autotest_lib.frontend.afe.json_rpc.proxy.JSONRPCException'>(JSONRPCException: DatabaseError: (1665, 'Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.')
Traceback (most recent call last):
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 109, in dispatchRequest
    results['result'] = self.invokeServiceEndpoint(meth, args)
  File "/usr/local/autotest/frontend/afe/json_rpc/serviceHandler.py", line 147, in invokeServiceEndpoint
    return meth(*args)
  File "/usr/local/autotest/frontend/afe/rpc_handler.py", line 270, in new_fn
    return f(*args, **keyword_args)
  File "/usr/local/autotest/frontend/afe/rpc_utils.py", line 1148, in replacement
    return func(**kwargs)
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 478, in host_remove_labels
    remove_labels_from_host(id, labels)
  File "/usr/local/autotest/frontend/afe/rpc_interface.py", line 468, in remove_labels_from_host
    models.Host.smart_get(id).labels.remove(*label_objs)
  File "/usr/local/autotest/site-packages/django/db/models/fields/related.py", line 658, in remove
    self._remove_items(self.source_field_name, self.target_field_name, *objs)
  File "/usr/local/autotest/site-packages/django/db/models/fields/related.py", line 775, in _remove_items
    '%s__in' % target_field_name: old_ids
  File "/usr/local/autotest/site-packages/django/db/models/query.py", line 537, in delete
    collector.delete()
  File "/usr/local/autotest/site-packages/django/db/models/deletion.py", line 62, in decorated
    func(self, *args, **kwargs)
  File "/usr/local/autotest/site-packages/django/db/models/deletion.py", line 293, in delete
    qs._raw_delete(using=self.using)
  File "/usr/local/autotest/site-packages/django/db/models/query.py", line 548, in _raw_delete
    sql.DeleteQuery(self.model).delete_qs(self, using)
  File "/usr/local/autotest/site-packages/django/db/models/sql/subqueries.py", line 84, in delete_qs
    self.get_compiler(using).execute_sql(None)
  File "/usr/local/autotest/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/autotest/site-packages/django/db/backends/mysql/base.py", line 128, in execute
    six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
  File "/usr/local/autotest/site-packages/django/db/backends/mysql/base.py", line 120, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/autotest/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/local/autotest/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
DatabaseError: (1665, 'Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is READ COMMITTED or READ UNCOMMITTED.')
 

Comment 1 by ihf@chromium.org, Dec 7 2017

Cc: gkihumba@chromium.org
Labels: -Pri-1 ReleaseBlock-Stable M-63 Pri-0
This happens on M-63 as well, so marking as release blocker.
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/160790846-chromeos-test/chromeos4-row12-rack11-host5/debug/

Comment 2 by x...@chromium.org, Dec 7 2017

Dup of 792985?

Comment 3 by ihf@chromium.org, Dec 7 2017

Mergedinto: 792985
Status: Duplicate (was: Untriaged)
I searched for a related bug but didn't find it. Yes it looks like a dupe. Glad it was fixed!

Sign in to add a comment