New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 843367 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Error when importing tests to the database

Project Member Reported by haddowk@chromium.org, May 15 2018

Issue description

ToT builds of moblab are crashing with

+ python2.7 /usr/local/autotest/utils/test_importer.py
15:33:23 INFO | Scanning /usr/local/autotest/server/tests
Traceback (most recent call last):
  File "/usr/local/autotest/utils/test_importer.py", line 615, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/autotest/utils/test_importer.py", line 564, in main
    options.add_experimental)
  File "/usr/local/autotest/utils/test_importer.py", line 76, in update_all
    autotest_dir=autotest_dir)
  File "/usr/local/autotest/utils/test_importer.py", line 234, in update_tests_in_db
    path=test.replace(autotest_dir, '').lstrip('/'))[0]
  File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 146, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 470, in get_or_create
    return self.get(**lookup), False
  File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 382, in get
    num = len(clone)
  File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 90, in __len__
    self._result_cache = list(self.iterator())
  File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", line 301, in iterator
    for row in compiler.results_iter():
  File "/usr/lib64/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib64/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py", line 128, in execute
    six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
  File "/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py", line 120, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
django.db.utils.DatabaseError: (1054, "Unknown column 'afe_autotests.test_retry' in 'field list'")

 
Probably the root cause is

Migration starting for database chromeos_autotest_db
Applying migration 058_drone_management up
Traceback (most recent call last):
  File "/usr/local/autotest/database/migrate.py", line 483, in <module>
    main()
  File "/usr/local/autotest/database/migrate.py", line 444, in main
    manager.do_sync_db(version)
  File "/usr/local/autotest/database/migrate.py", line 356, in do_sync_db
    self.migrate_to_version_or_latest(version)
  File "/usr/local/autotest/database/migrate.py", line 344, in migrate_to_version_or_latest
    self.migrate_to_latest()
  File "/usr/local/autotest/database/migrate.py", line 305, in migrate_to_latest
    self.migrate_to_version(latest_version)
  File "/usr/local/autotest/database/migrate.py", line 260, in migrate_to_version
    self.do_migration(migration, migrate_up)
  File "/usr/local/autotest/database/migrate.py", line 228, in do_migration
    migration.migrate_up(self)
  File "/usr/local/autotest/database/migrate.py", line 72, in migrate_up
    self._execute_migration(self._UP_ATTRIBUTES, manager)
  File "/usr/local/autotest/database/migrate.py", line 63, in _execute_migration
    manager.execute_script(sql)
  File "/usr/local/autotest/database/migrate.py", line 141, in execute_script
    self.execute(statement)
  File "/usr/local/autotest/database/migrate.py", line 128, in execute
    return self._database.execute(query, parameters)
  File "/usr/local/autotest/database/database_connection.py", line 304, in execute
    results = self._backend.execute(query, parameters)
  File "/usr/local/autotest/database/database_connection.py", line 54, in execute
    self._cursor.execute(query, parameters)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1050, "Table 'afe_drones' already exists")

Updating the migration version to 58 bypassed the issue but I am not sure of the root cause

 SELECT * FROM migrate_info, ()
    -> ;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1
MariaDB [chromeos_autotest_db]> SELECT * FROM migrate_info;   
+---------+
| version |
+---------+
|      57 |
+---------+
1 row in set (0.00 sec)

MariaDB [chromeos_autotest_db]> update migrate_info set version=58;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [chromeos_autotest_db]> SELECT * FROM migrate_info;
+---------+
| version |
+---------+
|      58 |
+---------+


I vaguely remember pprabhu bringing this up during our weekly team meeting.  Our migrations aren't actually consistent end to end.  

Can you post the affected builds?
Owner: ayatane@chromium.org
Status: Assigned (was: Untriaged)
Re #3: What I brought up in our meeting was only relevant for our TKO database instance. We dropped some tables manually in that instance of the DB at some point, so we can't apply migrations that modify those tables.

Shouldn't affect a brand new setup (as on moblab).

To me this just looks like too early a version to be migrated to. iirc, there is a db-dump at some point in the migration and we only apply migrations from that point onwards. I'm not sure what migration the dump is from, but 58 seems too old. 
I am cleaning my build out and re-syncing everything - I think somehow I got a bad build but not sure how getting a very old build would be possible.
Owner: haddowk@chromium.org
Status: WontFix (was: Assigned)
I could not reproduce this - putting down to somehow getting a bad build.  Sorry for the noise.

Sign in to add a comment