New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 653217 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Failed: scripts/export_to_gcloud_unittest

Project Member Reported by rajatja@google.com, Oct 5 2016

Issue description

This caused the expresso-release canary to fail:

https://uberchromegw.corp.google.com/i/chromeos/builders/expresso-release/builds/620

chromite-0.0.2-r2607: 05:23:02: ERROR: ### LOG: scripts/export_to_gcloud_unittest
chromite-0.0.2-r2607: Traceback (most recent call last):
chromite-0.0.2-r2607:   File "scripts/export_to_gcloud_unittest", line 35, in <module>
chromite-0.0.2-r2607:     wrapper.DoMain()
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/scripts/wrapper.py", line 164, in DoMain
chromite-0.0.2-r2607:     commandline.ScriptWrapperMain(FindTarget)
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/lib/commandline.py", line 814, in ScriptWrapperMain
chromite-0.0.2-r2607:     target = find_target_func(target)
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/scripts/wrapper.py", line 139, in FindTarget
chromite-0.0.2-r2607:     module = cros_import.ImportModule(target)
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/lib/cros_import.py", line 43, in ImportModule
chromite-0.0.2-r2607:     module = __import__(target)
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/scripts/export_to_gcloud_unittest.py", line 12, in <module>
chromite-0.0.2-r2607:     from chromite.lib import cros_test_lib
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/lib/cros_test_lib.py", line 32, in <module>
chromite-0.0.2-r2607:     from chromite.lib import cidb
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/lib/cidb.py", line 28, in <module>
chromite-0.0.2-r2607:     import sqlalchemy
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/venv/venv/lib/python2.7/site-packages/sqlalchemy/__init__.py", line 9, in <module>
chromite-0.0.2-r2607:     from .sql import (
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/venv/venv/lib/python2.7/site-packages/sqlalchemy/sql/__init__.py", line 8, in <module>
chromite-0.0.2-r2607:     from .expression import (
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/venv/venv/lib/python2.7/site-packages/sqlalchemy/sql/expression.py", line 31, in <module>
chromite-0.0.2-r2607:     from .functions import func, modifier, FunctionElement, Function
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/venv/venv/lib/python2.7/site-packages/sqlalchemy/sql/functions.py", line 11, in <module>
chromite-0.0.2-r2607:     from . import sqltypes, schema
chromite-0.0.2-r2607:   File "/mnt/host/source/chromite/venv/venv/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py", line 16, in <module>
chromite-0.0.2-r2607:     from .elements import quoted_name, type_coerce, _defer_name
chromite-0.0.2-r2607: UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 355: invalid start byte
 
Very strange indeed. Let's keep an eye on this in case it happens again.
Another possibly related flake symptom.

https://uberchromegw.corp.google.com/i/chromeos/builders/peach_pit-paladin/builds/13753/steps/UnitTest/logs/stdio

chromite-0.0.2-r2611: 19:04:35: ERROR: ### LOG: scripts/virtualenv_wrapper_unittest
chromite-0.0.2-r2611: scripts/virtualenv_wrapper_unittest: could not import chromite module: /mnt/host/source/chromite/scripts/virtualenv_wrapper_unittest.py: No module named six
chromite-0.0.2-r2611: Traceback (most recent call last):
chromite-0.0.2-r2611:   File "scripts/virtualenv_wrapper_unittest", line 35, in <module>
chromite-0.0.2-r2611:     wrapper.DoMain()
chromite-0.0.2-r2611:   File "/mnt/host/source/chromite/scripts/wrapper.py", line 164, in DoMain
chromite-0.0.2-r2611:     commandline.ScriptWrapperMain(FindTarget)
chromite-0.0.2-r2611:   File "/mnt/host/source/chromite/lib/commandline.py", line 814, in ScriptWrapperMain
chromite-0.0.2-r2611:     target = find_target_func(target)
chromite-0.0.2-r2611:   File "/mnt/host/source/chromite/scripts/wrapper.py", line 139, in FindTarget
chromite-0.0.2-r2611:     module = cros_import.ImportModule(target)
chromite-0.0.2-r2611:   File "/mnt/host/source/chromite/lib/cros_import.py", line 43, in ImportModule
chromite-0.0.2-r2611:     module = __import__(target)
chromite-0.0.2-r2611:   File "/mnt/host/source/chromite/scripts/virtualenv_wrapper_unittest.py", line 10, in <module>
chromite-0.0.2-r2611:     import six
chromite-0.0.2-r2611: ImportError: No module named six


My theory is there is some sort of race or contention over the setup of the virtualenv. Probably we need to add some kind of lock around the create_env.sh script
Project Member

Comment 4 by sheriffbot@chromium.org, Oct 7 2016

Labels: Hotlist-Google
Project Member

Comment 5 by bugdroid1@chromium.org, Oct 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/df44f8ee57d8aee89199bb6ad91bd36aaac50d0d

commit df44f8ee57d8aee89199bb6ad91bd36aaac50d0d
Author: Aviv Keshet <akeshet@chromium.org>
Date: Fri Oct 07 03:18:53 2016

virtualenv: acquire an exclusive lock when creating virtualenv

BUG= chromium:653217 
TEST=tested locally with varying timeouts, with multiple parallel calls
to create_env.sh

Change-Id: Ic19ce9269514650343e2e88fb9f446ee249dd585
Reviewed-on: https://chromium-review.googlesource.com/395108
Commit-Ready: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/df44f8ee57d8aee89199bb6ad91bd36aaac50d0d/venv/.gitignore
[modify] https://crrev.com/df44f8ee57d8aee89199bb6ad91bd36aaac50d0d/venv/create_env.sh

Status: Fixed (was: Untriaged)

Sign in to add a comment