New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 723493 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner:
Last visit > 30 days ago
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocked on:
issue 708361

Blocking:
issue 723367



Sign in to add a comment

sysmon fails to launch due to venv dependency error

Project Member Reported by pprabhu@chromium.org, May 17 2017

Issue description

This is happening on chromeos-server88

See issue 723367: puppet hadn't been updated since Nov 7. As a result, ts_mon wasn't using the new endpoint. I forced a ToT puppet run. But sysmon failed to start with:

Traceback (most recent call last):
  File "/usr/local/google/home/chromeos-test/chromiumos/chromite/bin/sysmon", line 38, in <module>
    cros_build_lib.RunCommand([create_cmd])
  File "/usr/local/google/home/chromeos-test/chromiumos/chromite/lib/cros_build_lib.py", line 624, in RunCommand
    raise RunCommandError(msg, cmd_result)
chromite.lib.cros_build_lib.RunCommandError: return code: 1; command: /usr/local/google/home/chromeos-test/chromiumos/chromite/venv/create_env.sh
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 825, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 993, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 961, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/usr/lib/python2.7/dist-packages/virtualenv.py", line 903, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /usr/local/google/ho...venv/venv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip failed with error code 2


Creating or updating virtualenv in /usr/local/google/home/chromeos-test/chromiumos/chromite/venv/venv/
New python executable in venv/bin/python
Installing setuptools, pip...
  Complete output from command /usr/local/google/ho...venv/venv/bin/python -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip:
  Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking setuptools
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/index.py", line 252, in find_requirement
    for page in self._get_pages(locations, req):
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/index.py", line 387, in _get_pages
    page = self._get_page(location, req)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 349, in request
    prep = self.prepare_request(req)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/_vendor/requests/sessions.py", line 287, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/_vendor/requests/models.py", line 287, in prepare
    self.prepare_url(url, params)
  File "/usr/share/python-virtualenv/pip-1.5debian1-py2.py3-none-any.whl/pip/_vendor/requests/models.py", line 338, in prepare_url
    "Perhaps you meant http://{0}?".format(url))
MissingSchema: Invalid URL u'pip_packages': No schema supplied. Perhaps you meant http://pip_packages?

Storing debug log for failure in /usr/local/google/home/chromeos-test/.pip/pip.log
----------------------------------------
...Installing setuptools, pip...done.

cwd=None
 
Blockedon: 708361
'repo sync'ed /usr/local/chromeos-test/chromiumos repo checkout. It's a KI that this is not updated automatically in any way in the lab currently.

The error above looks like some new pip wheels were missing from our infra_virtualenv project.

.... and we're back online.

NOTICE:root:ts_mon was set up.
DEBUG:chromite.scripts.sysmon.system_metrics:autoserv_count: 0
DEBUG:chromite.scripts.sysmon.system_metrics:sysmon_count: 0
fatal: Not a git repository: '/root/chromeos-admin/.git'
WARNING:chromite.scripts.sysmon.git_metrics:Error collecting git metrics for /root/chromeos-admin/.git: Command '[u'git', u'--git-dir', u'/root/chromeos-admin/.git', u'rev-parse', u'HEAD']' returned non-zero exit status 128
DEBUG:chromite.scripts.sysmon.git_metrics:Collecting Git hash '77315c6962d75dc46034b9eb26e5be8f6699e560' for u'~chromeos-test/chromiumos/chromite/.git'
DEBUG:chromite.scripts.sysmon.git_metrics:Collecting Git timestamp 1494892549 for u'~chromeos-test/chromiumos/chromite/.git'
fatal: Not a git repository: '/usr/local/autotest/.git'
WARNING:chromite.scripts.sysmon.git_metrics:Error collecting git metrics for /usr/local/autotest/.git: Command '[u'git', u'--git-dir', u'/usr/local/autotest/.git', u'rev-parse', u'HEAD']' returned non-zero exit status 128
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
DEBUG:oauth2client.crypt:[ REDACTED ]
INFO:oauth2client.client:Refreshing access_token
DEBUG:chromite.scripts.sysmon.loop:Sleeping for 60, 60 remaining

For now, we can manually sync this repo across the fleet.
Blocking: 723367
Owner: chingcodes@chromium.org
Status: Assigned (was: Started)
I'm going to hand off to deputy to sync this repo across the fleet.
Relevant repos are supposed to be synced by prod push.  If chromeos-server88 wasn't getting synced, it's either not cataloged as prod or there's a bug in the prod push.

Comment 6 by aut...@google.com, May 23 2017

Labels: -current-issue
Cc: dshi@chromium.org
Owner: ----
Maybe @dshi can take a look at the puppet issues

Comment 8 by dshi@chromium.org, May 24 2017

Owner: ayatane@chromium.org
Allen, can you take a look why repo sync is not working as expected in that server?
chromeos-server88 is back (See comment #1).
This bug is open to find out if there are other servers with the same problem.
chromeos-server88 is a crash server.  It is updated by neither devserver push nor prod push.  The fact that it's not getting synced is natural given the circumstances, but I'm not sure what's responsible for updating crash servers.
Owner: ----
Owner: shuqianz@chromium.org
Conclusion is: this is a crashserver and hence not getting updated at all.
Charlene's work on updating devservers as part of push can have a follow up to include crash-servers as well.

Assigning.
Mergedinto: 729691
Status: Duplicate (was: Assigned)

Sign in to add a comment