New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 629341 link

Starred by 2 users
Status: Fixed
Owner:
█ dshi@chromium.org
Last visit > 30 days ago
Closed: Sep 2016
Cc:
akes...@chromium.org
█ shuqianz@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

puppet failed to setup base container when provisioning shards

Project Member Reported by shuqianz@chromium.org, Jul 19 2016 
puppet still failed to setup base container when provisioning shards.
$ FACTER_server_type="autotest-shard" puppet/run_puppet --debug
...........
...........
otice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.25.66.62
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: DEBUG:urllib3.connectionpool:"POST /cautotest/container_create HTTP/1.1" 201 114
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: INFO:elasticsearch:POST http://172.25.66.62:9800/cautotest/container_create [status:201 request:0.036s]
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: DEBUG:elasticsearch:> {"drone": "chromeos-server64.cbf.corp.google.com", "time_recorded": 1468877783.835471, "job_id": "", "success": false, "force_delete": false, "args": "(<__main__.ContainerBucket object at 0x7f2eb36472d0>,)", "name": "base_01"}
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: DEBUG:elasticsearch:< {"_index":"cautotest_2016_07","_type":"container_create","_id":"AVX_8EMew0Ez1cTRXWfl","_version":1,"created":true}
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: No handlers could be found for logger "elasticsearch.trace"
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 1005, in <module>
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     main()
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 999, in main
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     bucket.setup_base(name=options.name, force_delete=options.force_delete)
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 230, in func_cleanup_if_fail
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     return func(*args, **kwargs)
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 796, in setup_base
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     if self.exist(name) and not force_delete:
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 705, in exist
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     return self.get(name) != None
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 694, in get
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     return self.get_all().get(name, None)
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 678, in get_all
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     info_collection = get_container_info(self.container_path)
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/site_utils/lxc.py", line 186, in get_container_info
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     output = utils.run(cmd).stdout
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:   File "/usr/local/autotest/client/common_lib/base_utils.py", line 738, in run
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     "Command returned non-zero exit status")
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: autotest_lib.client.common_lib.error.CmdError: Command <sudo lxc-ls -P /usr/local/autotest/containers -f -F name,state> failed, rc=1, Command returned non-zero exit status
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: * Command: 
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns:     sudo lxc-ls -P /usr/local/autotest/containers -f -F name,state
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: Exit status: 1
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: Duration: 0.163708925247
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: 
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: stderr:
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: sudo: lxc-ls: command not found
Error: sudo python /usr/local/autotest/site_utils/lxc.py -s -n base_01 returned 1 instead of one of [0]
Error: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[setup_latest_base_container]/returns: change from notrun to 0 failed: sudo python /usr/local/autotest/site_utils/lxc.py -s -n base_01 returned 1 instead of one of [0]
Debug: Exec[Ensure base container file permission](provider=posix): Executing check 'stat -c '%U:%G' /usr/local/autotest/containers/base_01 | grep -q root:root'
Debug: Executing 'stat -c '%U:%G' /usr/local/autotest/containers/base_01 | grep -q root:root'
Debug: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[Ensure base container file permission]/unless: stat: cannot stat ‘/usr/local/autotest/containers/base_01’: No such file or directory
Debug: Exec[Ensure base container file permission](provider=posix): Executing 'sudo chown -R root:root /usr/local/autotest/containers/base_01'
Debug: Executing 'sudo chown -R root:root /usr/local/autotest/containers/base_01'
Notice: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[Ensure base container file permission]/returns: chown: cannot access ‘/usr/local/autotest/containers/base_01’: No such file or directory
Error: sudo chown -R root:root /usr/local/autotest/containers/base_01 returned 1 instead of one of [0]
Error: /Stage[main]/Lab::Autotest_repo_postinstall/Exec[Ensure base container file permission]/returns: change from notrun to 0 failed: sudo chown -R root:root /usr/local/autotest/containers/base_01 returned 1 instead of one of [0]
Debug: Class[Lab::Core_cluster]: The container Stage[main] will propagate my refresh event
Debug: Class[Lab::Autotest_common]: The container Stage[main] will propagate my refresh event
Notice: /Stage[main]/Lab::Autotest_scheduler/File[/etc/init/host-scheduler.conf]: Dependency Exec[setup_latest_base_container] has failures: true
Notice: /Stage[main]/Lab::Autotest_scheduler/File[/etc/init/host-scheduler.conf]: Dependency Exec[Ensure base container file permission] has failures: true
Warning: /Stage[main]/Lab::Autotest_scheduler/File[/etc/init/host-scheduler.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Lab::Autotest_scheduler/File[/etc/init/scheduler.conf]: Dependency Exec[setup_latest_base_container] has failures: true
Notice: /Stage[main]/Lab::Autotest_scheduler/File[/etc/init/scheduler.conf]: Dependency Exec[Ensure base container file permission] has failures: true
Warning: /Stage[main]/Lab::Autotest_scheduler/File[/etc/init/scheduler.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Lab::Autotest_scheduler/Exec[update-rc.d autotest remove]: Dependency Exec[setup_latest_base_container] has failures: true
Notice: /Stage[main]/Lab::Autotest_scheduler/Exec[update-rc.d autotest remove]: Dependency Exec[Ensure base container file permission] has failures: true
Warning: /Stage[main]/Lab::Autotest_scheduler/Exec[update-rc.d autotest remove]: Skipping because of failed dependencies
Debug: Executing '/usr/bin/apt-cache policy libapache2-mod-wsgi'
Debug: Exec[Enable module wsgi](provider=posix): Executing check 'test -f /etc/apache2/mods-enabled/wsgi.load'
Debug: Executing 'test -f /etc/apache2/mods-enabled/wsgi.load'
Debug: Exec[Enable module version](provider=posix): Executing check 'test -f /etc/apache2/mods-enabled/version.load'

Comment 1 by shuqianz@chromium.org, Jul 19 2016 

More nice logs:

No handlers could be found for logger "elasticsearch.trace"
Traceback (most recent call last):
  File "/usr/local/autotest/site_utils/lxc.py", line 1005, in <module>
    main()
  File "/usr/local/autotest/site_utils/lxc.py", line 999, in main
    bucket.setup_base(name=options.name, force_delete=options.force_delete)
  File "/usr/local/autotest/site_utils/lxc.py", line 230, in func_cleanup_if_fail
    return func(*args, **kwargs)
  File "/usr/local/autotest/site_utils/lxc.py", line 796, in setup_base
    if self.exist(name) and not force_delete:
  File "/usr/local/autotest/site_utils/lxc.py", line 705, in exist
    return self.get(name) != None
  File "/usr/local/autotest/site_utils/lxc.py", line 694, in get
    return self.get_all().get(name, None)
  File "/usr/local/autotest/site_utils/lxc.py", line 678, in get_all
    info_collection = get_container_info(self.container_path)
  File "/usr/local/autotest/site_utils/lxc.py", line 186, in get_container_info
    output = utils.run(cmd).stdout
  File "/usr/local/autotest/client/common_lib/base_utils.py", line 738, in run
    "Command returned non-zero exit status")
autotest_lib.client.common_lib.error.CmdError: Command <sudo lxc-ls -P /usr/local/autotest/containers -f -F name,state> failed, rc=1, Command returned non-zero exit status
* Command: 
    sudo lxc-ls -P /usr/local/autotest/containers -f -F name,state
Exit status: 1
Duration: 0.163708925247

stderr:
sudo: lxc-ls: command not found

Comment 2 by shuqianz@chromium.org, Jul 19 2016 

When I manually installed the lxc package in the test server: chromeos-server64.cbf, the lxc-ls command exists now.
$ sudo apt-get install lxc

So, I think the new base container package lxc2 doesn't work well. When puppet install lxc2 on the server, it only has two command tools: "lxc", "lxcfs".

Comment 3 by shuqianz@chromium.org, Jul 19 2016 

what was the reason we install lxc2 not lxc now?
Project Member

Comment 4 by bugdroid1@chromium.org, Jul 19 2016 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/57b3048f238ead1cec8baf17e7f6327185231012

commit 57b3048f238ead1cec8baf17e7f6327185231012
Author: Dan Shi <dshi@google.com>
Date: Tue Jul 19 21:32:26 2016

Comment 5 by dshi@chromium.org, Sep 9 2016

Status: Fixed (was: Assigned)
►

Sign in to add a comment