New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 591965 link

Starred by 1 user
Status: Fixed
Owner:
█ sbasi@chromium.org
Last visit > 30 days ago
Closed: Mar 2016
Cc:
aaboagye@chromium.org
█ seanpaul@chromium.org
█ sbasi@chromium.org
█ wiley@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

guado_moblab-paladin: HWTest fails "bash: /tmp/stateful_update: Permission denied"

Project Member Reported by drinkcat@chromium.org, Mar 4 2016 
Some recent guado_moblab-paladin HWTest fail with this error:

https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/1868

03/04 02:08:33.336 INFO |        server_job:0128| 		FAIL	provision_AutoUpdate	provision_AutoUpdate	timestamp=1457086113	localtime=Mar 04 02:08:33	Unhandled AutoservRunError: command execution error
  * Command: 
      /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_q9rfvHssh-
      master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
      -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
      ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
      chromeos2-row5-rack10-host1 "export LIBC_FATAL_STDERR_=1;
      /tmp/stateful_update --stateful_change=reset 2>&1"
  Exit status: 126
  Duration: 0.162019014359
  
  stdout:
  bash: /tmp/stateful_update: Permission denied
  Traceback (most recent call last):
    File "/usr/local/autotest/client/common_lib/test.py", line 741, in _call_test_function
      return func(*args, **dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 398, in execute
      dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 284, in _call_run_once_with_retry
      postprocess_profiled_run, args, dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 313, in _call_run_once
      self.run_once(*args, **dargs)
    File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 136, in run_once
      force_full_update=force)
    File "/usr/local/autotest/server/afe_utils.py", line 192, in machine_install_and_update_labels
      image_name, host_attributes = host.machine_install(*args, **dargs)
    File "/usr/local/autotest/server/hosts/cros_host.py", line 768, in machine_install
      updater.run_update()
    File "/usr/local/autotest/site-packages/statsd/timer.py", line 95, in _decorator
      return function(*args, **kwargs)
    File "/usr/local/autotest/client/common_lib/cros/autoupdater.py", line 474, in run_update
      self.reset_stateful_partition()
    File "/usr/local/autotest/client/common_lib/cros/autoupdater.py", line 360, in reset_stateful_partition
      self._run(' '.join(statefuldev_cmd))
    File "/usr/local/autotest/client/common_lib/cros/autoupdater.py", line 268, in _run
      return self.host.run(cmd, *args, **kwargs)
    File "/usr/local/autotest/server/hosts/ssh_host.py", line 162, in run
      options, stdin, args, ignore_timeout)
    File "/usr/local/autotest/server/hosts/ssh_host.py", line 130, in _run
      raise error.AutoservRunError("command execution error", result)
  AutoservRunError: command execution error
  * Command: 
      /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_q9rfvHssh-
      master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
      -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
      ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
      chromeos2-row5-rack10-host1 "export LIBC_FATAL_STDERR_=1;
      /tmp/stateful_update --stateful_change=reset 2>&1"
  Exit status: 126
  Duration: 0.162019014359
  
  stdout:  bash: /tmp/stateful_update: Permission denied

Comment 1 by drinkcat@chromium.org, Mar 4 2016 

Might be related to Issue 585572...

There is only one moblab in pool:cq left... As a hunch, I'm going to:
 - lock chromeos2-row5-rack10-host1
 - unlock chromeos2-row5-rack10-host11

Comment 2 by drinkcat@chromium.org, Mar 4 2016

Labels: -Pri-0 Pri-1
Owner: sbasi@chromium.org
Status: Assigned (was: Untriaged)
I tried to run the test that failed in the CQ on chromeos2-row5-rack10-host11:
http://cautotest/afe/#tab_id=view_job&object_id=55766553

And that seemed to work.

Please debug what's wrong with chromeos2-row5-rack10-host1, if anything is wrong, and unlock ASAP.

Comment 3 by drinkcat@chromium.org, Mar 4 2016 

Next run passed with HWTest on chromeos2-row5-rack10-host11...

Comment 4 by aaboagye@chromium.org, Mar 4 2016 

...And then the next run failed (build #10377)
https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/10377

Comment 5 by aaboagye@chromium.org, Mar 4 2016 

Although the failure mentioned in #4, is different. It looks like it couldn't test suites for sqawks? Also, why is it looking for R46? Stable should be R48 at least by now. 

03/04 06:28:17.886 DEBUG|      abstract_ssh:0329| get_file. source: /tmp/autoserv-K4N0Hr/sysinfo.pickle, dest: /tmp/tmp9x3KTH, delete_dest: False,preserve_perm: True, preserve_symlinks:False
03/04 06:28:17.887 DEBUG|      abstract_ssh:0340| Using Rsync.
03/04 06:28:17.888 DEBUG|        base_utils:0177| Running 'rsync -L  --timeout=1800 --rsh='/usr/bin/ssh -a -x   -o ControlPath=/tmp/_autotmp_hd1ecNssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g root@chromeos2-row5-rack10-host11:"/tmp/autoserv-K4N0Hr/sysinfo.pickle" "/tmp/tmp9x3KTH"'
03/04 06:28:18.372 DEBUG|   moblab_RunSuite:0048| Running suite: dummy_server.
03/04 06:28:18.373 DEBUG|          ssh_host:0153| Running (ssh) 'su - moblab -c '/usr/local/autotest/site_utils/run_suite.py --pool='' --board=squawks --build=squawks-release/R46-7390.59.0 --suite_name=dummy_server''
03/04 06:28:20.746 ERROR|        base_utils:0268| [stderr] WARNING:root:/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'point_conversion_form_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
03/04 06:28:20.748 ERROR|        base_utils:0268| [stderr]   % self._get_c_name())
03/04 06:28:20.749 ERROR|        base_utils:0268| [stderr] 
03/04 06:28:20.910 ERROR|        base_utils:0268| [stderr] Autotest instance: localhost
03/04 06:28:20.911 ERROR|        base_utils:0268| [stderr] 03-04-2016 [06:28:20] Submitted create_suite_job rpc
03/04 06:28:26.010 ERROR|        base_utils:0268| [stderr] Error Message: StageControlFileFailure: Failed to stage squawks-release/R46-7390.59.0: 
03/04 06:28:26.011 ERROR|        base_utils:0268| [stderr] 
03/04 06:28:26.011 ERROR|        base_utils:0268| [stderr] 
03/04 06:28:26.012 ERROR|        base_utils:0268| [stderr]     
03/04 06:28:26.013 ERROR|        base_utils:0268| [stderr]     500 Internal Server Error
03/04 06:28:26.013 ERROR|        base_utils:0268| [stderr]     
03/04 06:28:26.014 ERROR|        base_utils:0268| [stderr]     #powered_by {
03/04 06:28:26.014 ERROR|        base_utils:0268| [stderr]         margin-top: 20px;
03/04 06:28:26.015 ERROR|        base_utils:0268| [stderr]         border-top: 2px solid black;
03/04 06:28:26.016 ERROR|        base_utils:0268| [stderr]         font-style: italic;
03/04 06:28:26.016 ERROR|        base_utils:0268| [stderr]     }
03/04 06:28:26.017 ERROR|        base_utils:0268| [stderr] 
03/04 06:28:26.017 ERROR|        base_utils:0268| [stderr]     #traceback {
03/04 06:28:26.018 ERROR|        base_utils:0268| [stderr]         color: red;
03/04 06:28:26.018 ERROR|        base_utils:0268| [stderr]     }
03/04 06:28:26.019 ERROR|        base_utils:0268| [stderr]     
03/04 06:28:26.020 ERROR|        base_utils:0268| [stderr] 
03/04 06:28:26.020 ERROR|        base_utils:0268| [stderr]     
03/04 06:28:26.021 ERROR|        base_utils:0268| [stderr]         500 Internal Server Error
03/04 06:28:26.021 ERROR|        base_utils:0268| [stderr]         The server encountered an unexpected condition which prevented it from fulfilling the request.
03/04 06:28:26.022 ERROR|        base_utils:0268| [stderr]         Traceback (most recent call last):
03/04 06:28:26.023 ERROR|        base_utils:0268| [stderr]   File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
03/04 06:28:26.023 ERROR|        base_utils:0268| [stderr]     response.body = self.handler()
03/04 06:28:26.024 ERROR|        base_utils:0268| [stderr]   File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
03/04 06:28:26.024 ERROR|        base_utils:0268| [stderr]     self.body = self.oldhandler(*args, **kwargs)
03/04 06:28:26.025 ERROR|        base_utils:0268| [stderr]   File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
03/04 06:28:26.026 ERROR|        base_utils:0268| [stderr]     return self.callable(*self.args, **self.kwargs)
03/04 06:28:26.026 ERROR|        base_utils:0268| [stderr]   File "/usr/lib/devserver/devserver.py", line 669, in is_staged
03/04 06:28:26.027 ERROR|        base_utils:0268| [stderr]     return str(dl.IsStaged(factory))
03/04 06:28:26.028 DEBUG|        base_utils:0268| [stdout] [?25h[?0cFAILED RPC CALL: create_suite_job {'job_retry': False, 'suite_min_duts': 0, 'builds': {'cros-version': 'squawks-release/R46-7390.59.0'}, 'check_hosts': True, 'wait_for_results': True, 'offload_failures_only': False, 'suite_args': None, 'test_source_build': 'squawks-release/R46-7390.59.0', 'file_bugs': False, 'build': 'squawks-release/R46-7390.59.0', 'priority': 40, 'max_retries': None, 'num': None, 'max_runtime_mins': 1440, 'board': 'squawks', 'run_prod_code': False, 'timeout_mins': 1440, 'delay_minutes': 0, 'pool': '', 'name': 'dummy_server'}
03/04 06:28:26.028 ERROR|        base_utils:0268| [stderr]   File "/usr/lib/devserver/downloader.py", line 211, in IsStaged
03/04 06:28:26.029 ERROR|        base_utils:0268| [stderr]     raise DownloaderException(exceptions)
03/04 06:28:26.029 ERROR|        base_utils:0268| [stderr] DownloaderException: Could not find test_suites.tar.bz2 in Google Storage at gs://chromeos-image-archive/squawks-release/R46-7390.59.0
03/04 06:28:26.030 ERROR|        base_utils:0268| [stderr] 
03/04 06:28:26.030 ERROR|        base_utils:0268| [stderr]     
03/04 06:28:26.031 ERROR|        base_utils:0268| [stderr]     Powered by CherryPy 3.2.2

Comment 6 by sbasi@chromium.org, Mar 4 2016 

We need to update the repair build for moblab, I'll put up a CL
Project Member

Comment 7 by bugdroid1@chromium.org, Mar 4 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c882fe2ed29dde6040a48d42d579d36e502499dc

commit c882fe2ed29dde6040a48d42d579d36e502499dc
Author: Simran Basi <sbasi@google.com>
Date: Fri Mar 04 19:01:46 2016

[autotest] Update cros stable version.

R46 was deleted from Google Storage. Using the new default from
atest.

BUG= chromium:591965 
TEST=None

Change-Id: If7803ba3721fc314f908ac73336327d2fadaf605
Reviewed-on: https://chromium-review.googlesource.com/330426
Tested-by: Simran Basi <sbasi@chromium.org>
Reviewed-by: Dan Shi <dshi@chromium.org>

[modify] https://crrev.com/c882fe2ed29dde6040a48d42d579d36e502499dc/global_config.ini

Comment 8 by briannorris@chromium.org, Mar 5 2016 

I think R49-7834.42.0 is missing, too...

See this failure:

https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/1875

[...]
          500 Internal Server Error
          The server encountered an unexpected condition which prevented it from fulfilling the request.
          Traceback (most recent call last):
    File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
      response.body = self.handler()
    File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
      self.body = self.oldhandler(*args, **kwargs)
    File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
      return self.callable(*self.args, **self.kwargs)
    File "/usr/lib/devserver/devserver.py", line 669, in is_staged
      return str(dl.IsStaged(factory))
    File "/usr/lib/devserver/downloader.py", line 211, in IsStaged
      raise DownloaderException(exceptions)
  DownloaderException: Could not find test_suites.tar.bz2 in Google Storage at gs://chromeos-image-archive/peppy-release/R49-7834.42.0

Comment 9 by sbasi@chromium.org, Mar 7 2016

Status: Fixed (was: Assigned)
No different error, someone broke gsutil (used to download the test_suites tarball). ALl fixed now.
►

Sign in to add a comment