New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 710740 link

Starred by 1 user
Status: Archived
Owner:
haddowk@chromium.org
Closed: Apr 2017
Cc:
xixuan@chromium.org
█ stephenlin@chromium.org
█ ntang@chromium.org
█ twreid@chromium.org
dchan@chromium.org
█ krk@chromium.org
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Moblab M58 Beta fails to run CTS/GTS suites

Project Member Reported by haddowk@chromium.org, Apr 12 2017 
 500 Internal Server Error
        The server encountered an unexpected condition which prevented it from fulfilling the request.
        Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/devserver/devserver.py", line 788, in is_staged
    response = str(dl.IsStaged(factory))
  File "/usr/lib/devserver/downloader.py", line 216, in IsStaged
    raise DownloaderException(exceptions)
DownloaderException: 'NoneType' object has no attribute 'startswith'

Comment 1 by haddowk@chromium.org, Apr 12 2017

Cc: stephenlin@chromium.org

Comment 2 by haddowk@chromium.org, Apr 12 2017

Status: Assigned (was: Untriaged)

Comment 3 by haddowk@chromium.org, Apr 14 2017 

Seems to be tracked down to this so far:

GSCommandError: return code: None; command: /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls -- gs://chromeos-moblab-haddowktest/snappy-release/R59-9452.0.0
[Errno 13] Permission denied; does the program need `chmod a+x`?

Comment 4 by haddowk@chromium.org, Apr 14 2017 

I do not really understand how this is possible

moblab@localhost /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil $ sudo chmod -R +x /tmp
moblab@localhost /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil $ ls -lrt gsutil
-r-xr-xr-x 1 moblab moblab 165 Apr 13 17:23 gsutil
moblab@localhost /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil $ ./gsutil
-bash: ./gsutil: Permission denied
moblab@localhost /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil $ cat ./gsutil
#!/usr/bin/env python

"""Wrapper Python script for running Google Cloud Storage command line tool."""

import gsutil

if __name__ == '__main__':
  gsutil.RunMain()
moblab@localhost /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil $ /usr/bin/env python
Python 2.7.10 (default, Apr  6 2017, 22:00:05) 
[GCC 4.2.1 Compatible Chromium OS 4.0_pre285905-r7 Clang 4.0.0 (/var/cache/chro on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gsutil
>>> gsutil.RunMain()
Usage: gsutil [-D] [-DD] [-h header]... [-m] [-o] [-q] [command [opts...] args...]
Available commands:
  acl            Get, set, or change bucket and/or object ACLs
  cat            Concatenate object content to stdout
  compose        Concatenate a sequence of objects into a new composite object.
  config         Obtain credentials and create configuration file
  cors           Set a CORS JSON document for one or more buckets
  cp             Copy files and objects
  defacl         Get, set, or change default ACL on buckets
  du             Display object size usage
  hash           Calculate file hashes
  help           Get help about commands and topics
  lifecycle      Get or set lifecycle configuration for a bucket
  logging        Configure or retrieve logging on buckets
  ls             List providers, buckets, or objects
  mb             Make buckets
  mv             Move/rename objects and/or subdirectories
  notification   Configure object change notification
  perfdiag       Run performance diagnostic
  rb             Remove buckets
  rewrite        Rewrite objects
  rm             Remove objects
  rsync          Synchronize content of two buckets/directories
  setmeta        Set metadata on already uploaded objects
  signurl        Create a signed url
  stat           Display object status
  test           Run gsutil tests
  update         Update to the latest gsutil release
  version        Print version info about gsutil
  versioning     Enable or suspend versioning for one or more buckets
  web            Set a main page and/or error page for one or more buckets

Additional help topics:
  acls           Working With Access Control Lists
  anon           Accessing Public Data Without Credentials
  apis           Cloud Storage APIs
  crc32c         CRC32C and Installing crcmod
  creds          Credential Types Supporting Various Use Cases
  csek           Supplying Your Own Encryption Keys
  dev            Contributing Code to gsutil
  encoding       Filename encoding and interoperability problems
  metadata       Working With Object Metadata
  naming         Object and Bucket Naming
  options        Top-Level Command-Line Options
  prod           Scripting Production Transfers
  projects       Working With Projects
  retries        Retry Handling Strategy
  security       Security and Privacy Considerations
  subdirs        How Subdirectories Work
  support        Google Cloud Storage Support
  throttling     Throttling gsutil
  versions       Object Versioning and Concurrency Control
  wildcards      Wildcard Names

Use gsutil help <command or topic> for detailed help.

Comment 5 by haddowk@chromium.org, Apr 14 2017

Cc: xixuan@chromium.org
Adding xixuan to see if she has any ideas since I think she was involved with the switch of gsutil to chromite version.

Comment 6 by xixuan@chromium.org, Apr 14 2017 

could you provide a log link for this failed test?

Comment 7 by haddowk@chromium.org, Apr 14 2017 


The tests will not run - the devserver is not able to stage the files so the suite will not even run.

The moblab I am useing and the command to get the error below,  I have made some file changes to this moblab to try and get better error messages vs the one that first came out - Happy to flash it back to pristine R58 if you prefer.

haddowk@haddow:~/chromiumos_internal_moblab_R58$ ssh moblab@100.107.2.199
moblab@localhost ~ $ curl "http://192.168.231.1:8080/stage?archive_url=gs://chromeos-moblab-haddowktest/snappy-release/R59-9452.0.0&artifacts=test_suites"
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>500 Internal Server Error</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>500 Internal Server Error</h2>
        <p>The server encountered an unexpected condition which prevented it from fulfilling the request.</p>
        <pre id="traceback">Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/devserver/devserver.py", line 877, in stage
    dl.Download(factory, async=async)
  File "/usr/lib/devserver/downloader.py", line 196, in Download
    self._DownloadArtifactsSerially(required_artifacts, no_wait=True)
  File "/usr/lib/devserver/downloader.py", line 236, in _DownloadArtifactsSerially
    artifact.Process(self, no_wait)
  File "/usr/lib/devserver/build_artifact.py", line 337, in Process
    self.name, self.is_regex_name, timeout)
  File "/usr/lib/devserver/downloader.py", line 333, in Wait
    is_regex_pattern=is_regex_name)
  File "/usr/lib64/python2.7/site-packages/chromite/lib/gs.py", line 1257, in GetGsNamesWithWait
    lambda x: not x, _GetGsName, timeout=timeout, period=period)
  File "/usr/lib64/python2.7/site-packages/chromite/lib/timeout_util.py", line 292, in WaitForSuccess
    return retry()
  File "/usr/lib64/python2.7/site-packages/chromite/lib/timeout_util.py", line 270, in retry
    value = func(*func_args, **func_kwargs)
  File "/usr/lib64/python2.7/site-packages/chromite/lib/gs.py", line 1235, in _GetGsName
    uploaded_list = [os.path.basename(p.url) for p in self.List(url)]
  File "/usr/lib64/python2.7/site-packages/chromite/lib/gs.py", line 937, in List
    lines = self.DoCommand(cmd, **kwargs).output.splitlines()
  File "/usr/lib64/python2.7/site-packages/chromite/lib/gs.py", line 782, in DoCommand
    raise GSCommandError(e.msg, e.result, e.exception)
GSCommandError: return code: None; command: /tmp/chromeos-cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls -- gs://chromeos-moblab-haddowktest/snappy-release/R59-9452.0.0
[Errno 13] Permission denied; does the program need `chmod a+x`?
</pre>
    <div id="powered_by">
    <span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span>
    </div>
    </body>
</html>

Comment 8 by xixuan@chromium.org, Apr 14 2017 

I suggest to use user root to log into the DUT, since /tmp/ seems only to be available for root. "root" is also used in real tests & cros flash ssh://.

Comment 9 by haddowk@chromium.org, Apr 14 2017 

It is not possible to log into the moblab as root - work was done to specifically deny this.

I flashed again with a new 58 build that had this cl merged in https://chromium-review.googlesource.com/#/c/457859/

However devserver still can not issue gsutil commands - any other CL's you think might need to be CP back to 58 ?

moblab@localhost ~ $ !373                                                                                                                                 
curl "http://192.168.231.1:8080/stage?archive_url=gs://chromeos-moblab-haddowktest/snappy-release/R59-9452.0.0&artifacts=test_suites"
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>500 Internal Server Error</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>500 Internal Server Error</h2>
        <p>The server encountered an unexpected condition which prevented it from fulfilling the request.</p>
        <pre id="traceback">Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/usr/lib64/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/devserver/devserver.py", line 877, in stage
    dl.Download(factory, async=async)
  File "/usr/lib/devserver/downloader.py", line 196, in Download
    self._DownloadArtifactsSerially(required_artifacts, no_wait=True)
  File "/usr/lib/devserver/downloader.py", line 235, in _DownloadArtifactsSerially
    artifact.Process(self, no_wait)
  File "/usr/lib/devserver/build_artifact.py", line 351, in Process
    raise ArtifactDownloadError('An error occurred: %s' % e)
ArtifactDownloadError: An error occurred: 'NoneType' object has no attribute 'startswith'
</pre>
    <div id="powered_by">
    <span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span>
    </div>
    </body>
</html>

Comment 10 by xixuan@chromium.org, Apr 14 2017 

No clue for where the statswith comes from. Could you merge CL https://chromium-review.googlesource.com/c/452806/ for better checking the traceback?

Comment 12 by stephenlin@google.com, Apr 14 2017 

Just trying things out, you can copy an executable file from one location to another and it will not execute in the other location even with proper permissions. For example:

whoami
moblab

cd /home/moblab
cp /bin/ls .
moblab@localhost ~ $ ls -l 
-r-xr-xr-x 1 moblab moblab  49 Apr 14 09:57 ls
./ls
-bash: ./ls: Permission denied
sudo ./ls
sudo: unable to execute ./ls: Permission denied

ls -ld /home
drwxr-xr-x 8 root root 4096 Mar 10 11:36 /home
ls -ld /home/moblab
drwxr-xr-x 4 moblab moblab 4096 Apr 14 09:57 /home/moblab

You can also just create any old executable script and it won't execute either. This problem doesn't seem specific to gsutil.

Moblab IP is 100.107.2.199 if you want to take a direct look.

Comment 13 by stephenlin@google.com, Apr 14 2017 

In chatting with yueherngl we looked at the mount:

mount | grep '/tmp'
tmp on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)

mount | grep '/home'
/dev/sda1 on /home type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30,data=ordered)

As you can see both /tmp and /home are mounted with noexec.

How is moblab different than the testlab? Is the file put into the same location? Is the mounting different?

Comment 14 by haddowk@chromium.org, Apr 14 2017 

I made a new build with all the CP requested

https://crosland.corp.google.com/log/9334.46.0..9334.47.0

However to real difference to the problem

doing sudo mount -o remount,exec /tmp

Does seem to fix the problem.   Is there anything we can do to move where these scripts so they can be in a partition of our choice ?

Comment 15 by xixuan@chromium.org, Apr 14 2017 

I don't actually get it, seems it's the partition mount's problem on a particular DUT (or all moblabs?)

To summarize, there's a case in CTS testing for moblab 1) requires 'moblab' is the user (cannot be root), 2) but normally user 'moblab' cannot execute any commands in /tmp/ on any working DUT, is that right? 

If it's the case I can make a fix for that.

Comment 16 by xixuan@chromium.org, Apr 14 2017 

what confuses me is why user 'moblab' cannot execute any commands in /tmp/. 

Sounds like a DUT problem. I test with my local desktop moblab and it's fine.

Comment 17 by haddowk@chromium.org, Apr 15 2017 


So just to be clear - we are not executing anything on any DUT's  This is strictly on the devserver on the moblab.

Most processes on the moblab run under user moblab

The moblab partition is mounted in such a way that no scripts can be executed in /tmp

I checked several moblab and they are all the same

tmp on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)
Project Member

Comment 18 by bugdroid1@chromium.org, Apr 26 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/53a0a4d9ae98ae76d860a5bf46c4200ace3cbf19

commit 53a0a4d9ae98ae76d860a5bf46c4200ace3cbf19
Author: Keith Haddow <haddowk@chromium.org>
Date: Wed Apr 26 20:37:04 2017

[moblab] Make change so chromite install dir is in moblab static dir.

Autotest devserver moved to using the chromite gsutil.  By default
chromite gsutil installs the executables in the /tmp directory.
Moblab mounts the /tmp directory noexec so no scripts can be run.

Set enviroment variable CROS_CACHEDIR to change where chromite
installs the gsutil, set it to be under /mnt/moblab/static so it
will be executable.

BUG= chromium:710740 
TEST=Manual testing on affected moblab

Change-Id: Icc723f5d66057e91e829d8397ed459429ac3a2fe
Reviewed-on: https://chromium-review.googlesource.com/479895
Commit-Ready: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[rename] https://crrev.com/53a0a4d9ae98ae76d860a5bf46c4200ace3cbf19/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.5-r53.ebuild
[modify] https://crrev.com/53a0a4d9ae98ae76d860a5bf46c4200ace3cbf19/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf
Project Member

Comment 19 by bugdroid1@chromium.org, Apr 26 2017

Labels: merge-merged-release-R58-9334.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b

commit 111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b
Author: Keith Haddow <haddowk@chromium.org>
Date: Wed Apr 26 20:49:24 2017

[moblab] Make change so chromite install dir is in moblab static dir.

Autotest devserver moved to using the chromite gsutil.  By default
chromite gsutil installs the executables in the /tmp directory.
Moblab mounts the /tmp directory noexec so no scripts can be run.

Set enviroment variable CROS_CACHEDIR to change where chromite
installs the gsutil, set it to be under /mnt/moblab/static so it
will be executable.

BUG= chromium:710740 
TEST=Manual testing on affected moblab

Change-Id: Icc723f5d66057e91e829d8397ed459429ac3a2fe
Reviewed-on: https://chromium-review.googlesource.com/487607
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Michael Tang <ntang@chromium.org>
Commit-Queue: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Trybot-Ready: Keith Haddow <haddowk@chromium.org>

[rename] https://crrev.com/111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.5-r53.ebuild
[modify] https://crrev.com/111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf
Project Member

Comment 20 by bugdroid1@chromium.org, Apr 26 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b

commit 111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b
Author: Keith Haddow <haddowk@chromium.org>
Date: Wed Apr 26 20:49:24 2017

[moblab] Make change so chromite install dir is in moblab static dir.

Autotest devserver moved to using the chromite gsutil.  By default
chromite gsutil installs the executables in the /tmp directory.
Moblab mounts the /tmp directory noexec so no scripts can be run.

Set enviroment variable CROS_CACHEDIR to change where chromite
installs the gsutil, set it to be under /mnt/moblab/static so it
will be executable.

BUG= chromium:710740 
TEST=Manual testing on affected moblab

Change-Id: Icc723f5d66057e91e829d8397ed459429ac3a2fe
Reviewed-on: https://chromium-review.googlesource.com/487607
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Michael Tang <ntang@chromium.org>
Commit-Queue: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Trybot-Ready: Keith Haddow <haddowk@chromium.org>

[rename] https://crrev.com/111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.5-r53.ebuild
[modify] https://crrev.com/111a4109ad33ea2ed95c64210d92d9ab6e6c3d6b/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf
Project Member

Comment 21 by bugdroid1@chromium.org, Apr 26 2017

Labels: merge-merged-release-R59-9460.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/adc746459815945e322027023d7ba370236217a1

commit adc746459815945e322027023d7ba370236217a1
Author: Keith Haddow <haddowk@chromium.org>
Date: Wed Apr 26 20:51:16 2017

[moblab] Make change so chromite install dir is in moblab static dir.

Autotest devserver moved to using the chromite gsutil.  By default
chromite gsutil installs the executables in the /tmp directory.
Moblab mounts the /tmp directory noexec so no scripts can be run.

Set enviroment variable CROS_CACHEDIR to change where chromite
installs the gsutil, set it to be under /mnt/moblab/static so it
will be executable.

BUG= chromium:710740 
TEST=Manual testing on affected moblab

Change-Id: Icc723f5d66057e91e829d8397ed459429ac3a2fe
Reviewed-on: https://chromium-review.googlesource.com/487608
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Michael Tang <ntang@chromium.org>
Commit-Queue: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Trybot-Ready: Keith Haddow <haddowk@chromium.org>

[rename] https://crrev.com/adc746459815945e322027023d7ba370236217a1/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.5-r53.ebuild
[modify] https://crrev.com/adc746459815945e322027023d7ba370236217a1/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf
Project Member

Comment 22 by bugdroid1@chromium.org, Apr 26 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/adc746459815945e322027023d7ba370236217a1

commit adc746459815945e322027023d7ba370236217a1
Author: Keith Haddow <haddowk@chromium.org>
Date: Wed Apr 26 20:51:16 2017

[moblab] Make change so chromite install dir is in moblab static dir.

Autotest devserver moved to using the chromite gsutil.  By default
chromite gsutil installs the executables in the /tmp directory.
Moblab mounts the /tmp directory noexec so no scripts can be run.

Set enviroment variable CROS_CACHEDIR to change where chromite
installs the gsutil, set it to be under /mnt/moblab/static so it
will be executable.

BUG= chromium:710740 
TEST=Manual testing on affected moblab

Change-Id: Icc723f5d66057e91e829d8397ed459429ac3a2fe
Reviewed-on: https://chromium-review.googlesource.com/487608
Reviewed-by: Keith Haddow <haddowk@chromium.org>
Reviewed-by: Michael Tang <ntang@chromium.org>
Commit-Queue: Keith Haddow <haddowk@chromium.org>
Tested-by: Keith Haddow <haddowk@chromium.org>
Trybot-Ready: Keith Haddow <haddowk@chromium.org>

[rename] https://crrev.com/adc746459815945e322027023d7ba370236217a1/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.5-r53.ebuild
[modify] https://crrev.com/adc746459815945e322027023d7ba370236217a1/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf

Comment 23 by haddowk@chromium.org, Apr 26 2017

Status: Fixed (was: Assigned)

Comment 24 by dchan@google.com, May 30 2017

Labels: VerifyIn-60

Comment 25 by dchan@chromium.org, Aug 1 2017

Labels: VerifyIn-61

Comment 26 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)
►

Sign in to add a comment