New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 808081 link

Starred by 1 user
Status: Fixed
Owner:
█ jkop@chromium.org
Last visit > 30 days ago
Closed: Feb 2018
Cc:
█ vpalatin@chromium.org
█ ravisadineni@chromium.org
█ shuqianz@chromium.org
abhishekbh@chromium.org
█ jrbarnette@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Scarlet image missing in GS

Project Member Reported by jkop@chromium.org, Feb 1 2018 
All Scarlet bvt boards are failing with 500 errors when they attempt to retrieve images for R64.

https://storage.cloud.google.com/chromeos-autotest-results/hosts/chromeos2-row1-rack11-host5/62247353-repair/status.log

  
      
      500 Internal Server Error
      
      #powered_by {
          margin-top: 20px;
          border-top: 2px solid black;
          font-style: italic;
      }
  
      #traceback {
          color: red;
      }
      
  
      
          500 Internal Server Error
          The server encountered an unexpected condition which prevented it from fulfilling the request.
          Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond
      response.body = self.handler()
    File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__
      self.body = self.oldhandler(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__
      return self.callable(*self.args, **self.kwargs)
    File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 796, in is_staged
      response = str(dl.IsStaged(factory))
    File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 216, in IsStaged
      raise DownloaderException(exceptions)
  DownloaderException: return code: 1; command: /home/chromeos-test/chromeos-cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls -- gs://chromeos-image-archive/scarlet-release/R64-10176.61.0
  CommandException: One or more URLs matched no objects.
  
  cmd=['/home/chromeos-test/chromeos-cache/common/gsutil_4.27.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'ls', '--', u'gs://chromeos-image-archive/scarlet-release/R64-10176.61.0'], extra env={'BOTO_CONFIG': '/home/chromeos-test/.boto'}
  Traceback (most recent call last):
    File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 338, in Process
      self.name, self.is_regex_name, timeout)
    File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 332, in Wait
      is_regex_pattern=is_regex_name)
    File "/home/chromeos-test/chromiumos/chromite/lib/gs.py", line 1361, in GetGsNamesWithWait
      lambda x: not x, _GetGsName, timeout=timeout, period=period)
    File "/home/chromeos-test/chromiumos/chromite/lib/timeout_util.py", line 312, in WaitForSuccess
      return retry()
    File "/home/chromeos-test/chromiumos/chromite/lib/timeout_util.py", line 290, in retry
      value = func(*func_args, **func_kwargs)
    File "/home/chromeos-test/chromiumos/chromite/lib/gs.py", line 1339, in _GetGsName
      uploaded_list = [os.path.basename(p.url) for p in self.List(url)]
    File "/home/chromeos-test/chromiumos/chromite/lib/gs.py", line 1032, in List
      lines = self.DoCommand(cmd, **kwargs).output.splitlines()
    File "/home/chromeos-test/chromiumos/chromite/lib/gs.py", line 879, in DoCommand
      extra_env=extra_env, **kwargs)
    File "/home/chromeos-test/chromiumos/chromite/lib/retry_stats.py", line 181, in RetryWithStats
      *args, **kwargs)
    File "/home/chromeos-test/chromiumos/chromite/lib/retry_util.py", line 244, in GenericRetry
      return _run()
    File "/home/chromeos-test/chromiumos/chromite/lib/retry_util.py", line 201, in _Wrapper
      if not self._handler(e):
    File "/home/chromeos-test/chromiumos/chromite/lib/gs.py", line 745, in _RetryFilter
      raise error_details.exception
  GSNoSuchKey: return code: 1; command: /home/chromeos-test/chromeos-cache/common/gsutil_4.27.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' ls -- gs://chromeos-image-archive/scarlet-release/R64-10176.61.0
  CommandException: One or more URLs matched no objects.
  
  cmd=['/home/chromeos-test/chromeos-cache/common/gsutil_4.27.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'ls', '--', u'gs://chromeos-image-archive/scarlet-release/R64-10176.61.0'], extra env={'BOTO_CONFIG': '/home/chromeos-test/.boto'}
  
  
      
      Powered by CherryPy 3.2.2

Comment 1 by jkop@chromium.org, Feb 1 2018

Owner: jkop@chromium.org
Status: Assigned (was: Untriaged)
The DUTs are requesting R64-10176.61.0, but we only store R64-10176.0.0

pantheon/storage/browser/chromeos-image-archive/scarlet-release/?project=chromeos-bot&prefix=R64-1017

Fix may be to tell them to request R64-10177.0.0, which we also store.

Comment 2 by pprabhu@chromium.org, Feb 1 2018 

The default stable version is broken then: 

pprabhu@pprabhu:puppet$ atest stable_version list | grep 'DEFAULT\|scarlet'
DEFAULT                | R64-10176.61.0

This _should_ affect more than just scarlet. Any board that doesn't have a custom stable version.

Comment 3 by pprabhu@chromium.org, Feb 1 2018

Cc: jrbarnette@chromium.org
And here is a message about the automated stable version roll: http://shortn/_3aEkj1N6FF

That did the wrong thing. Immediate response should be to reset the DEFAULT stable version to the one that was before that roll (R64-10176.54.0)

Then we need to figure out why the automated roll is broken.

Comment 4 by jrbarnette@chromium.org, Feb 1 2018 

> Then we need to figure out why the automated roll is broken.

The automated roll isn't broken.  Most likely, scarlet is so
new that it has no Beta channel release.  Without a Beta channel,
there's no automated version selection.

When DUTs were added to the lab, the deployment process assigns
a known working build to be the repair image for the board.
So, either the deployment was done wrong, or something subsequently
deleted the scarlet entry from the table.

Comment 5 by jkop@chromium.org, Feb 1 2018 

Note that scarlet doesn't have artifacts for the previous stable version either: pantheon/storage/browser/chromeos-image-archive/scarlet-release?project=chromeos-bot&prefix=R64-1017

Comment 6 by jrbarnette@chromium.org, Feb 1 2018 

The scarlet deployment ticket is b/71489288.  I've looked it
over, and I don't see anything immediately wrong.  The request
was for "dru", not "scarlet"; however, the ticket says the DUTs
were actually deployed as "scarlet".

Comment 7 by jkop@chromium.org, Feb 1 2018

Labels: -Pri-1 Pri-2
Changed scarlet's default version to  R64-10176.0.0, which it has build artifacts for.

I'll hold this open to check on later.

Comment 8 by jrbarnette@chromium.org, Feb 1 2018 

The deployment logs are here:
    $ gsutil ls gs://chromeos-install-logs | grep scarlet
    gs://chromeos-install-logs/2018-01-23T12:48:26.808887-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-23T13:45:51.415480-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T10:35:17.806716-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T10:43:56.294587-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T10:48:43.088913-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T10:52:23.210918-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T10:57:03.853327-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T12:02:10.820140-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T15:18:58.145520-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T15:23:24.843154-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-24T15:32:31.405241-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T07:07:12.233268-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T08:24:32.774455-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T08:32:03.086567-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T08:41:47.079715-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T08:46:38.648059-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T08:49:41.979015-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T08:54:48.143349-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T09:42:21.213007-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T09:44:38.046337-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T09:53:15.307289-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T09:59:40.794234-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-25T10:02:18.720913-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-26T13:41:41.052970-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-26T14:44:19.079225-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-26T14:51:13.525896-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-26T15:18:36.086969-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-26T15:39:48.443933-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T14:44:15.851402-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T14:47:25.578032-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T14:49:59.724219-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:18:11.652965-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:20:38.447101-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:24:53.987187-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:26:35.246648-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:29:19.879066-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:30:47.265169-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-29T15:33:25.711586-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-30T08:01:47.276276-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-30T08:04:06.228316-08:00-scarlet/
    gs://chromeos-install-logs/2018-01-30T08:06:37.407844-08:00-scarlet/

The final deployment report can be seen here:
$ gsutil cat gs://chromeos-install-logs/2018-01-30T08:06:37.407844-08:00-scarlet/report.log
AFE    version is R64-10176.61.0.
Omaha  version is R61-9765.79.0.
AFE   firmware is None.

============
Repair version for board scarlet is now R64-10176.61.0.
Firmware       for board scarlet is now None.

============
Successes
============
chromeos2-row2-rack11-host20   Host added to pool:suites

Installation complete:  1 successes, 0 failures.
Logs will be uploaded to gs://chromeos-install-logs/2018-01-30T08:06:37.407844-08:00-scarlet

That shows that at the end, scarlet was assigned build R64-10176.61.0
for its repair image.  That's just the default Beta build for all
boards.

However, if that repair image didn't exist, the deployment should
have failed, because deployment installs the repair image.  I suspect
the deployment was done using the --noinstall option, which isn't
supposed to be used for production deployments.

Comment 9 by jrbarnette@chromium.org, Feb 1 2018 

> Changed scarlet's default version to  R64-10176.0.0, which it has build artifacts for.

That's not the tested version from the deployment ticket.

I've applied the versions requested by the ticket:
    $ atest stable_version modify -b scarlet -i R65-10294.0.0 
    Stable version for board scarlet is changed from R64-10176.0.0 to R65-10294.0.0.
    $ atest stable_version modify -b scarlet/rwfw -i Google_Scarlet.10210.0.0
    Stable version for board scarlet/rwfw is changed from R64-10176.61.0 to Google_Scarlet.10210.0.0.

It's unclear why the devices were deployed with the full install,
and that also means that we haven't tested servo on any of the
DUTs.  I'm running a spot check on one DUT to make sure that
servo works as expected.

Comment 10 by jrbarnette@chromium.org, Feb 1 2018 

The sanity check failed:  Servo for that DUT isn't working.


AFE    version is R65-10294.0.0.
Omaha  version is R61-9765.79.0.
AFE   firmware is Google_Scarlet.10210.0.0.

============
Repair version for board scarlet is now R65-10294.0.0.
Firmware       for board scarlet is now Google_Scarlet.10210.0.0.

============
Failures
============
chromeos2-row2-rack11-host20   No USB stick detected on Servo host

Comment 11 by jrbarnette@chromium.org, Feb 2 2018

Status: Fixed (was: Assigned)
This problem is fixed.  The underlying lab issue (fix the servos)
is being addressed in b/72827412.
►

Sign in to add a comment