Update/Provision devservers in chromeos15 & add into Prod |
|||||||||||
Issue descriptionPlease provision and add below devservers into prod. chromeos15-infra-devserver15.cros.corp.google.com chromeos15-infra-devserver16.cros.corp.google.com chromeos15-infra-devserver17.cros.corp.google.com chromeos15-infra-devserver18.cros.corp.google.com chromeos15-infra-devserver19.cros.corp.google.com chromeos15-infra-devserver20.cros.corp.google.com Please use the new netblock (100.90.8.0/21) while adding the attributes. Attributes : {u'restricted_subnet': u'100.90.8.0/21', u'port': u'8082'} Once added please provide the output of 'atest server list' of all devservers in chromeos15. Please ping me if you have any query. Thanks, Mani
,
Sep 27
,
Sep 28
Any update on this bug.
,
Sep 29
,
Oct 1
Reassigning to the next deputy.
,
Oct 2
The task was blocked by an issue that cannot resolve www.googleapis.com, though I think the task should had been finished successfully. [chromeos15-infra-devserver15.cros.corp.google.com] run: curl "http://localhost:8082/stage?build_id=2457013&files=&target=shamu-user debug&artifacts=bootloader_image,zip_images,test_zip&branch=git_mnc-release&os_type=android" [chromeos15-infra-devserver15.cros.corp.google.com] out: [chromeos15-infra-devserver15.cros.corp.google.com] out: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" [chromeos15-infra-devserver15.cros.corp.google.com] out: "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> [chromeos15-infra-devserver15.cros.corp.google.com] out: <html> [chromeos15-infra-devserver15.cros.corp.google.com] out: <head> [chromeos15-infra-devserver15.cros.corp.google.com] out: <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></me ta> [chromeos15-infra-devserver15.cros.corp.google.com] out: <title>500 Internal Server Error</title> [chromeos15-infra-devserver15.cros.corp.google.com] out: <style type="text/css"> [chromeos15-infra-devserver15.cros.corp.google.com] out: #powered_by { [chromeos15-infra-devserver15.cros.corp.google.com] out: margin-top: 20px; [chromeos15-infra-devserver15.cros.corp.google.com] out: border-top: 2px solid black; [chromeos15-infra-devserver15.cros.corp.google.com] out: font-style: italic; [chromeos15-infra-devserver15.cros.corp.google.com] out: } [chromeos15-infra-devserver15.cros.corp.google.com] out: [chromeos15-infra-devserver15.cros.corp.google.com] out: #traceback { [chromeos15-infra-devserver15.cros.corp.google.com] out: color: red; [chromeos15-infra-devserver15.cros.corp.google.com] out: } [chromeos15-infra-devserver15.cros.corp.google.com] out: </style> [chromeos15-infra-devserver15.cros.corp.google.com] out: </head> [chromeos15-infra-devserver15.cros.corp.google.com] out: <body> [chromeos15-infra-devserver15.cros.corp.google.com] out: <h2>500 Internal Server Error</h2> [chromeos15-infra-devserver15.cros.corp.google.com] out: <p>The server encountered an unexpected condition which prevented i t from fulfilling the request.</p> [chromeos15-infra-devserver15.cros.corp.google.com] out: <pre id="traceback">Traceback (most recent call last): [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond [chromeos15-infra-devserver15.cros.corp.google.com] out: response.body = self.handler() [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 18 8, in __call__ [chromeos15-infra-devserver15.cros.corp.google.com] out: self.body = self.oldhandler(*args, **kwargs) [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__ [chromeos15-infra-devserver15.cros.corp.google.com] out: return self.callable(*self.args, **self.kwargs) [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 881, in stage [chromeos15-infra-devserver15.cros.corp.google.com] out: dl.Download(factory, async=async) [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", lin e 196, in Download [chromeos15-infra-devserver15.cros.corp.google.com] out: self._DownloadArtifactsSerially(required_artifacts, no_wait=True) [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 235, in _DownloadArtifactsSerially [chromeos15-infra-devserver15.cros.corp.google.com] out: artifact.Process(self, no_wait) [chromeos15-infra-devserver15.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 352, in Process [chromeos15-infra-devserver15.cros.corp.google.com] out: raise ArtifactDownloadError('An error occurred: %s' % e) [chromeos15-infra-devserver15.cros.corp.google.com] out: ArtifactDownloadError: An error occurred: Unable to find the server at www.googleapis.com [chromeos15-infra-devserver15.cros.corp.google.com] out: </pre> [chromeos15-infra-devserver15.cros.corp.google.com] out: <div id="powered_by"> [chromeos15-infra-devserver15.cros.corp.google.com] out: <span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span> [chromeos15-infra-devserver15.cros.corp.google.com] out: </div> [chromeos15-infra-devserver15.cros.corp.google.com] out: </body> [chromeos15-infra-devserver15.cros.corp.google.com] out: </html> [chromeos15-infra-devserver15.cros.corp.google.com] out:
,
Oct 2
Seems it's helpful to run the deployment task more times.
For now, chromeos15-infra-devserver{15,16,19,20} has been done. But for 17 and 18, they were failed due to below error.
[chromeos15-infra-devserver17.cros.corp.google.com] run: curl "http://localhost:8082/stage?build_id=2457013&files=&target=shamu-user
debug&artifacts=bootloader_image,zip_images,test_zip&branch=git_mnc-release&os_type=android"
[chromeos15-infra-devserver17.cros.corp.google.com] out: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
[chromeos15-infra-devserver17.cros.corp.google.com] out: "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
[chromeos15-infra-devserver17.cros.corp.google.com] out: <html>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <head>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></me
ta>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <title>500 Internal Server Error</title>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <style type="text/css">
[chromeos15-infra-devserver17.cros.corp.google.com] out: #powered_by {
[chromeos15-infra-devserver17.cros.corp.google.com] out: margin-top: 20px;
[chromeos15-infra-devserver17.cros.corp.google.com] out: border-top: 2px solid black;
[chromeos15-infra-devserver17.cros.corp.google.com] out: font-style: italic;
[chromeos15-infra-devserver17.cros.corp.google.com] out: }
[chromeos15-infra-devserver17.cros.corp.google.com] out:
[chromeos15-infra-devserver17.cros.corp.google.com] out: #traceback {
[chromeos15-infra-devserver17.cros.corp.google.com] out: color: red;
[chromeos15-infra-devserver17.cros.corp.google.com] out: }
[chromeos15-infra-devserver17.cros.corp.google.com] out: </style>
[chromeos15-infra-devserver17.cros.corp.google.com] out: </head>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <body>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <h2>500 Internal Server Error</h2>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <p>The server encountered an unexpected condition which prevented i
t from fulfilling the request.</p>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <pre id="traceback">Traceback (most recent call last):
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656,
in respond
[chromeos15-infra-devserver17.cros.corp.google.com] out: response.body = self.handler()
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__
[chromeos15-infra-devserver17.cros.corp.google.com] out: self.body = self.oldhandler(*args, **kwargs)
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__
[chromeos15-infra-devserver17.cros.corp.google.com] out: return self.callable(*self.args, **self.kwargs)
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 877, in stage
[chromeos15-infra-devserver17.cros.corp.google.com] out: dl.Download(factory, async=async)
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 196, in Download
[chromeos15-infra-devserver17.cros.corp.google.com] out: self._DownloadArtifactsSerially(required_artifacts, no_wait=True)
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 235, in _DownloadArtifactsSerially
[chromeos15-infra-devserver17.cros.corp.google.com] out: artifact.Process(self, no_wait)
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/build_artifact.py", line 337, in Process
[chromeos15-infra-devserver17.cros.corp.google.com] out: self.name, self.is_regex_name, timeout)
[chromeos15-infra-devserver17.cros.corp.google.com] out: File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 455, in Wait
[chromeos15-infra-devserver17.cros.corp.google.com] out: (name, self.target, self.build_id))
[chromeos15-infra-devserver17.cros.corp.google.com] out: ArtifactDownloadError: No artifact found with given name: .*-img-[^-]*\.zip for shamu-userdebug-2457013
[chromeos15-infra-devserver17.cros.corp.google.com] out: </pre>
[chromeos15-infra-devserver17.cros.corp.google.com] out: <div id="powered_by">
[chromeos15-infra-devserver17.cros.corp.google.com] out: <span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span>
[chromeos15-infra-devserver17.cros.corp.google.com] out: </div>
[chromeos15-infra-devserver17.cros.corp.google.com] out: </body>
[chromeos15-infra-devserver17.cros.corp.google.com] out: </html>
[chromeos15-infra-devserver17.cros.corp.google.com] out:
,
Oct 2
The issue of ArtifactDownloadError seems caused by stale code :-/ I resolved it by run `repo sync` (our puppet code doesn't do that?!) and restart devserver.
,
Oct 3
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/infra_internal/skylab_inventory/+/24fd3548ec7a4a64e53340966e3315a5da903f56 commit 24fd3548ec7a4a64e53340966e3315a5da903f56 Author: Congbin Guo <guocb@google.com> Date: Wed Oct 03 00:34:09 2018
,
Oct 3
$ atest server list |rg -A7 chromeos15 hostname: "chromeos15-infra-devserver1.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver10.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver11.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver12.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver13.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver14.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver15.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver16.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver17.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver18.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver19.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver2.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver20.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.90.8.0/21" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver3.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver4.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver5.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver6.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver7.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver8.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 } -- hostname: "chromeos15-infra-devserver9.cros.corp.google.com" environment: ENVIRONMENT_PROD status: STATUS_PRIMARY roles: ROLE_DEVSERVER attributes { devserver_restricted_subnet: "100.115.124.0/22" devserver_port: 8082 }
,
Oct 4
The provision was not fully successfully, see issue 891765 for details. I guess the reason is, there were some old checkout of devserver repo, and our provision script doesn't update the checkout, so they were running the old version which results in many weird problems. For devserver19, when I tried to fix this problem, I got error of read-only file system: chromeos-test@chromeos15-infra-devserver19:~/chromiumos/src/platform/dev$ repo sync . error: cannot open /home/chromeos-test/chromiumos/.repo/manifests.git/FETCH_HEAD: Read-only file system error: cannot open /home/chromeos-test/chromiumos/.repo/manifests.git/FETCH_HEAD: Read-only file system ... Did someone configure something?
,
Oct 4
The readonly FS issue should be related to this configuration chromeos-test@chromeos15-infra-devserver19:~/images$ mount /dev/sda1 on / type ext4 (rw,errors=remount-ro) ... Since / is readonly, i don't believe it can continue serve. So stop the devserver process and also remove it from the output of `atest server list` Reassign the ticket back to requester and wait until the 'readonly fs' issue get resolved.
,
Oct 4
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/infra_internal/skylab_inventory/+/daf05f8f3afde12b4b456591e58f25ea973d0ce9 commit daf05f8f3afde12b4b456591e58f25ea973d0ce9 Author: Congbin Guo <guocb@google.com> Date: Thu Oct 04 01:37:57 2018
,
Oct 4
,
Oct 4
Fixed the FS issue in chromeos15-infra-devserver19 Can you please check and let me know.
,
Oct 4
,
Oct 4
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/infra_internal/skylab_inventory/+/23527f519589c05f1d9103e9263c42a70181f306 commit 23527f519589c05f1d9103e9263c42a70181f306 Author: Congbin Guo <guocb@google.com> Date: Thu Oct 04 22:07:56 2018 |
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by mjayapal@chromium.org
, Sep 26