Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 2 users
Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Apr 10
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 0
Type: Bug

Blocking:
issue 692179



Sign in to add a comment
chromeos2-row8-rack9-host14 unable to reach devserver
Project Member Reported by shuqianz@chromium.org, Feb 13 2017 Back to list
Provision failed due to "client-autotest.tar.bz2 could not be fetched from any of the repos ['http://100.115.245.198:8082/static/kevin-paladin/R58-9280.0.0-rc2/autotest/packages']"

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row8-rack9-host14/59966813-provision/debug/

It has been caused kevin paladin failed for three times.
 
Cc: akes...@chromium.org
Owner: xixuan@chromium.org
Summary: Provision failed to fetch client-autotest.tar.bz2 for kevin paladin (was: kevin paladin fails due to bad image)
Summary of the bug:
build failure: https://uberchromegw.corp.google.com/i/chromeos/builders/kevin-paladin/builds/198
The log of the failed provsion:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row8-rack9-host14/59966813-provision/debug/

It complains that client-autotest.tar.bz2 could not be fetched from any of the repos ['http://100.115.245.198:8082/static/kevin-paladin/R58-9280.0.0-rc2/autotest/packages'

I manually download the autotest_package.tar from the image https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/kevin-paladin/R58-9280.0.0-rc2/. The client-autotest.tar.bz2 is actually there
Comment 3 by dshi@chromium.org, Feb 13 2017
Try to stage the artifact for the same build again? It could be some devserver issue. Also, check the timestamp of that client-autotest.tar.bz2 on GS, maybe there is a timing issue?
Yeah, I think all three builds 198, 197, 196 were trying to stage for the same build and failed. The timestamp for the autotest_packages.tar is 2/13/17 5:41 AM. So, I don't think it is a timing issue that can affect all three builds.
ping
Comment 6 by xixuan@chromium.org, Feb 14 2017
looking
Comment 7 by xixuan@chromium.org, Feb 14 2017
Cc: xixuan@chromium.org
Owner: shuqianz@chromium.org
It's because host chromeos2-row8-rack9-host14.cros (100.115.231.55) fails to run 'wget --connect-timeout=15 --retry-connrefused --wait=5 -nv http://100.115.245.198:8082/static/kevin-paladin/R58-9280.0.0-rc2/autotest/packages/client-autotest.tar.bz2 -O /tmp/sysinfo/autoserv-4oNwST/packages/client-autotest.tar.bz2' command. I ssh to the host and manually run this cmd, still fail.

Instead I tried to ssh chromeos2-row6-rack5-host17.cros (100.115.228.226), and successfully executes this cmd. 

Usually the host will find a devserver in its subnet and should be able to run wget. I think there's some network config/connection issue between host chromeos2-row8-rack9-host14.cros and devserver 100.115.245.198.

So reassign to deputy to file a bug to engLab.
Labels: -Pri-1 Pri-0
$ dut-status -f chromeos2-row8-rack9-host14
chromeos2-row8-rack9-host14
    2017-02-14 09:25:39  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59983972-provision/
    2017-02-14 08:18:32  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59983270-repair/
    2017-02-14 06:45:58  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59982087-provision/
    2017-02-14 05:33:17  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59981280-repair/
    2017-02-14 04:01:15  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59979979-provision/
    2017-02-14 02:42:58  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59979040-repair/
    2017-02-14 01:08:35  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59978033-provision/
    2017-02-13 23:56:02  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59977262-repair/
    2017-02-13 22:25:42  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59976051-provision/
    2017-02-13 21:02:33  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59974796-repair/
    2017-02-13 19:28:09  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59973211-provision/
    2017-02-13 18:13:48  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59971829-repair/
    2017-02-13 16:42:27  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59970425-provision/
    2017-02-13 15:15:27  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59969179-repair/
    2017-02-13 13:43:08  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59968106-provision/
    2017-02-13 13:06:39  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59967817-verify/
    2017-02-13 12:31:24  OK http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59967641-repair/
    2017-02-13 10:58:09  -- http://cautotest/tko/retrieve_logs.cgi?job=/results/hosts/chromeos2-row8-rack9-host14/59966813-provision/
Issue 692161 has been merged into this issue.
This took down several paladin runs. I've locked the DUT in question. Raised to P0. We can't leave the CQ broken for extended periods like this.
Cc: shuqianz@chromium.org
Owner: haoweiw@chromium.org
Summary: chromeos2-row8-rack9-host14 unable to reach devserver (was: Provision failed to fetch client-autotest.tar.bz2 for kevin paladin)
to haowei@ any idea if there's been some network reconfiguration that would cause this?
Blocking: 692179
Comment 14 Deleted
Comment 15 by haoweiw@google.com, Feb 21 2017
Question.

How we name the path in the software? 

If this commend is correct. 
wget --connect-timeout=15 --retry-connrefused --wait=5 -nv http://100.115.245.198:8082/static/kevin-paladin/R58-9280.0.0-rc2/autotest/packages/client-autotest.tar.bz2 -O /tmp/sysinfo/autoserv-4oNwST/packages/client-autotest.tar.bz2'

I don't see the path above in that server. I only can find the package of Kevin-paladin here. 
chromeos-test@chromeos2-devserver6:~/images/kevin-paladin$ pwd
/home/chromeos-test/images/kevin-paladin

And the build number is 
chromeos-test@chromeos2-devserver6:~/images/kevin-paladin$ ls
R58-9304.0.0-rc3
Comment 16 by haoweiw@google.com, Feb 21 2017
I tested with current build:
wget --connect-timeout=15 --retry-connrefused --wait=5 -nv http://100.115.245.198:8082/static/kevin-paladin/R58-9304.0.0-rc3/autotest/packages/client-autotest.tar.bz2 -O /tmp/1.tar.bz2

Looks good to me
Project Member Comment 17 by sheriffbot@chromium.org, Feb 28 2017
Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member Comment 18 by sheriffbot@chromium.org, Mar 15 2017
Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Status: WontFix
I'm going to close this bug due to no action. Please reopen it if issue persists. 
Sign in to add a comment