New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 664226 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug-Regression

Blocking:
issue 677219



Sign in to add a comment

devservers down in cassandra lab

Project Member Reported by shchen@google.com, Nov 10 2016

Issue description

UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36

Example URL:

Steps to reproduce the problem:
provision_FirmwareUpdate autotest is failing

What is the expected behavior?
This test should pass.

What went wrong?
11/10 09:23:09.890 DEBUG|        dev_server:0583| The host chromeos1-row1-rack9-host6 (172.27.212.135) is in a restricted subnet. Try to locate a devserver inside subnet 172.27.212.0:22.
11/10 09:23:09.891 DEBUG|        base_utils:0185| Running 'ssh 172.27.215.252 'curl "http://172.27.215.252:8082/check_health?"''
11/10 09:23:18.982 ERROR|        dev_server:0397| Devserver check_health failed. Free disk space is low. Only 18GB is available.
11/10 09:23:18.984 DEBUG|        base_utils:0185| Running 'ssh 172.27.215.248 'curl "http://172.27.215.248:8082/check_health?"''
11/10 09:24:18.984 ERROR|        dev_server:0313| Devserver call failed: "http://172.27.215.248:8082/check_health?", timeout: 60 seconds, Error: Call is timed out.
11/10 09:24:18.985 DEBUG|        base_utils:0185| Running 'ssh 172.27.215.249 'curl "http://172.27.215.249:8082/check_health?"''
11/10 09:25:18.986 ERROR|        dev_server:0313| Devserver call failed: "http://172.27.215.249:8082/check_health?", timeout: 60 seconds, Error: Call is timed out.
11/10 09:25:18.987 ERROR|        dev_server:0629| All devservers are currently down: set(['http://172.27.215.252:8082', 'http://172.27.215.249:8082', 'http://172.27.215.248:8082']). dut hostname: chromeos1-row1-rack9-host6
11/10 09:25:18.988 ERROR|provision_Firmware:0061| All devservers are currently down: set(['http://172.27.215.252:8082', 'http://172.27.215.249:8082', 'http://172.27.215.248:8082']). dut hostname: chromeos1-row1-rack9-host6

Did this work before? Yes 

Chrome version: 54.0.2840.71  Channel: n/a
OS Version: OS X 10.12.1
Flash Version: Shockwave Flash 23.0 r0
 

Comment 1 by shchen@chromium.org, Nov 10 2016

Cc: waihong@chromium.org dchan@chromium.org
Components: -Internals>Network Infra>Client>ChromeOS
Labels: -Pri-2 -OS-Mac -Via-Wizard-NetworkDownloading faft-in-lab OS-Chrome Pri-1
Owner: dschimmels@chromium.org
logfiles:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/84980320-chromeos-test/chromeos1-row1-rack9-host6/debug

Comment 2 by shchen@chromium.org, Nov 10 2016

Cc: shchen@chromium.org

Comment 3 by dchan@chromium.org, Nov 10 2016

Cc: jashur@chromium.org
Status: Assigned (was: Unconfirmed)

Comment 4 by waihong@google.com, Nov 10 2016

The error message said "Devserver check_health failed. Free disk space is low. Only 18GB is available". Is there any standard procedure to clean up the space? or better do it regularly?

Comment 5 by jashur@chromium.org, Nov 10 2016

Devservers have been rebuilt. For updates, refer to issue: https://bugs.chromium.org/p/chromium/issues/detail?id=664333
Blocking: 677219
Labels: akeshet-pending-downgrade
ChromeOS Infra P1 Bugscrub.

P1 Bugs in this component should be important enough to get weekly status updates.

Is this already fixed?  -> Fixed
Is this no longer relevant? -> Archived or WontFix
Is this not a P1, based on go/chromeos-infra-bug-slo rubric? -> lower priority.
Is this a Feature Request rather than a bug? Type -> Feature
Is this missing important information or scope needed to decide how to proceed? -> Ask question on bug, possibly reassign.
Does this bug have the wrong owner? -> reassign.

Bugs that remain in this state next week will be downgraded to P2.
Labels: -akeshet-pending-downgrade Pri-2
ChromeOS Infra P1 Bugscrub.

Issue untouched in a week after previous message. Downgrading to P2.
Hi, this bug has not been updated recently. Please acknowledge the bug and provide status within two weeks (6/22/2018), or the bug will be archived. Thank you.

Sign in to add a comment