New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 814472 link

Starred by 1 user
Status: WontFix
Owner:
ayatane@chromium.org
Closed: Mar 2018
Cc:
akes...@chromium.org
█ shuqianz@chromium.org
davidri...@chromium.org
xixuan@chromium.org
dgarr...@chromium.org
█ jrbarnette@chromium.org
█ lunarobert@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Failed to push 10 out of 49 devservers

Project Member Reported by nxia@chromium.org, Feb 21 2018 
 {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos1-dev-infra-devserver1.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos1-dev-infra-devserver1.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos1-dev-infra-devserver1.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos1-dev-infra-devserver.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos1-dev-infra-devserver.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos1-dev-infra-devserver.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos9-infra-devserver4.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos9-infra-devserver4.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos9-infra-devserver4.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos9-infra-devserver6.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos9-infra-devserver6.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos9-infra-devserver6.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos9-infra-devserver5.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos9-infra-devserver5.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos9-infra-devserver5.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos2-devserver8.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos2-devserver8.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos2-devserver8.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos1-infra-devserver.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos1-infra-devserver.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos1-infra-devserver.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos11-infra-devserver2.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos11-infra-devserver2.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos11-infra-devserver2.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "NetworkError()",
      "is_successful": false,
      "description": "Failed to push chromeos11-infra-devserver1.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos11-infra-devserver1.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos11-infra-devserver1.cros.corp.google.com)"
    },
    {
      "sub_reports": [],
      "exception": "FabricException('run() received nonzero return code 128 while executing!\\n\\nRequested: repo forall -vc \"git reset --hard\"\\nExecuted: /bin/bash -l -c \"cd ~/chromiumos >/dev/null && repo forall -vc \\\\\"git reset --hard\\\\\"\"',)",
      "is_successful": false,
      "description": "Failed to push chromeos15-infra-devserver3.cros.corp.google.com",
      "arguments_used": {
        "devserver": "chromeos15-infra-devserver3.cros.corp.google.com"
      },
      "task_name": "DevserverPushOneTask (chromeos15-infra-devserver3.cros.corp.google.com)"
    }
  ],
  "exception": null,
  "is_successful": false,
  "description": "Failed to push 10 out of 49 devservers",
  "arguments_used": {},
  "task_name": "DevserverPushAllTask"
}

Comment 1 by nxia@chromium.org, Feb 21 2018 

The first couple failed devservers are not reachable.
The last failed devserver failed because it's in "read-only file system":

chromeos-test@chromeos-15-infra-devserver3:~/chromiumos$ repo forall -vc "git reset --hard"
fatal: Unable to create '/home/chromeos-test/chromiumos/chromite/.git/index.lock': Read-only file system
fatal: Unable to create '/home/chromeos-test/chromiumos/infra_virtualenv/.git/index.lock': Read-only file system
fatal: Unable to create '/home/chromeos-test/chromiumos/src/aosp/system/update_engine/.git/index.lock': Read-only file system
fatal: Unable to create '/home/chromeos-test/chromiumos/src/platform/dev/.git/index.lock': Read-only file system
fatal: Unable to create '/home/chromeos-test/chromiumos/src/third_party/autotest/files/.git/index.lock': Read-only file system

Comment 2 by nxia@chromium.org, Feb 21 2018 

Filed b/73736072 and b/73736076

Comment 3 by dgarr...@chromium.org, Feb 21 2018 

Unless I'm mistaken, only one of those (chromeos2-devserver8.cros.corp.google.com) is in a lab owned by our team.

IE: We only own the even numbered labs... right?

Comment 4 by nxia@chromium.org, Feb 21 2018 

Re#3,I don't know. how do we restart the servers owned by ourselves? I didn't find that one in portal.

Comment 5 by dgarr...@chromium.org, Feb 21 2018 

ssh in and reboot them, or file bugs to syslab if hands on is needed.

Comment 6 by nxia@chromium.org, Feb 21 2018 

they're not reachable, so I can't ssh into them

Comment 7 by dgarr...@chromium.org, Feb 21 2018 

Exactly.

Comment 8 by davidri...@chromium.org, Feb 22 2018

Cc: davidri...@chromium.org
I don't know what's up with those other devservers because they result in a huge amount of flake in the provision stats as well.  What are they?

Comment 9 by nxia@chromium.org, Feb 22 2018

Cc: akes...@chromium.org jrbarnette@chromium.org shuqianz@chromium.org
No idea, and I don't know which lab to file the ticket for chromeos15-infra-devserver3.cros.corp.google.com.

+ more people.

Comment 10 Deleted

Comment 12 by akes...@chromium.org, Feb 23 2018 

anything from the odd-numbered chromeos prefixes are from sattelite labs, ownership of which are documented at https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/hardware-labs

Comment 13 by nxia@chromium.org, Feb 23 2018

Cc: lunarobert@chromium.org
lunarobert@ confirmed they don't use chromeos9-* devservers anymore, can we remove them from the server list ?

Comment 14 by akes...@chromium.org, Feb 23 2018 

yes, please delete them from both serverdb and from puppet.

Can we also delete all the chromeos9 DUTs?

Comment 15 by davidri...@chromium.org, Feb 23 2018 

Can chromeos9-test-station be used as a cyan dut?  The lab is lacking cyan's.

Comment 16 by lunarobert@google.com, Feb 23 2018 

Lab team will migrate cros9-test-station to cros15.

Comment 17 by jrbarnette@chromium.org, Feb 23 2018 

> Can chromeos9-test-station be used as a cyan dut?  The lab is lacking cyan's.

No.  For DUTs to be usable for general testing, they must live
in chromeos[246], so that we have physical access if it needs
fixing or debug.

Comment 18 by davidri...@chromium.org, Feb 23 2018 

I mean, can we move it over to the other lab once it's turned down and not being used?

Comment 19 by jrbarnette@chromium.org, Feb 23 2018 

> I mean, can we move it over to the other lab once it's turned down and not being used?

Ah.  Yes and no.

Roughly speaking, if the problem is we need more cyan DUTs, then
the process is "ask cros-distros for more".  When a DUT is
decommissioned, the proper process is to return it to cros-distros.

So, in principle, if that DUT is properly returned to cros-distros,
the supply available to be given to the lab should go up by 1.

Comment 20 by nxia@chromium.org, Mar 3 2018

Owner: pprabhu@chromium.org
how was the latest devserver push ?

Comment 21 by nxia@chromium.org, Mar 3 2018 

pprabhu@, please update the list if any devservers are still unreachable in you last devserver push

Comment 22 by ayatane@chromium.org, Mar 5 2018

Owner: ayatane@chromium.org
Status: Assigned (was: Untriaged)

Comment 23 by ayatane@chromium.org, Mar 5 2018 

chromeos1-dev-infra-devserver1.cros.corp.google.com is unreachable during my push

Comment 24 by ayatane@chromium.org, Mar 19 2018

Cc: xixuan@chromium.org
Status: WontFix (was: Assigned)
I remember my push being rather uneventful, however it seems like it is common for a small number of devservers to be stubborn during push, requiring a long time or a re-push.

But there's no actionable item for this currently.
►

Sign in to add a comment