New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 764521 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner: ----
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

AFDO_record fails on chell-chrome-pfq: Devserver fails to resolve DUT hostname

Project Member Reported by cmt...@chromium.org, Sep 12 2017

Issue description

It seems like the fix for https://bugs.chromium.org/p/chromium/issues/detail?id=757604  fixed the problem for about 3-4 days, and now the problem is back again.

If you look at the chell-chrome-pfq builder:
https://uberchromegw.corp.google.com/i/chromeos/builders/chell-chrome-pfq

The last build that had a successful AFDO_record/AFDO_generate stage was build #983 on Sept. 6 (we are now up to build 1011).  Here's the most recent AFDO_record build failure:  

https://uberchromegw.corp.google.com/i/chromeos/builders/chell-chrome-pfq/builds/1011

http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=141470063

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/141470131-chromeos-test/chromeos4-row12-rack10-host1/telemetry_AFDOGenerate/debug


This time it looks like we're getting past the profiler_sync issue and running into some kind of DNS name resolution issue.  Below is a snippet of the debug log referenced above:

09/12 13:04:38.864 DEBUG|              test:0490| Profilers present. Profiling run started
09/12 13:04:38.889 DEBUG|          ssh_host:0296| Running (ssh) 'cat /etc/lsb-release' from 'run_once|get_board|parse_cmd_output|run|wrapper|run_very_slowly'
09/12 13:04:39.232 DEBUG|          ssh_host:0296| Running (ssh) '/opt/google/chrome/chrome --version' from 'run_once|_parse_args|get_chrome_version|run|wrapper|run_very_slowly'
09/12 13:04:39.561 DEBUG|             utils:0280| [stdout] Google Chrome 63.0.3213.0 unknown
09/12 13:04:39.608 DEBUG|  telemetry_runner:0151| Setting up telemetry for devserver testing
09/12 13:04:39.609 DEBUG|  telemetry_runner:0152| Grabbing build from AFE.
09/12 13:04:39.611 DEBUG|  telemetry_runner:0160| Setting up telemetry for build: chell-chrome-pfq/R63-9935.0.0-rc1
09/12 13:04:39.612 INFO |        dev_server:0706| Getting devservers for host: chromeos4-row12-rack10-host1
09/12 13:04:39.614 DEBUG|        dev_server:0727| The host chromeos4-row12-rack10-host1 (100.115.203.93) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.192.0:19.
09/12 13:04:39.617 DEBUG|        dev_server:0678| Pick one healthy devserver from ['http://100.115.219.129:8082', 'http://100.115.219.130:8082', 'http://100.115.219.132:8082', 'http://100.115.219.133:8082', 'http://100.115.219.134:8082', 'http://100.115.219.135:8082', 'http://100.115.219.136:8082', 'http://100.115.219.139:8082']
09/12 13:04:39.618 DEBUG|        dev_server:0682| Check health for http://100.115.219.136:8082
09/12 13:04:39.619 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/check_health?"''
09/12 13:04:40.432 DEBUG|        dev_server:0687| Pick http://100.115.219.136:8082
09/12 13:04:40.435 INFO |        dev_server:1125| Staging artifacts on devserver http://100.115.219.136:8082: build=chell-chrome-pfq/R63-9935.0.0-rc1, artifacts=['autotest_packages'], files=, archive_url=gs://chromeos-image-archive/chell-chrome-pfq/R63-9935.0.0-rc1
09/12 13:04:40.437 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/stage?artifacts=autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/chell-chrome-pfq/R63-9935.0.0-rc1"''
09/12 13:04:41.120 DEBUG|        dev_server:1067| response for RPC: 'Success'
09/12 13:04:41.123 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/is_staged?artifacts=autotest_packages&files=&archive_url=gs://chromeos-image-archive/chell-chrome-pfq/R63-9935.0.0-rc1"''
09/12 13:04:41.745 DEBUG|        dev_server:1023| whether artifact is staged: 'True'
09/12 13:04:41.748 INFO |        dev_server:1143| Finished staging artifacts: build=chell-chrome-pfq/R63-9935.0.0-rc1, artifacts=['autotest_packages'], files=, archive_url=gs://chromeos-image-archive/chell-chrome-pfq/R63-9935.0.0-rc1
09/12 13:04:41.751 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 'curl "http://100.115.219.136:8082/setup_telemetry?archive_url=gs://chromeos-image-archive/chell-chrome-pfq/R63-9935.0.0-rc1"''
09/12 13:04:50.581 DEBUG|  telemetry_runner:0146| Telemetry Path: /home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src
09/12 13:04:50.582 INFO |telemetry_AFDOGene:0195| Starting run for Telemetry benchmark page_cycler_v2.typical_25
09/12 13:04:50.583 DEBUG|  telemetry_runner:0381| Running telemetry benchmark: page_cycler_v2.typical_25
09/12 13:04:50.584 DEBUG|  telemetry_runner:0313| Running Telemetry: ssh 100.115.219.136 python /home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src/tools/perf/run_benchmark --verbose --browser=cros-chrome --output-format=chartjson --output-dir=/home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src --remote=chromeos4-row12-rack10-host1 --pageset-repeat=1 page_cycler_v2.typical_25
09/12 13:04:50.584 DEBUG|  telemetry_runner:0274| Running: ssh 100.115.219.136 python /home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src/tools/perf/run_benchmark --verbose --browser=cros-chrome --output-format=chartjson --output-dir=/home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src --remote=chromeos4-row12-rack10-host1 --pageset-repeat=1 page_cycler_v2.typical_25
09/12 13:04:50.585 DEBUG|             utils:0212| Running 'ssh 100.115.219.136 python /home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src/tools/perf/run_benchmark --verbose --browser=cros-chrome --output-format=chartjson --output-dir=/home/chromeos-test/images/chell-chrome-pfq/R63-9935.0.0-rc1/telemetry_src/src --remote=chromeos4-row12-rack10-host1 --pageset-repeat=1 page_cycler_v2.typical_25'
09/12 13:04:55.898 DEBUG|  telemetry_runner:0285| Error occurred executing.
09/12 13:04:55.900 DEBUG|  telemetry_runner:0291| Completed with exit code: 1.
stdout:
stderr:(ERROR) 2017-09-12 13:04:55,811 platform.GetPlatformForDevice:69  Fail to create platform instance for ChromeOs with host chromeos4-row12-rack10-host1.
(INFO) 2017-09-12 13:04:55,811 exception_formatter.PrintFormattedException:19  Try printing formatted exception: <class 'telemetry.internal.browser.browser_finder_exceptions.BrowserFinderException'> Unable to resolve the hostname for: chromeos4-row12-rack10-host1 <traceback object at 0x7f281a94a320>


 
Cc: -pprabhu@chromium.org shuqianz@chromium.org
Summary: AFDO_record fails on chell-chrome-pfq: Devserver fails to resolve DUT hostname (was: AFDO_record is failing on chell-chrome-pfq [ again! ])
+ deputy
- me

We've indeed seen some devserver DNS issues lately (e.g. issue 757943)
Mergedinto: 712682
Status: Duplicate (was: Untriaged)

Sign in to add a comment