New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 620850 link

Starred by 1 user
Status: Verified
Owner:
xixuan@chromium.org
Closed: Jun 2016
Cc:
█ dshi@chromium.org
█ bmahadev@chromium.org
█ dschimmels@chromium.org
█ krisr@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

[Chaos] Staging artifacts timeouts

Project Member Reported by tienchang@chromium.org, Jun 16 2016 
Chaos (chromeos3) is under the new ACL. 

Devserver seems unable to stage artifacts and continuously retries until it timeouts.

06/15 15:17:16.392 DEBUG|provision_AutoUpda:0091| Start provisioning <remote host: chromeos3-row1-rack2-host16> to falco_li-release/R53-8436.0.0
06/15 15:17:16.789 DEBUG|        dev_server:0554| The host chromeos3-row1-rack2-host16 (172.22.38.62) is in a restricted subnet. Try to locate a devserver inside subnet 172.22.38.0:23.
06/15 15:17:16.791 DEBUG|        base_utils:0185| Running 'ssh 172.22.39.164 'curl "http://172.22.39.164:8082/check_health?"''
06/15 15:17:17.509 INFO |        dev_server:0931| Staging artifacts on devserver http://172.22.39.164:8082: build=falco_li-release/R53-8436.0.0, artifacts=['full_payload', 'stateful', 'autotest_packages'], files=, archive_url=gs://chromeos-image-archive/falco_li-release/R53-8436.0.0
06/15 15:17:17.511 DEBUG|        base_utils:0185| Running 'ssh 172.22.39.164 'curl "http://172.22.39.164:8082/stage?artifacts=full_payload,stateful,autotest_packages&files=&async=True&archive_url=gs://chromeos-image-archive/falco_li-release/R53-8436.0.0"''
06/15 15:17:18.079 DEBUG|        base_utils:0185| Running 'ssh 172.22.39.164 'curl "http://172.22.39.164:8082/is_staged?artifacts=full_payload,stateful,autotest_packages&files=&archive_url=gs://chromeos-image-archive/falco_li-release/R53-8436.0.0"''

Job here: http://cautotest/afe/#tab_id=view_job&object_id=66818338

Comment 1 by xixuan@chromium.org, Jun 16 2016 

seems it's the same question with https://bugs.chromium.org/p/chromium/issues/detail?id=619711. The package is not downloaded successfully from google storage.

@dshi has landed a new CL https://chrome-internal-review.googlesource.com/#/c/265032/ for logging the contents list in the image storage directory, It helps to log when the call is timedout. Could you update the autotest to get the latest changes?

Comment 2 by tienchang@chromium.org, Jun 16 2016 

Oh that CL seems to link to removing a devserver?

By "update the autotest", do you mean running with the latest autotest bundle?

Comment 3 by xixuan@chromium.org, Jun 16 2016 

oh, I'm not sure whether you kick off some tests by yourself. If the tests go through drone/shard, the CL would be landed now.

The CL just log whether the files for auto-update have been downloaded. If the call timeouts error happens again, we can see whether it's because of the devserver, or the connection to google storage, or the files are too big to download in time.

Comment 4 by tienchang@chromium.org, Jun 16 2016 

Perfect, OK! I can launch a short job (with test network_WiFi_ChaosConnectDisconnect.debug) through AFE on the latest ToT to try to produce logs results. The 

Note: the time for retries to stage artifacts is 30 minutes, so the test will be at least that long.

Comment 5 by tienchang@chromium.org, Jun 29 2016

Status: Verified (was: Assigned)
This error does not occur anymore, so I will close this bug as Verified.

However, there is a failure with wget commands. I will open a new bug.
►

Sign in to add a comment