New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 771823 link

Starred by 1 user
Status: Assigned
Owner: ----
Cc:
█ dshi@chromium.org
akes...@chromium.org
█ sbasi@chromium.org
█ kevcheng@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

Sometimes DUT repair "verify.update" stage spews tons of "ERROR Unable to search directory <path> for control files." messages

Project Member Reported by djkurtz@chromium.org, Oct 5 2017 


For example:

https://storage.cloud.google.com/chromeos-autotest-results/hosts/chromeos6-row2-rack20-host20/743908-verify/20170410151603/debug/autoserv.DEBUG?_ga=2.203803862.-734044362.1501703718



10/04 15:16:10.236 INFO |            repair:0327| Verifying this condition: servo host software is up-to-date
10/04 15:16:10.244 DEBUG|          ssh_host:0296| Running (ssh) 'grep -q CHROMEOS /etc/lsb-release' from 'verify|update_image|is_cros_host|run|run|run_very_slowly'
10/04 15:16:10.639 DEBUG|          ssh_host:0296| Running (ssh) 'cat /etc/lsb-release' from 'verify|update_image|get_board|run|run|run_very_slowly'
10/04 15:16:10.996 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_APPID={41D57E57-2150-BB76-2730-EC8AFD1D835D}
10/04 15:16:10.996 DEBUG|             utils:0280| [stdout] CHROMEOS_BOARD_APPID={41D57E57-2150-BB76-2730-EC8AFD1D835D}
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] DEVICETYPE=CHROMEBOX
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BUILDER_PATH=guado_labstation-release/R61-9672.0.0
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] GOOGLE_RELEASE=9672.0.0
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] CHROMEOS_DEVSERVER=
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BOARD=guado_labstation
10/04 15:16:10.997 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BUILD_NUMBER=9672
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BRANCH_NUMBER=0
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_CHROME_MILESTONE=61
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_PATCH_NUMBER=0
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_TRACK=testimage-channel
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_DESCRIPTION=9672.0.0 (Official Build) dev-channel guado_labstation test
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BUILD_TYPE=Official Build
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_NAME=Chrome OS
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_VERSION=9672.0.0
10/04 15:16:10.998 DEBUG|             utils:0280| [stdout] CHROMEOS_AUSERVER=https://tools.google.com/service/update2
10/04 15:16:11.095 INFO |        dev_server:0707| Getting devservers for host: chromeos6-row2-rack20-labstation2
10/04 15:16:11.134 DEBUG|        dev_server:0728| The host chromeos6-row2-rack20-labstation2 (100.115.172.156) is in a restricted subnet. Try to locate a devserver inside subnet 100.115.128.0:18.
10/04 15:16:11.135 DEBUG|        dev_server:0679| Pick one healthy devserver from ['http://100.115.185.226:8082', 'http://100.115.185.227:8082', 'http://100.115.185.228:8082']
10/04 15:16:11.135 DEBUG|        dev_server:0683| Check health for http://100.115.185.226:8082
10/04 15:16:11.135 DEBUG|             utils:0212| Running 'ssh 100.115.185.226 'curl "http://100.115.185.226:8082/check_health?"''
10/04 15:16:12.193 DEBUG|        dev_server:0688| Pick http://100.115.185.226:8082
10/04 15:16:12.202 DEBUG|          ssh_host:0296| Running (ssh) 'cat "/etc/lsb-release"' from 'update_image|_check_for_reboot|_get_release_version|run|run|run_very_slowly'
10/04 15:16:12.577 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_APPID={41D57E57-2150-BB76-2730-EC8AFD1D835D}
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_BOARD_APPID={41D57E57-2150-BB76-2730-EC8AFD1D835D}
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] DEVICETYPE=CHROMEBOX
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BUILDER_PATH=guado_labstation-release/R61-9672.0.0
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] GOOGLE_RELEASE=9672.0.0
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_DEVSERVER=
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BOARD=guado_labstation
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BUILD_NUMBER=9672
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BRANCH_NUMBER=0
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_CHROME_MILESTONE=61
10/04 15:16:12.578 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_PATCH_NUMBER=0
10/04 15:16:12.579 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_TRACK=testimage-channel
10/04 15:16:12.579 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_DESCRIPTION=9672.0.0 (Official Build) dev-channel guado_labstation test
10/04 15:16:12.579 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_BUILD_TYPE=Official Build
10/04 15:16:12.579 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_NAME=Chrome OS
10/04 15:16:12.579 DEBUG|             utils:0280| [stdout] CHROMEOS_RELEASE_VERSION=9672.0.0
10/04 15:16:12.579 DEBUG|             utils:0280| [stdout] CHROMEOS_AUSERVER=https://tools.google.com/service/update2
10/04 15:16:12.589 DEBUG|          ssh_host:0296| Running (ssh) '/usr/bin/update_engine_client -status | grep CURRENT_OP' from 'update_image|_check_for_reboot|check_update_status|run|run|run_very_slowly'
10/04 15:16:12.977 ERROR|             utils:0280| [stderr] [1004/151612:INFO:update_engine_client.cc(493)] Querying Update Engine status...
10/04 15:16:12.980 DEBUG|             utils:0280| [stdout] CURRENT_OP=UPDATE_STATUS_UPDATED_NEED_REBOOT
10/04 15:16:13.173 INFO |        servo_host:0458| servo host has the following duts: ['chromeos6-row2-rack20-host4', 'chromeos6-row2-rack20-host2', 'chromeos6-row2-rack20-host10', 'chromeos6-row2-rack20-host8', 'chromeos6-row2-rack20-host14', 'chromeos6-row2-rack20-host12', 'chromeos6-row2-rack20-host6', 'chromeos6-row2-rack20-host16', 'chromeos6-row2-rack20-host20', 'chromeos6-row2-rack20-host22', 'chromeos6-row2-rack20-host18']
10/04 15:16:16.777 ERROR|control_file_gette:0214| Unable to search directory /usr/local/autotest/site-packages/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/.git/objects/49 for control files.
10/04 15:16:17.032 ERROR|control_file_gette:0214| Unable to search directory /usr/local/autotest/site-packages/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/venv/chromite/.git/objects/3b for control files.

...

1 hour and 9,330 "Unable to search..." lines later...

...

10/04 16:20:00.644 ERROR|control_file_gette:0214| Unable to search directory /usr/local/autotest/containers/test_146476177_1506979022_4331 for control files.
10/04 16:20:00.644 ERROR|control_file_gette:0214| Unable to search directory /usr/local/autotest/containers/test_146396008_1506978668_9413 for control files.
10/04 16:20:01.125 ERROR|            repair:0332| Failed: servo host software is up-to-date
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 329, in _verify_host
    self.verify(host)
  File "/usr/local/autotest/server/hosts/servo_repair.py", line 28, in verify
    host.update_image(wait_for_update=False)
  File "/usr/local/autotest/server/hosts/servo_host.py", line 550, in update_image
    status, current_build_number = self._check_for_reboot(updater)
  File "/usr/local/autotest/server/hosts/servo_host.py", line 462, in _check_for_reboot
    self.schedule_synchronized_reboot(dut_list, afe)
  File "/usr/local/autotest/server/hosts/servo_host.py", line 421, in schedule_synchronized_reboot
    control_file = getter.get_control_file_contents_by_name(test)
  File "/usr/local/autotest/server/cros/dynamic_suite/control_file_getter.py", line 152, in get_control_file_contents_by_name
    path = self.get_control_file_path(test_name)
  File "/usr/local/autotest/server/cros/dynamic_suite/control_file_getter.py", line 136, in get_control_file_path
    raise error.ControlFileNotFound(test_name + ' is not unique.')
ControlFileNotFound: servohost_Reboot is not unique.
10/04 16:20:01.126 INFO |        server_job:0214| 	FAIL	----	verify.update	timestamp=1507159201	localtime=Oct 04 16:20:01	servohost_Reboot is not unique.

Comment 1 by ayatane@chromium.org, Oct 5 2017

Status: Started (was: Available)

Comment 2 by kevcheng@chromium.org, Oct 5 2017 

I have a fix for the "ControlFileNotFound" error here: crosreview.com/435478

But I've held off on submitting because I wanted to submit crosreview.com/435480 first.

I'll revisit and get those in asap.
Project Member

Comment 3 by bugdroid1@chromium.org, Oct 6 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/39251952eaaafe1a01c27cbcf5a04efc32f4cb44

commit 39251952eaaafe1a01c27cbcf5a04efc32f4cb44
Author: Allen Li <ayatane@chromium.org>
Date: Fri Oct 06 04:24:47 2017

[autotest] Do not recurse into site-packages

Beside being a waste of time, this recurses infinitely due to the
venv symlink in chromite.

BUG=chromium:771823
TEST=None

Change-Id: I325d4827a3f0f36320eba90b4d3d37eddf6e2110
Reviewed-on: https://chromium-review.googlesource.com/701915
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/39251952eaaafe1a01c27cbcf5a04efc32f4cb44/server/cros/dynamic_suite/control_file_getter.py

Comment 4 by ayatane@chromium.org, Oct 6 2017

Owner: kevcheng@chromium.org
Status: Assigned (was: Started)
I fixed the recursion issue, but I think the ControlFileNotFound is an unrelated error.

Comment 5 by kevcheng@chromium.org, Apr 9 2018

Owner: akes...@chromium.org
Bulk-edit, reassigning to Aviv to assign to someone.

Comment 6 by akes...@chromium.org, Apr 19 2018

Components: Infra>Client>ChromeOS>Test
Owner: ----
Gathered onto a servo v4 hotlist, removing self as owner.
►

Sign in to add a comment