Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Starred by 1 user
Status: Archived
Owner:
Closed: Feb 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment
All nyan_kitty DUTs in pool:cq failed around 4pm
Project Member Reported by shuqianz@chromium.org, Feb 17 2017 Back to list
All nyan_kitty DUTs in pool:cq failed around 4pm, which failed the HWTest on that CQ run. 

https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fnyan_kitty-paladin%2F760%2F%2B%2Frecipes%2Fsteps%2FHWTest__bvt-cq_%2F0%2Fstdout

log:
...
host: chromeos4-row13-rack8-host7, status: Ready, locked: False diagnosis: Working
labels: ['gpu_family:tegra', 'nyan_kitty', 'board_freq_mem:nyan_kitty_2.1GHz_4GB', 'graphics:gles', '4k_video', 'bluetooth', 'power:AC_only', 'ec:cros', 'audio_loopback_dongle', 'board:nyan_kitty', 'cts_abi_arm', 'internal_display', 'os:cros', 'storage:mmc', 'webcam', 'servo', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'hw_video_acc_enc_vp8', 'pool:cq', 'variant:kitty', 'sku:kitty_cpu_nyan_4Gb', 'phase:PVT']
Last 10 jobs within 1:48:00:
2537763 Repair started on: 2017-02-16 17:44:15 status PASS
2537692 Provision started on: 2017-02-16 17:28:46 status FAIL
2537687 Repair started on: 2017-02-16 17:27:55 status PASS
2537627 Provision started on: 2017-02-16 17:10:14 status FAIL
2537600 Repair started on: 2017-02-16 17:09:15 status PASS
2537334 Provision started on: 2017-02-16 16:29:49 status FAIL
2537321 Repair started on: 2017-02-16 16:28:44 status PASS
2537211 Provision started on: 2017-02-16 16:11:35 status FAIL

host: chromeos4-row13-rack8-host9, status: Repairing, locked: False diagnosis: Working
labels: ['bluetooth', 'power:AC_only', 'gpu_family:tegra', 'ec:cros', 'nyan_kitty', 'board:nyan_kitty', 'board_freq_mem:nyan_kitty_2.1GHz_4GB', 'graphics:gles', '4k_video', 'audio_loopback_dongle', 'cts_abi_arm', 'internal_display', 'os:cros', 'storage:mmc', 'webcam', 'servo', 'variant:kitty', 'sku:kitty_cpu_nyan_4Gb', 'phase:PVT', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'hw_video_acc_enc_vp8', 'pool:cq']
Last 10 jobs within 1:48:00:
2537691 Provision started on: 2017-02-16 17:28:36 status FAIL
2537686 Repair started on: 2017-02-16 17:27:46 status PASS
2537624 Provision started on: 2017-02-16 17:09:58 status FAIL
2537598 Repair started on: 2017-02-16 17:09:00 status PASS
2537340 Provision started on: 2017-02-16 16:29:54 status FAIL
2537325 Repair started on: 2017-02-16 16:28:57 status PASS
2537210 Provision started on: 2017-02-16 16:11:35 status FAIL

host: chromeos4-row13-rack8-host10, status: Ready, locked: False diagnosis: Working
labels: ['bluetooth', 'power:AC_only', 'gpu_family:tegra', 'nyan_kitty', 'board:nyan_kitty', 'board_freq_mem:nyan_kitty_2.1GHz_4GB', 'graphics:gles', '4k_video', 'audio_loopback_dongle', 'variant:kitty', 'sku:kitty_cpu_nyan_4Gb', 'cts_abi_arm', 'internal_display', 'os:cros', 'storage:mmc', 'webcam', 'phase:PVT', 'servo', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'hw_video_acc_enc_h264', 'hw_video_acc_enc_vp8', 'pool:cq', 'ec:cros', 'cros-version:nyan_kitty-release/R56-9000.82.0']
Last 10 jobs within 1:48:00:
2537725 Repair started on: 2017-02-16 17:35:17 status PASS
2537717 Verify started on: 2017-02-16 17:34:30 status FAIL
2537473 Repair started on: 2017-02-16 16:51:09 status FAIL
2537332 Provision started on: 2017-02-16 16:29:28 status FAIL
2537312 Repair started on: 2017-02-16 16:28:15 status PASS
2537215 Provision started on: 2017-02-16 16:11:35 status FAIL
....

I checked one of the broken DUT: chromeos4-row13-rack8-host11, and find in the following error in one of the Provisioning job:

		END GOOD	----	----	timestamp=1487292749	localtime=Feb 16 16:52:29	
		FAIL	provision_AutoUpdate	provision_AutoUpdate	timestamp=1487293748	localtime=Feb 16 17:09:08	Unhandled DevServerException: CrOS auto-update failed for host chromeos4-row13-rack8-host11: AttributeError: 'ChromiumOSUpdater' object has no attribute 'STATEFUL_FOLDER_TO_CHECK'
  Traceback (most recent call last):
    File "/usr/local/autotest/client/common_lib/test.py", line 804, in _call_test_function
      return func(*args, **dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 461, in execute
      dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
      postprocess_profiled_run, args, dargs)
    File "/usr/local/autotest/client/common_lib/test.py", line 376, in _call_run_once
      self.run_once(*args, **dargs)
    File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once
      force_full_update=force)
    File "/usr/local/autotest/server/afe_utils.py", line 232, in machine_install_and_update_labels
      *args, **dargs)
    File "/usr/local/autotest/server/hosts/cros_host.py", line 728, in machine_install_by_devserver
      full_update=force_full_update)
    File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 2013, in auto_update
      raise DevServerException(error_msg % (host_name, error_list[0]))
  DevServerException: CrOS auto-update failed for host chromeos4-row13-rack8-host11: AttributeError: 'ChromiumOSUpdater' object has no attribute 'STATEFUL_FOLDER_TO_CHECK'
	END FAIL	provision_AutoUpdate	provision_AutoUpdate	timestamp=1487293748	localtime=Feb 16 17:09:08	
END FAIL	----	provision	timestamp=1487293748	localtime=Feb 16 17:09:08	

Link for the log:https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos4-row13-rack8-host11/2537696-provision/20171602172909/debug/
 
Project Member Comment 1 by bugdroid1@chromium.org, Feb 17 2017
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/e365d309a43e98579039fe60961a8083ec576314

commit e365d309a43e98579039fe60961a8083ec576314
Author: xixuan <xixuan@chromium.org>
Date: Fri Feb 17 03:03:53 2017

auto_updater: emergency fix for name inconsiscency caused by git rebase.

BUG= chromium:693331 
TEST=None

Change-Id: I2288f74a714113975843e8971e7f11e4caa68e1b
Reviewed-on: https://chromium-review.googlesource.com/444217
Commit-Queue: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/e365d309a43e98579039fe60961a8083ec576314/lib/auto_updater.py

Comment 2 by xixuan@chromium.org, Feb 17 2017
This is caused by I have two ready CL https://chromium-review.googlesource.com/c/414429/ & https://chromium-review.googlesource.com/c/415429. At the meantime, I made some name changes due to comments and first submit cl:414429. Then after a time I do a rebase of cl:415429, and this rebase doesn't change all of changed names due to the fact that there're some contents both include those changed names and other changes.

Fix has been in. I have tested it locally. Now it can successfully provision my local DUT.
Comment 3 by xixuan@chromium.org, Feb 27 2017
Status: Fixed
Labels: VerifyIn-59
Labels: VerifyIn-60
Labels: VerifyIn-61
Comment 7 by dchan@chromium.org, Oct 14 (4 days ago)
Status: Archived
Sign in to add a comment