New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 881562 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Sep 7
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

test_push failing due to software version desync

Project Member Reported by pprabhu@chromium.org, Sep 6

Issue description

2018-09-06 13:09:34,355 INFO| Verifying all the servers on the same HEAD for autotest

[chromeos-staging-master2.hot.corp.google.com] run: git rev-parse HEAD
[chromeos-staging-master2.hot.corp.google.com] out: b8aed267cb460e099e3268f8288cbf7041c1a4bb
[chromeos-staging-master2.hot.corp.google.com] out: 

[chromeos-staging-shard2.cbf.corp.google.com] run: git rev-parse HEAD
[chromeos-staging-shard2.cbf.corp.google.com] out: b8aed267cb460e099e3268f8288cbf7041c1a4bb
[chromeos-staging-shard2.cbf.corp.google.com] out: 

[chromeos-staging-drone2.hot.corp.google.com] run: git rev-parse HEAD
[chromeos-staging-drone2.hot.corp.google.com] out: b8aed267cb460e099e3268f8288cbf7041c1a4bb
[chromeos-staging-drone2.hot.corp.google.com] out: 
2018-09-06 13:09:34,641 INFO| Verifying all the servers on the same HEAD for chromite

[chromeos-staging-master2.hot.corp.google.com] run: git rev-parse HEAD
[chromeos-staging-master2.hot.corp.google.com] out: 23d65fd54de1bdbf5db29eb9d8a2944b3bbb6daa
[chromeos-staging-master2.hot.corp.google.com] out: 

[chromeos-staging-shard2.cbf.corp.google.com] run: git rev-parse HEAD
[chromeos-staging-shard2.cbf.corp.google.com] out: 23d65fd54de1bdbf5db29eb9d8a2944b3bbb6daa
[chromeos-staging-shard2.cbf.corp.google.com] out: 

[chromeos-staging-drone2.hot.corp.google.com] run: git rev-parse HEAD
[chromeos-staging-drone2.hot.corp.google.com] out: 5389d34c833dae162248c0c8ddb1b71d31851924
[chromeos-staging-drone2.hot.corp.google.com] out: 
2018-09-06 13:09:35,091 ERRO| Traceback (most recent call last):
  File "/usr/local/google/home/chromeos-test/chromiumos/chromeos-admin/venv/server_management_lib/tasks/task.py", line 66, in run
    self._run()
  File "/usr/local/google/home/chromeos-test/chromiumos/chromeos-admin/venv/server_management_lib/tasks/staging_lab_task.py", line 503, in _run
    good_hashes = self.verify_all_repos_synced_on_servers()
  File "/usr/local/google/home/chromeos-test/chromiumos/chromeos-admin/venv/server_management_lib/tasks/staging_lab_task.py", line 382, in verify_all_repos_synced_on_servers
    commit_map[repo] = self.verify_repo_synced(repo, commit_map.get(repo))
  File "/usr/local/google/home/chromeos-test/chromiumos/chromeos-admin/venv/server_management_lib/tasks/staging_lab_task.py", line 413, in verify_repo_synced
    '\nActual:   %s' % (server, repo, commit, head))
PushServerInconsistentException: chromeos-staging-drone2.hot.corp.google.com is on a different chromite hash from other servers.
Expected: 23d65fd54de1bdbf5db29eb9d8a2944b3bbb6daa
Actual:   5389d34c833dae162248c0c8ddb1b71d31851924

 
I manually ran /usr/local/autotest/utils/build_externals.py --use_chromite_master on chromeos-staging-drone2.hot and verified that a following puppet run does not push the /usr/local/autotest/site-packages/chromite git # back.

The problem is that build_externals isn't run as part of deploy_server_local on drones at all.

This means that all our drones likely have a very dated chromite.
Status: Started (was: Assigned)
main CL: https://chrome-internal-review.googlesource.com/c/chromeos/chromeos-admin/+/674430

There are other cleanup-y CLs.
Labels: -Pri-2 Pri-1
Blocks stagin --> Pri-1
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 6

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/48d67fe053882ff40f5f1a1046b0d5121a1fdf02

commit 48d67fe053882ff40f5f1a1046b0d5121a1fdf02
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Sep 06 23:21:28 2018

Project Member

Comment 5 by bugdroid1@chromium.org, Sep 7

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/7781d8d9ad69f911881f99802823eafb887db528

commit 7781d8d9ad69f911881f99802823eafb887db528
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Fri Sep 07 03:48:57 2018

Status: Fixed (was: Started)
Got green staging runs last night.
Project Member

Comment 7 by bugdroid1@chromium.org, Sep 8

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/f2470a59d4e9b4e2ef4272d5be59958e09657e27

commit f2470a59d4e9b4e2ef4272d5be59958e09657e27
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Sat Sep 08 16:59:55 2018

Sign in to add a comment