New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 852028 link

Starred by 2 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

test_push: Should bootstrap staging-master repos before using them

Project Member Reported by pprabhu@chromium.org, Jun 12 2018

Issue description

staging test uses the /root/chromeos-admin, autotest  and ~/chromiumos checkouts for the sml task for test_push itself.
[1] To run test_push (out of /root/chromeos-admin).
[2] To obtain the list of servers to update
[3] To update all push servers (via deploy_server)

sml takes no pain to ensure that updated version of these repos are used.

For example, because [1] uses old code, any changes to the test_push task itself will not automatically get picked up in test_push. If this makes test_push fail before it can force a puppet run on the staging master (that updates /root/chromeos-admin checkout), we're stuck until hopefully the puppet cron job updates this checkout.

For example, because [2] uses old code, the fix for  issue 851538  was never automatically picked up. The list of servers to be updated was wrong, and so test_push failed early enough that it never updated the staging master itself.

For example, because [3] uses old code, the fix for issue 850118 was never automatically picked up. deploy_server kept failing until I manually updated the autotest checkout on staging master.
 
There was another problem:

[4] staging push fails early on puppet failures. These same puppet failures do not block prod push, and anyway puppet changes are synced to all servers without blocking on staging. This means that the puppet failure abort adds little to no value, but blocked necessary changes from getting deployed to staging, once again requiring manual intervention.

Fix for this is at https://chrome-internal-review.googlesource.com/c/chromeos/chromeos-admin/+/639522
Actually, test_push executes out of ~/chromiumos/chromeos-admin/...

So, [4] isn't fixed yet. I still need to update ~/chromiumos manually on the staging master.
 
So, rewording [1]: To run test_push (out of ~/chromiumos/chromeos-admin)


Labels: -Pri-2 Chase-Pending Pri-1
Chase-Pending justification: broken staging leaves us unable to push for long periods.
Most of the fixes required here are independent and relatively easy to implement.
Labels: -Chase-Pending
If this happens once again, causes delay to recovery, we should make it Chase.

Comment 6 by jkop@chromium.org, Jun 21 2018

Cc: pprabhu@chromium.org jkop@chromium.org
Status: Available (was: Untriaged)

Sign in to add a comment