New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 809789 link

Starred by 1 user
Status: Verified
Owner:
█ pho...@chromium.org
Last visit > 30 days ago
Closed: Feb 2018
Cc:
█ pho...@chromium.org
xixuan@chromium.org
ayatane@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug

Blocked on:
issue 810047


Sign in to add a comment

"prod" refs not being updated on Go infra packages

Project Member Reported by derat@chromium.org, Feb 7 2018 
The "prod" refs for our Go infra packages look like they haven't been updated since Feb 2:

% cipd instances chromiumos/infra/lucifer
Instance ID                              │ Timestamp             │ Uploader                  │ Refs        
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
f6d671a2688c315b490e99d2b8b4d2e9deaac87d │ Feb 06 12:46 PST 2018 │ cipd-uploader@chromeos... │ staging latest
6c5297dc8ad3743a4547d8dd4a8767e58c77578c │ Feb 06 05:25 PST 2018 │ cipd-uploader@chromeos... │             
950ec3f2d8e91ee22e7a2d7f09fbf6de3db905c6 │ Jan 31 19:17 PST 2018 │ cipd-uploader@chromeos... │ prod prod-next
...

% cipd instances chromiumos/infra/tast-cmd
Instance ID                              │ Timestamp             │ Uploader                  │ Refs        
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
5364720c169f5bd1e3ac187f1be2a85f521e32cc │ Feb 06 11:10 PST 2018 │ cipd-uploader@chromeos... │ staging latest
969741bd2d9bc3b4de5aef7bfc14e25d5c5305d2 │ Feb 06 05:25 PST 2018 │ cipd-uploader@chromeos... │             
beaa730ae043f7730e1b93099900899bdfcf9974 │ Feb 04 19:43 PST 2018 │ cipd-uploader@chromeos... │ prod-next   
233d61b6d34fe346884b895d6e8f005cfa09dd55 │ Jan 29 17:13 PST 2018 │ cipd-uploader@chromeos... │ prod
...

% cipd instances chromiumos/infra/tast-remote-tests-cros
Instance ID                              │ Timestamp             │ Uploader                  │ Refs        
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
e0893474c09533d65c976d519a6580308bcb9faf │ Feb 06 07:34 PST 2018 │ cipd-uploader@chromeos... │ staging latest
70237105b9a19adde819684f9fc0e18363a6315f │ Feb 06 05:25 PST 2018 │ cipd-uploader@chromeos... │             
5c0b0378450bd5e9aebd2d191cc1874971f68fa1 │ Feb 04 19:43 PST 2018 │ cipd-uploader@chromeos... │ prod-next   
2980dcad1a60f9388947e8a40fc7f48d5be8e41a │ Feb 02 07:34 PST 2018 │ cipd-uploader@chromeos... │ prod

Per phobbs on crosoncall, "The lucifer prod / prod-next labels should be controlled by our push infrastructure. I did a push yesterday, so it should be nearly up-to-date. Not sure why it's so old."

Allen, do know what's going on here?

Comment 1 by ayatane@chromium.org, Feb 7 2018 

The latest ref is the latest, naturally.

staging is the ref used by the staging lab to mark what the staging lab is testing.

If the staging lab tests pass, prod-next is moved to staging.  During push prod is moved to prod-next.

Staging lab has been bad lately, so prod-next is old.

I believe phobbs pushed yesterday, so prod should be on prod-next (like lucifer is).  Not sure, can phobbs comment?  Were you perhaps not using ToT for the push?

Comment 2 by derat@chromium.org, Feb 7 2018 

Thanks for the details. Is there a bug (or bugs) tracking the current lab issue(s)?

I think I misunderstood earlier, in that I thought that new code would get pushed automatically (assuming that tests pass). Per your comment, I surmise that we still need a manual prod push to update the "prod" refs (at which point Puppet quickly deploys the updated CIPD packages, right?).

Comment 3 by ayatane@chromium.org, Feb 7 2018 

Yes, all our prod Autotest changes need manual pushing, mostly for the human supervision if things go south.  We do pushes regularly (avg 2 times a week).

Comment 4 by derat@chromium.org, Feb 7 2018

Cc: xixuan@chromium.org
Owner: pho...@chromium.org
Status: Assigned (was: Available)
Paul, do you know if there are bugs tracking the staging lab issues that are preventing these refs from being updated?

Comment 5 by pho...@chromium.org, Feb 7 2018 

No, I started looking into the staging lab problem yesterday but didn't find anything. I'll open a bug.

Comment 6 by ayatane@chromium.org, Feb 7 2018

Blockedon: 810047

Comment 7 by derat@chromium.org, Feb 10 2018 

This will hopefully go through with the next push to prod now:

% cipd instances chromiumos/infra/tast-remote-tests-cros
Instance ID                              │ Timestamp             │ Uploader                  │ Refs
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
e0893474c09533d65c976d519a6580308bcb9faf │ Feb 06 07:34 PST 2018 │ cipd-uploader@chromeos... │ prod-next staging latest
70237105b9a19adde819684f9fc0e18363a6315f │ Feb 06 05:25 PST 2018 │ cipd-uploader@chromeos... │
5c0b0378450bd5e9aebd2d191cc1874971f68fa1 │ Feb 04 19:43 PST 2018 │ cipd-uploader@chromeos... │
2980dcad1a60f9388947e8a40fc7f48d5be8e41a │ Feb 02 07:34 PST 2018 │ cipd-uploader@chromeos... │ prod
d87c2b9c9769e35c83dd683eb455041f14dbb834 │ Jan 29 17:14 PST 2018 │ cipd-uploader@chromeos... │
8c483ac1264c68c4921a3c4a36b3fdf120c7603c │ Jan 25 23:11 PST 2018 │ cipd-uploader@chromeos... │

Comment 8 by pho...@chromium.org, Feb 13 2018

Status: Verified (was: Assigned)
►

Sign in to add a comment