New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment
link

Issue 904453: Ack bad archivist messages and send them back into the pubsub queue

Reported by hinoka@chromium.org, Nov 12 Project Member

Issue description

This is a shorterm mitigation to turn certain classes of bugs in the archivist from "catastrophic" to "annoying"

The idea is that, currently if a archival task is failing (eg due to bad permissions), it gets nacked and goes into the front of the pubsub queue.  This is bad because legitimate requests are in the back of the queue and may never get processed.  If the message is acked instead, it allows legitimate requests to get processed first before the bad request is processed again.
 

Comment 1 by hinoka@chromium.org, Nov 20

Owner: hinoka@chromium.org
Status: Started (was: Available)
Proposal:
https://docs.google.com/document/d/1ltYCbObD_uQsuEwEYP3JPKmAK3M3as1Y6A1kbN_glfg/edit#

Comment 2 by bugdroid1@chromium.org, Dec 20

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/511486c4da79cfe73784dee7e826bef5c3cdbb65

commit 511486c4da79cfe73784dee7e826bef5c3cdbb65
Author: Ryan Tseng <hinoka@google.com>
Date: Thu Dec 20 19:50:33 2018

[logdog] Add RescheduleArchiveTask endpoint

This adds a new RescheduleArchiveTask coordinator endpoint
to re-task archive tasks.

ArchiveDispatchTask was never used, so the name was repurposed.

Changed assumption in appengine/coordinator/archival.go that
a log stream archival can get tasked multiple times.

Bug: 904453
Change-Id: I4832731de224d7a74d6a4341bf27f9d2ebb8aff2
Reviewed-on: https://chromium-review.googlesource.com/c/1356805
Commit-Queue: Ryan Tseng <hinoka@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/api/endpoints/coordinator/services/v1/pb.discovery.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/api/endpoints/coordinator/services/v1/service.pb.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/api/endpoints/coordinator/services/v1/service.proto
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/api/endpoints/coordinator/services/v1/servicesserver_dec.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/api/endpoints/coordinator/services/v1/tasks.pb.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/api/endpoints/coordinator/services/v1/tasks.proto
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/appengine/coordinator/archival.go
[add] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/appengine/coordinator/endpoints/services/rescheduleArchiveTask.go
[add] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/appengine/coordinator/endpoints/services/rescheduleArchiveTask_test.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/appengine/coordinator/logStreamState.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/appengine/coordinator/mutations/createArchiveTask.go
[modify] https://crrev.com/511486c4da79cfe73784dee7e826bef5c3cdbb65/logdog/server/retryServicesClient/client.go

Sign in to add a comment