New issue
Advanced search Search tips

Issue 604610 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2016
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

getting 500 trying to view stdio in logdog

Project Member Reported by estaab@chromium.org, Apr 19 2016

Issue description

Here's the log I'm trying to view:

https://luci-logdog.appspot.com/v/?s=bb%2Ftryserver.chromium.win%2Fwin_optional_gpu_tests_rel%2F869%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout

I'd be happy to learn how to debug this as well. Could be good for a playbook entry before the official launch.
 

Comment 1 by d...@chromium.org, Apr 19 2016

Looking at the logs, it is some sort of broken Google Storage interaction:

Failed to create stream Reader. :: {"error":"Get https://storage.googleapis.com/luci-logdog/logs/bb/tryserver.chromium.win/win_optional_gpu_tests_rel/869/+/recipes/steps/bot_update/0/stdout/logstream.entries: API error 3 (urlfetch: UNSPECIFIED_ERROR)"}

The log does exist:

$ gsutil ls gs://luci-logdog/logs/bb/tryserver.chromium.win/win_optional_gpu_tests_rel/869/+/recipes/steps/bot_update/0/stdout/logstream.entries

Will look into this tomorrow.

Comment 2 by d...@chromium.org, Apr 19 2016

RE playbook, the only entry I can think of is fairly vague: "look at the log in the app and see if it says anything useful" :(
Project Member

Comment 4 by bugdroid1@chromium.org, Apr 20 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/1ec9fb582858f2063f64030370c9912b518726f6

commit 1ec9fb582858f2063f64030370c9912b518726f6
Author: dnj <dnj@chromium.org>
Date: Wed Apr 20 19:35:04 2016

LogDog: Fix archived log stream read errors.

Fix some bugs in LogDog and associated libraries that prevented reading
very large Google Storage archived streams.

Fundamentally, Google Storage reads were not upper-bounded, despite the
fact that we had the information to do so. This caused (minimally) more
data than necessary to be fetched for a given request, and (in this
case) a request to be issued that exceeded GAE's `urlfetch` upper bound,
32MB. We fix this by updating the archive log storage instance to
calculate an upper offset when requesting log stream ranges.

This exposed a bug in Google Storage library, where "from" and "to"
parameters (absolute byte offsets) were being translated to "offset" and
"length" parameters (with "length" relative to "offset"). This is fixed
by explicitly standardizing on offset/length.

Some `logs.Get` endpoint tests constrain byte count. Because we're now
specifying upper bound, this caused them to return fewer logs than
expected because the byte count being applied didn't include the RecordIO
frame size. We fix this by calculating the frame size and factoring that
into the byte range requests.

BUG= chromium:604610 
TEST=live
  - Deployed to afflicted system, verified that log stream is now
    functional.

Review URL: https://codereview.chromium.org/1904503003

[modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/appengine/logdog/coordinator/endpoints/logs/get.go
[modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/appengine/logdog/coordinator/endpoints/logs/get_test.go
[modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/common/gcloud/gs/gs.go
[add] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/common/recordio/size.go
[add] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/common/recordio/size_test.go
[modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/server/logdog/storage/archive/storage.go

Comment 5 by d...@chromium.org, Apr 20 2016

Status: Fixed (was: Assigned)

Comment 6 by aga...@chromium.org, Apr 27 2016

Components: Infra>Platform
Labels: -Infra-Platform

Sign in to add a comment