getting 500 trying to view stdio in logdog |
|||
Issue descriptionHere's the log I'm trying to view: https://luci-logdog.appspot.com/v/?s=bb%2Ftryserver.chromium.win%2Fwin_optional_gpu_tests_rel%2F869%2F%2B%2Frecipes%2Fsteps%2Fbot_update%2F0%2Fstdout I'd be happy to learn how to debug this as well. Could be good for a playbook entry before the official launch.
,
Apr 19 2016
RE playbook, the only entry I can think of is fairly vague: "look at the log in the app and see if it says anything useful" :(
,
Apr 19 2016
Fixed with: https://codereview.chromium.org/1904503003
,
Apr 20 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/1ec9fb582858f2063f64030370c9912b518726f6 commit 1ec9fb582858f2063f64030370c9912b518726f6 Author: dnj <dnj@chromium.org> Date: Wed Apr 20 19:35:04 2016 LogDog: Fix archived log stream read errors. Fix some bugs in LogDog and associated libraries that prevented reading very large Google Storage archived streams. Fundamentally, Google Storage reads were not upper-bounded, despite the fact that we had the information to do so. This caused (minimally) more data than necessary to be fetched for a given request, and (in this case) a request to be issued that exceeded GAE's `urlfetch` upper bound, 32MB. We fix this by updating the archive log storage instance to calculate an upper offset when requesting log stream ranges. This exposed a bug in Google Storage library, where "from" and "to" parameters (absolute byte offsets) were being translated to "offset" and "length" parameters (with "length" relative to "offset"). This is fixed by explicitly standardizing on offset/length. Some `logs.Get` endpoint tests constrain byte count. Because we're now specifying upper bound, this caused them to return fewer logs than expected because the byte count being applied didn't include the RecordIO frame size. We fix this by calculating the frame size and factoring that into the byte range requests. BUG= chromium:604610 TEST=live - Deployed to afflicted system, verified that log stream is now functional. Review URL: https://codereview.chromium.org/1904503003 [modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/appengine/logdog/coordinator/endpoints/logs/get.go [modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/appengine/logdog/coordinator/endpoints/logs/get_test.go [modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/common/gcloud/gs/gs.go [add] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/common/recordio/size.go [add] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/common/recordio/size_test.go [modify] https://crrev.com/1ec9fb582858f2063f64030370c9912b518726f6/server/logdog/storage/archive/storage.go
,
Apr 20 2016
,
Apr 27 2016
|
|||
►
Sign in to add a comment |
|||
Comment 1 by d...@chromium.org
, Apr 19 2016Looking at the logs, it is some sort of broken Google Storage interaction: Failed to create stream Reader. :: {"error":"Get https://storage.googleapis.com/luci-logdog/logs/bb/tryserver.chromium.win/win_optional_gpu_tests_rel/869/+/recipes/steps/bot_update/0/stdout/logstream.entries: API error 3 (urlfetch: UNSPECIFIED_ERROR)"} The log does exist: $ gsutil ls gs://luci-logdog/logs/bb/tryserver.chromium.win/win_optional_gpu_tests_rel/869/+/recipes/steps/bot_update/0/stdout/logstream.entries Will look into this tomorrow.