New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 849157 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Dashboard: creating a Reader for some GCS object fails with timeout

Project Member Reported by mmoroz@chromium.org, Jun 4 2018

Issue description

https://godoc.org/cloud.google.com/go/storage#ObjectHandle.NewReader

This function fails with the following errors:

2018/06/04 02:53:15 WARNING: GcsPathReader failed: Failed to create GCS Reader: Get https://storage.googleapis.com/chromium-coverage/reports/563960/linux/metadata/tests_and_fuzzers.profdata: API error 5 (urlfetch: DEADLINE_EXCEEDED): ('The read operation timed out',). Stacktrace:
	src/gcs.GcsObjectReader() from src/gcs/gcs.go:32
	src/gcs.GcsPathReader() from src/gcs/gcs.go:53
	src/dashboard.handleReportsPage() from src/dashboard/reports_page.go:130
	src/dashboard.HandleReports() from src/dashboard/reports_page.go:37
	net/http.HandlerFunc.ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:1942
	net/http.(*ServeMux).ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:2238
	appengine_internal.handleFilteredHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/appengine_internal/api_dev.go:102
	net/http.HandlerFunc.ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:1942
	net/http.serverHandler.ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:2568
	net/http.(*conn).serve() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:1825



Another case:

2018/06/04 03:54:30 WARNING: GcsPathReader failed: Failed to create GCS Reader: Get https://storage.googleapis.com/chromium-coverage/reports/563960/linux/metadata/tests_and_fuzzers.profdata: Call error 11: Deadline exceeded (timeout). Stacktrace:
	src/gcs.GcsObjectReader() from src/gcs/gcs.go:32
	src/gcs.GcsPathReader() from src/gcs/gcs.go:53
	src/dashboard.handleReportsPage() from src/dashboard/reports_page.go:134
	src/dashboard.HandleReports() from src/dashboard/reports_page.go:41
	net/http.HandlerFunc.ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:1942
	net/http.(*ServeMux).ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:2238
	appengine_internal.handleFilteredHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/appengine_internal/api_dev.go:102
	net/http.HandlerFunc.ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:1942
	net/http.serverHandler.ServeHTTP() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:2568
	net/http.(*conn).serve() from /home/mmoroz/tools/google-cloud-sdk/platform/google_appengine/goroot-1.8/src/net/http/server.go:1825


And, most importantly, that happens on the server as well:


GcsPathReader failed: Failed to create GCS Reader: Get https://storage.googleapis.com/chromium-coverage/reports/558262/linux/metadata/tests_and_fuzzers.profdata: API error 3 (urlfetch: UNSPECIFIED_ERROR). Stacktrace:
	src/gcs.GcsObjectReader() from src/gcs/gcs.go:32
	src/gcs.GcsPathReader() from src/gcs/gcs.go:53
	src/dashboard.handleReportsPage() from src/dashboard/reports_page.go:130
	src/dashboard.HandleReports() from src/dashboard/reports_page.go:37
	net/http.HandlerFunc.ServeHTTP() from go/src/net/http/server.go:1942
	net/http.(*ServeMux).ServeHTTP() from go/src/net/http/server.go:2238
	appengine_internal.executeRequestSafely() from go/src/appengine_internal/api_prod.go:296
	appengine_internal.(*server).HandleRequest() from go/src/appengine_internal/api_prod.go:232
	runtime.call64() from /goroot/src/runtime/asm_amd64.s:515
	reflect.Value.call() from go/src/reflect/value.go:434
	reflect.Value.Call() from go/src/reflect/value.go:302
	_() from _:427GcsPathReader failed: Failed



Looks like this is happening with large files only (e.g. .profdata), as all regular files work fast.
 
Here is a similar error: https://github.com/GoogleCloudPlatform/google-cloud-go/issues/950


I've tried using a timeout and that worked locally, but doesn't seem to work on the server: https://chrome-internal-review.googlesource.com/c/chrome/tools/code-coverage/+/634928/1/src/dashboard/reports_page.go

But I have a backup solution, I think.
Project Member

Comment 2 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/5d41915709c43dcce82a276f37dd16545a8bd746

commit 5d41915709c43dcce82a276f37dd16545a8bd746
Author: Max Moroz <mmoroz@google.com>
Date: Mon Jun 04 06:04:53 2018

Status: Fixed (was: Started)
Cc: mmoroz@chromium.org
 Issue 848160  has been merged into this issue.

Sign in to add a comment