New issue
Advanced search Search tips

Issue 861527 link

Starred by 1 user
Status: WontFix
Owner:
vadimsh@chromium.org
Closed: Jul 10
Cc:
eyaich@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Perf dashboard upload fails with "Invalid or revoked service account key"

Project Member Reported by nednguyen@chromium.org, Jul 7 
In https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2Flinux-perf%2F415%2F%2B%2Frecipes%2Fsteps%2Fperformance_test_suite_on_NVIDIA_GPU_on_Linux%2F0%2Flogs%2FMerge_script_log%2F0, 

We have the following error:
[W2018-07-06T22:59:26.251499-07:00 4473 0 service.go:130] Invalid or revoked service account key - oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error" : "invalid_grant",
  "error_description" : "Invalid JWT: No valid verifier found for issuer."
} {"key":"service_account/chromium-perf-histograms@chromeperf.iam.gserviceaccount.com/8052b319b45f9c43275da26cec8b8b6dc35efa69", "scopes":"https://www.googleapis.com/auth/userinfo.email"}
[W2018-07-06T22:59:26.251600-07:00 4473 0 auth.go:1276] Failed to refresh the token: invalid service account credentials {"key":"service_account/chromium-perf-histograms@chromeperf.iam.gserviceaccount.com/8052b319b45f9c43275da26cec8b8b6dc35efa69", "scopes":"https://www.googleapis.com/auth/userinfo.email"}
invalid service account credentials
Error uploading perf result of dummy_benchmark.noisy_benchmark_1.reference
[W2018-07-06T22:59:26.257918-07:00 4502 0 service.go:130] Invalid or revoked service account key - oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error" : "invalid_grant",
  "error_description" : "Invalid JWT: No valid verifier found for issuer."
} {"key":"service_account/chromium-perf-histograms@chromeperf.iam.gserviceaccount.com/8052b319b45f9c43275da26cec8b8b6dc35efa69", "scopes":"https://www.googleapis.com/auth/userinfo.email"}
[W2018-07-06T22:59:26.258018-07:00 4502 0 auth.go:1276] Failed to refresh the token: invalid service account credentials {"key":"service_account/chromium-perf-histograms@chromeperf.iam.gserviceaccount.com/8052b319b45f9c43275da26cec8b8b6dc35efa69", "scopes":"https://www.googleapis.com/auth/userinfo.email"}
invalid service account credentials
Error uploading perf result of rasterize_and_record_micro.top_25.reference

Simon, can you take a look at this or triage?
Project Member

Comment 1 by bugdroid1@chromium.org, Jul 8 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a3f710d70a68f43e9424907df12547581f146b78

commit a3f710d70a68f43e9424907df12547581f146b78
Author: Ned Nguyen <nednguyen@google.com>
Date: Sun Jul 08 00:36:28 2018

Correct the return value of _upload_individual_benchmark in case of Exception raised.

Previously, when there is any exception, this return (benchmark_name, False). The correct value is (benchmark_name, True) since the second param describes "whether the benchmark has failed".

This also add logic to handles the case perf dashboard output_file doesn't exist
in _write_perf_data_to_logfile (see the error stack at the end of
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2Flinux-perf%2F415%2F%2B%2Frecipes%2Fsteps%2Fperformance_test_suite_on_NVIDIA_GPU_on_Linux%2F0%2Flogs%2FMerge_script_log%2F0)

Bug:860518, 861527 
Change-Id: I17bc1b0a66670301c00c7aec9f1ec8e61446bc3d
TBR=eyaich@chromium.org
Cq-Include-Trybots: master.tryserver.chromium.perf:obbs_fyi

NOTRY=true

Change-Id: I17bc1b0a66670301c00c7aec9f1ec8e61446bc3d
Reviewed-on: https://chromium-review.googlesource.com/1128410
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#573176}
[modify] https://crrev.com/a3f710d70a68f43e9424907df12547581f146b78/tools/perf/process_perf_results.py
Project Member

Comment 2 by bugdroid1@chromium.org, Jul 8 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ea28776424d02e0c21242069bdfe89d90315a951

commit ea28776424d02e0c21242069bdfe89d90315a951
Author: Ned Nguyen <nednguyen@google.com>
Date: Sun Jul 08 06:13:57 2018

Fix reference to output_json_file.get_viewer_url with viewer_url variable

This also wrap the call to _handle_perf_results in a try catch to block to
handle any failures of that method gracefully.

NOTRY=true
TBR=eyaich@chromium.org

Bug: 860518, 861527 
Change-Id: Ibf99b132d2520227b4da861935ef939ebafbd542
Reviewed-on: https://chromium-review.googlesource.com/1128418
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#573183}
[modify] https://crrev.com/ea28776424d02e0c21242069bdfe89d90315a951/tools/perf/process_perf_results.py

Comment 3 by simonhatch@chromium.org, Jul 9

Owner: nednguyen@chromium.org
Don't think I'm the right person to look at this, I don't know anything about infra's auth tool which is what looks like is failing.

Comment 4 by nednguyen@chromium.org, Jul 9

Owner: vadimsh@chromium.org
Vadim: look like you contributed lots of code in this area. Can you look at this, or help with routing this bug to the righet person?

Comment 5 by vadimsh@chromium.org, Jul 9 

Is this still happening? The key exists, so it shouldn't be happening... I suspect it is some sort of a transient error on Google OAuth side (as supported by a bunch of bugs in Buganizer).

Comment 6 by nednguyen@chromium.org, Jul 10

Status: WontFix (was: Assigned)
I haven't since this again, so this maybe just a transient error as you said
►

Sign in to add a comment