This patch: [Catapult] Update gsutil to 4.30 (https://crrev.com/c/985895) may be related to this issue. it is landed on Chrome 67.0.3385.0 and Chrome OS 10554.0.0.

The following patch could solve this issue by copying .boto file from local to the DUT.
https://crrev.com/c/1021201

an example from 10554.0.0 failing:
(INFO) 2018-04-12 06:50:49,333 archive_info.DownloadArchivesIfNeeded:68  Downloading WPR archives. This can take a long time.
(INFO) 2018-04-12 06:50:49,334 cloud_storage._GetLocked:340  Downloading gs://chrome-partner-telemetry/ea990882aea7b63d9c22c4c079a5756164bd69ff to /usr/local/telemetry/src/tools/perf/page_sets/data/cros_ui_cases_000.wprgo
(ERROR) 2018-04-12 06:50:51,586 archive_info.download_if_needed:93  You either aren't authenticated or don't have permission to use the archives for this page set.
You may need to run gsutil config.
You can find instructions for gsutil config at: http://www.chromium.org/developers/telemetry/upload_to_cloud_storage
(INFO) 2018-04-12 06:50:51,587 archive_info.DownloadArchivesIfNeeded:110  All WPR archives are downloaded, took 2.25339221954 seconds.
(CRITICAL) 2018-04-12 06:50:51,588 story_runner.RunBenchmark:367  Benchmark execution interrupted by a fatal exception.
(INFO) 2018-04-12 06:50:51,588 exception_formatter.PrintFormattedException:19  Try printing formatted exception: None None None

the previous 10553.0.0 didn't fail, but it also didn't try to download anything:
(INFO) 2018-04-10 17:04:18,106 archive_info.DownloadArchivesIfNeeded:68  Downloading WPR archives. This can take a long time.
(INFO) 2018-04-10 17:04:18,115 archive_info.DownloadArchivesIfNeeded:110  All WPR archives are downloaded, took 0.00790405273438 seconds.

i don't know if there's a way to grep the logs more generally to find a passing case which actually downloaded a file to see if there's any relationship to the gsutil upgrade.

at any rate, if the DUT is downloading a file directly, and those files aren't readable by the world (e.g. a "public-read" ACL), then it sounds like correct behavior that gsutil was unable to download things.

The cause of this issue:
TelemetryRunner._ensure_deps (https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/cros/telemetry_runner.py) parse the stdout of fetch_benchmark_deps.py to download wprgo DEPS to DUT.
https://crrev.com/c/984381 changed the output of fetch_benchmark_deps.py, and _ensure_deps() is broken.

if _ensure_deps is fixed, the patch to copy .boto file (https://crrev.com/c/1021201) is not necessary.

Hi Vovo,

The fix can be done by adding an option to fetch_benchmark_deps.py to make it output the dependencies in a parser friendly way, such as json. Then adjust _ensure_deps() accordingly.

A quick hack would be adding -v when calling fetch_benchmark_deps.py in _ensure_deps(), if you can't wait for the deps roll and chrome uprev.

I will fix with the following steps:
1. Adding option --output-deps to fetch_benchmark_deps.py, the output will be the same as before https://crrev.com/c/984381 applied.
2. Using option --output-deps in TelemetryRunner._ensure_deps().

The reviewing CLs to fix this issue:
1. Adding option --output-deps in fetch_benchmark_deps.py: https://crrev.com/c/1025612
2. Using option --output-deps in telemetry_runner: https://crrev.com/c/1027371

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e6b675d6797927ddcf9378edd53f55d5bd48d591

commit e6b675d6797927ddcf9378edd53f55d5bd48d591
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Wed Apr 25 15:19:16 2018

[tools/perf] Add option --output-deps to output deps to stdout

The output is used by CrOS autotest telemetry_runner to upload
dependencies to DUT.

Bug:  834659 
Change-Id: I5845fa293e07b35bdcb27030da7fc87ca5b61505
Reviewed-on: https://chromium-review.googlesource.com/1025612
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553559}
[modify] https://crrev.com/e6b675d6797927ddcf9378edd53f55d5bd48d591/tools/perf/fetch_benchmark_deps.py
[modify] https://crrev.com/e6b675d6797927ddcf9378edd53f55d5bd48d591/tools/perf/fetch_benchmark_deps_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e8323033c5ff5755abe6e45ba05060c5a5d90111

commit e8323033c5ff5755abe6e45ba05060c5a5d90111
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Fri Apr 27 06:20:39 2018

[tools/perf] Option --output-deps outputs deps to a json file

Bug:  834659 
Change-Id: I5df0bc9b923c7a780de896f71cc68443902de12b
Reviewed-on: https://chromium-review.googlesource.com/1031991
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Vovo Yang <vovoy@chromium.org>
Cr-Commit-Position: refs/heads/master@{#554325}
[modify] https://crrev.com/e8323033c5ff5755abe6e45ba05060c5a5d90111/tools/perf/fetch_benchmark_deps.py
[modify] https://crrev.com/e8323033c5ff5755abe6e45ba05060c5a5d90111/tools/perf/fetch_benchmark_deps_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4a00617f40f5484c1b80e8ea4813587b7cec22ec

commit 4a00617f40f5484c1b80e8ea4813587b7cec22ec
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Thu May 03 11:47:34 2018

Using fetch_benchmark_deps.py --output-deps to get the dependencies

BUG= chromium:834659 
TEST=Run benchmark telemetry_Benchmarks.cros_ui_smoothness
CQ-DEPEND=CL:1031991

Change-Id: Icd0e8d9de39dab86cbe43a3bfe0b5f6f1773a752
Reviewed-on: https://chromium-review.googlesource.com/1027371
Commit-Ready: Vovo Yang <vovoy@chromium.org>
Tested-by: Vovo Yang <vovoy@chromium.org>
Reviewed-by: Chung-yih Wang <cywang@chromium.org>

[modify] https://crrev.com/4a00617f40f5484c1b80e8ea4813587b7cec22ec/server/cros/telemetry_runner.py

Verified the CrOS telemetry autotests are working on R68-10644.0.0

https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-04-23&last_date=2018-05-04&suite=%5Ecrosbolt%5C_perf%5C_perbuild%24&test=%5Etelemetry_

The fix is also needed on M67, otherwise the CrOS telemetry tests would fail on M67 builds.

This bug requires manual review: DEPS changes referenced in bugdroid comments.
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Request to merge e6b675d6797927ddcf9378edd53f55d5bd48d591 and e8323033c5ff5755abe6e45ba05060c5a5d90111 to M67 branch 3396, and merge 4a00617f40f5484c1b80e8ea4813587b7cec22ec to CrOS M67.

How critical is this for M67?  Limited to testing w/o prod impact?  Thanks

it's limited to testing w/o prod impact. It would affect several telemetry tests on CrOS.

Approving merge to M67 Chrome OS since it's limited to testing.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/21d7d065431a3f1e6259c1ca6b60c302ff494378

commit 21d7d065431a3f1e6259c1ca6b60c302ff494378
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Wed May 16 10:08:34 2018

[tools/perf] Add option --output-deps to output deps to stdout

The output is used by CrOS autotest telemetry_runner to upload
dependencies to DUT.

Bug:  834659 
Change-Id: I5845fa293e07b35bdcb27030da7fc87ca5b61505
Reviewed-on: https://chromium-review.googlesource.com/1025612
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#553559}(cherry picked from commit e6b675d6797927ddcf9378edd53f55d5bd48d591)
Reviewed-on: https://chromium-review.googlesource.com/1059067
Cr-Commit-Position: refs/branch-heads/3396@{#613}
Cr-Branched-From: 9ef2aa869bc7bc0c089e255d698cca6e47d6b038-refs/heads/master@{#550428}
[modify] https://crrev.com/21d7d065431a3f1e6259c1ca6b60c302ff494378/tools/perf/fetch_benchmark_deps.py
[modify] https://crrev.com/21d7d065431a3f1e6259c1ca6b60c302ff494378/tools/perf/fetch_benchmark_deps_unittest.py

Request to merge e8323033c5ff5755abe6e45ba05060c5a5d90111 to M67 branch 3396, and merge 4a00617f40f5484c1b80e8ea4813587b7cec22ec to CrOS M67.

This bug requires manual review: We are only 12 days from stable.
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I need more context for #22.  Why do you need more merges? Was there a bug in #17?      Please comment on testing as well.

In #17, I try to request to merge 3 CLs to M67. But after the first CL is merged to M67, the merge-approved-67 tag was removed automatically and I couldn't merged the remaining 2 CLs (1 for Cr, 1 for CrOS). What's the process to request to merge more than 1 CL?

ok, merge-approved 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd460fc8745ca2241d2f93c54e7bd60cff3aed98

commit bd460fc8745ca2241d2f93c54e7bd60cff3aed98
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Thu May 17 17:23:14 2018

[tools/perf] Option --output-deps outputs deps to a json file

Bug:  834659 
Change-Id: I5df0bc9b923c7a780de896f71cc68443902de12b
Reviewed-on: https://chromium-review.googlesource.com/1031991
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Vovo Yang <vovoy@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#554325}(cherry picked from commit e8323033c5ff5755abe6e45ba05060c5a5d90111)
Reviewed-on: https://chromium-review.googlesource.com/1061653
Cr-Commit-Position: refs/branch-heads/3396@{#623}
Cr-Branched-From: 9ef2aa869bc7bc0c089e255d698cca6e47d6b038-refs/heads/master@{#550428}
[modify] https://crrev.com/bd460fc8745ca2241d2f93c54e7bd60cff3aed98/tools/perf/fetch_benchmark_deps.py
[modify] https://crrev.com/bd460fc8745ca2241d2f93c54e7bd60cff3aed98/tools/perf/fetch_benchmark_deps_unittest.py

Request to merge 4a00617f40f5484c1b80e8ea4813587b7cec22ec to CrOS M67.
Sorry for the inconvenience, the Merge-Approved-67 label was removed automatically in #27, request merge again.

This bug requires manual review: We are only 10 days from stable.
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Covering for Kevin, updating merge-review back to merge-approved per #20.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/e0f00ee7d310ce5b458e43cce7b66fac19698f7a

commit e0f00ee7d310ce5b458e43cce7b66fac19698f7a
Author: Kuo-Hsin Yang <vovoy@chromium.org>
Date: Sun May 20 06:20:52 2018

Using fetch_benchmark_deps.py --output-deps to get the dependencies

BUG= chromium:834659 
TEST=Run benchmark telemetry_Benchmarks.cros_ui_smoothness
CQ-DEPEND=CL:1031991

Change-Id: Icd0e8d9de39dab86cbe43a3bfe0b5f6f1773a752
Reviewed-on: https://chromium-review.googlesource.com/1027371
Commit-Ready: Vovo Yang <vovoy@chromium.org>
Tested-by: Vovo Yang <vovoy@chromium.org>
Reviewed-by: Chung-yih Wang <cywang@chromium.org>
(cherry picked from commit 4a00617f40f5484c1b80e8ea4813587b7cec22ec)
Reviewed-on: https://chromium-review.googlesource.com/1064650
Trybot-Ready: Vovo Yang <vovoy@chromium.org>
Reviewed-by: Chung-yih Wang <cywang@google.com>
Commit-Queue: Vovo Yang <vovoy@chromium.org>

[modify] https://crrev.com/e0f00ee7d310ce5b458e43cce7b66fac19698f7a/server/cros/telemetry_runner.py

Verified the CrOS telemetry autotests are working on R67-10575.45.0

https://stainless.corp.google.com/search?view=matrix&row=test&col=build&first_date=2018-05-15&last_date=2018-05-22&suite=%5Ecrosbolt%5C_perf%5C_perbuild%24&test=%5Etelemetry_&build=R67-