New issue
Advanced search Search tips

Issue 905729 link

Starred by 1 user

Issue metadata

Status: Available
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: 2018-11-19
OS: ----
Pri: 3
Type: Bug

Blocking:
issue 903417



Sign in to add a comment

HTML trace missing from story artifacts in Telemetry output.json

Project Member Reported by charliea@google.com, Nov 15

Issue description

I've had to debug several trace import error issues lately. This requires getting a copy of the .html trace file that corresponds to each failing story.

Recently this file has become harder to access. Before OBBS, it was accessible at the bottom of the output. Now, it isn't.

Example: I'm trying to get the output trace from the failure of gmail-foreground.

Here's a link to the run: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/mac-10_12_laptop_low_end-perf/2019

Searching the output JSON (https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8930418175158560832/+/steps/performance_test_suite_on_Intel_GPU_on_Mac_on_Mac-10.12/0/logs/json.output/0) for "gmail-foreground" shows that the only failure artifact attached to the story is the logs for that story.

Searching the logs for that story (https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/637209a3-e368-11e8-b19c-7c04d0d1d2bc) for "cloudstorage" shows that there's no HTML file upload available.

Searching for GTEST_SHARD_INDEX in that same file shows that this story ran on shard #13.

Searching the shard #13 output (https://chrome-swarming.appspot.com/task?id=410c830202948f10&refresh=10&show_raw=1) for "gmail-foreground" shows that the story did indeed run on this shard. However, searching for "cloudstorage" similarly doesn't turn up any HTML files that can be downloaded to debug failures.

This is blocking the debugging of at least 2 flaky failures that I'm not able to reproduce locally. Do we know how we can get the HTML trace file for a story added as a test artifact to that story? That'd make the trace easily accessible via the output JSON, which seems like probably the right place for it to be.
 
Blocking: -900935
Actually, removing 900935 the list of blocked bugs. For some reason or another, the HTML files for the Telemetry tests on the internal clank waterfall are at least available at the bottom of the Telemetry standard output. This doesn't seem to be the case for OBBS Telemetry output.

(Even for that file, it took me ~30m of digging in order to find the HTML file, so it'd be really useful still to have an easy way to access the HTML file as a story artifact listed in the json.output file.)
Cc: -eyaich@chromium.org
Owner: eyaich@chromium.org
Adding eyaich@ as the likely owner here, given that this seems like it might be related to the OBBS migration.
Cc: charliea@chromium.org
NextAction: 2018-11-19
So I think what is happening here is Ned implemented a feature in telemetry to output logs for each benchmark and include that in the artifacts so there was an easy way to access the logs for individual benchmarks instead of having to load the entire shard which is the results of multiple benchmarks running.

ie we wanted you to click on the "Benchmark Logs" link and lookup your benchmark instead of going to the chrome-swarming* page and loading the entire run.  

That being said, I do remember a design decision when we were talking about the feature (although I can't find the thread/bug atm) that we accepted that these logs only include what is run during the benchmark execution itself. 

Ned do you remember the implementation here and exactly what gets excluded? 

That being said, the trace link is in the larger shard output: 

https://chrome-swarming.appspot.com/task?id=410c830202948f10&refresh=10&show_raw=1

View generated trace files online at https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/long_running_tools_gmail_foreground_2018-11-08_06-50-27_42952.html for story long_running:tools:gmail-foreground

I don't know why your search for cloudstorage didn't show any results, maybe the shard hadn't finished loading yet?
Would it be possible to just add the URL of the uploaded trace file as a test artifact like we do for the uploaded logs?
(And thank you for finding the trace! It looks like after searching, I didn't wait long enough for the search to actually complete.)
#6: yes, I think we can add the URL of the uploaded trace as a test artifact. Though it would require wranggling with the life cycle of the trace a bit. 

I recall the reason we didn't do in the first place is because we have one system of uploading traces to cloud storage [1] that is separate from the system that uploads artifcats to cloud storage [2]. The challenge with merging [1] to [2] has something to do with being able to write the trace URLs to the results format :-/

[1] https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/value/trace.py?rcl=eaddc34b5d484d7a65cfdb2cf8a0373cd2e09025&l=138
[2] https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/results/page_test_results.py?rcl=eaddc34b5d484d7a65cfdb2cf8a0373cd2e09025&l=624
Cc: crouleau@chromium.org
The NextAction date has arrived: 2018-11-19
This might not be too bad.  We could pass state here from one call to another, or save it in the page_test_results:

https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py?l=419

Comment 12 by benhenry@google.com, Jan 16 (6 days ago)

Components: Test>Telemetry

Comment 13 by benhenry@google.com, Jan 16 (6 days ago)

Components: -Speed>Telemetry

Sign in to add a comment