Issue metadata
Sign in to add a comment
|
HTML trace missing from story artifacts in Telemetry output.json |
||||||||||||||||||||
Issue descriptionI've had to debug several trace import error issues lately. This requires getting a copy of the .html trace file that corresponds to each failing story. Recently this file has become harder to access. Before OBBS, it was accessible at the bottom of the output. Now, it isn't. Example: I'm trying to get the output trace from the failure of gmail-foreground. Here's a link to the run: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/mac-10_12_laptop_low_end-perf/2019 Searching the output JSON (https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8930418175158560832/+/steps/performance_test_suite_on_Intel_GPU_on_Mac_on_Mac-10.12/0/logs/json.output/0) for "gmail-foreground" shows that the only failure artifact attached to the story is the logs for that story. Searching the logs for that story (https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/637209a3-e368-11e8-b19c-7c04d0d1d2bc) for "cloudstorage" shows that there's no HTML file upload available. Searching for GTEST_SHARD_INDEX in that same file shows that this story ran on shard #13. Searching the shard #13 output (https://chrome-swarming.appspot.com/task?id=410c830202948f10&refresh=10&show_raw=1) for "gmail-foreground" shows that the story did indeed run on this shard. However, searching for "cloudstorage" similarly doesn't turn up any HTML files that can be downloaded to debug failures. This is blocking the debugging of at least 2 flaky failures that I'm not able to reproduce locally. Do we know how we can get the HTML trace file for a story added as a test artifact to that story? That'd make the trace easily accessible via the output JSON, which seems like probably the right place for it to be.
,
Nov 15
Adding eyaich@ as the likely owner here, given that this seems like it might be related to the OBBS migration.
,
Nov 15
,
Nov 15
,
Nov 15
So I think what is happening here is Ned implemented a feature in telemetry to output logs for each benchmark and include that in the artifacts so there was an easy way to access the logs for individual benchmarks instead of having to load the entire shard which is the results of multiple benchmarks running. ie we wanted you to click on the "Benchmark Logs" link and lookup your benchmark instead of going to the chrome-swarming* page and loading the entire run. That being said, I do remember a design decision when we were talking about the feature (although I can't find the thread/bug atm) that we accepted that these logs only include what is run during the benchmark execution itself. Ned do you remember the implementation here and exactly what gets excluded? That being said, the trace link is in the larger shard output: https://chrome-swarming.appspot.com/task?id=410c830202948f10&refresh=10&show_raw=1 View generated trace files online at https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/long_running_tools_gmail_foreground_2018-11-08_06-50-27_42952.html for story long_running:tools:gmail-foreground I don't know why your search for cloudstorage didn't show any results, maybe the shard hadn't finished loading yet?
,
Nov 15
Would it be possible to just add the URL of the uploaded trace file as a test artifact like we do for the uploaded logs?
,
Nov 15
(And thank you for finding the trace! It looks like after searching, I didn't wait long enough for the search to actually complete.)
,
Nov 15
#6: yes, I think we can add the URL of the uploaded trace as a test artifact. Though it would require wranggling with the life cycle of the trace a bit. I recall the reason we didn't do in the first place is because we have one system of uploading traces to cloud storage [1] that is separate from the system that uploads artifcats to cloud storage [2]. The challenge with merging [1] to [2] has something to do with being able to write the trace URLs to the results format :-/ [1] https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/value/trace.py?rcl=eaddc34b5d484d7a65cfdb2cf8a0373cd2e09025&l=138 [2] https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/results/page_test_results.py?rcl=eaddc34b5d484d7a65cfdb2cf8a0373cd2e09025&l=624
,
Nov 15
,
Nov 19
The NextAction date has arrived: 2018-11-19
,
Nov 19
This might not be too bad. We could pass state here from one call to another, or save it in the page_test_results: https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py?l=419
,
Jan 16
(6 days ago)
,
Jan 16
(6 days ago)
|
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by charliea@google.com
, Nov 15