Telemetry failure log has no exceptions or clear errors |
||||||
Issue descriptionI ran this pinpoint job: https://pinpoint-dot-chromeperf.appspot.com/job/17cc8235a40000 All the runs with the patch applied failed. But neither pinpoint nor I can see why. Some example logs: https://chrome-swarming.appspot.com/task?id=3eccf72131ebeb10&refresh=10&show_raw=1 https://chrome-swarming.appspot.com/task?id=3eccf72291fcda10&refresh=10&show_raw=1 https://chrome-swarming.appspot.com/task?id=3eccf723675d7810&refresh=10&show_raw=1 https://chrome-swarming.appspot.com/task?id=3eccf724420b8b10&refresh=10&show_raw=1 NOTE: The patch in question re-recorded the story. It's possible the recording is broken or nonexistent.
,
Jul 19
Note: I'm seeing similar logs on Linux (first link is Mac): https://pinpoint-dot-chromeperf.appspot.com/job/11e25c3ea40000
,
Jul 20
,
Jul 20
I am able to reproduce the problem on my Mac. Steps: $ git cl patch 1143585 $ ./tools/perf/run_benchmark system_health.common_desktop --story-filter multitab:misc:typical24 --pageset-repeat 1 --browser=system --also-run-disabled-tests Investigating
,
Jul 20
With https://chromium-review.googlesource.com/c/catapult/+/1145042, I can now see the error is due to trace processing: (ERROR) 2018-07-20 07:48:49,201 page_test_results.Fail:540 Failure recorded: TraceImportError: Unexpected token Ö in JSON at position 0 at JSON.parse (<anonymous>) at TraceEventImporter.createAsyncSlices_ (/tracing/extras/importer/trace_event_importer.html:1645:31) at TraceEventImporter.importEvents (/tracing/extras/importer/trace_event_importer.html:1150:12) at importer (/tracing/importer/import.html:198:65) at task.subTask (/tracing/importer/import.html:145:32) at Task.run (/tracing/base/task.html:80:50) at Function.Task.RunSynchronously (/tracing/base/task.html:125:25) at Import.importTraces (/tracing/importer/import.html:74:17) at createModelFromTraceData (/Users/nednguyen/projects/chromium/src/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:38:9) at eval (/Users/nednguyen/projects/chromium/src/third_party/catapult/tracing/tracing/mre/map_single_trace_cmdline.html:57:25) [ FAILED ] system_health.common_desktop/multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'} (139881 ms) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] system_health.common_desktop/multitab:misc:typical24@{'case': 'multitab', 'group': 'misc'} 1 FAILED TEST So it seems like we collected some weird character in the multitab traces.
,
Jul 20
Thanks! I'm going to try to just re-do the recording in that CL.
,
Jul 20
Tried re-recording on two different machines, I still get the same issue. I don't have the cycles to dig into the weird character in the trace at the moment.
,
Jul 20
Annie: can you move #7 to issue 828917? This should be fixed with https://chromium-review.googlesource.com/c/catapult/+/1145042 landing now. Feel free to reopen if you still see this
,
Jul 20
Yep, moving #7.
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/213eaf4f79bcdfc088cb7a6512948b7777403c7f commit 213eaf4f79bcdfc088cb7a6512948b7777403c7f Author: Nghia Nguyen <nednguyen@google.com> Date: Fri Jul 20 14:53:05 2018 Make sure Telemetry log error recorded by results failure There are two ways error with benchmark running are logged on Telemetry: (1) Python exception stack are directly logged by call sites (see example in https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py?rcl=d26f5c315d5ed3e9389ea209a51611f979a18db7&l=100) (2) Call sites create their own exception message and invokes results.Fail(error_message) This ensures that results.Fail(..) always log error message created by call site. This is the safest way to ensure that we never miss logging error by mistakes from call sites. The cons of this approach is sometimes client may mistakenly log error twice, but such error is less severe and easier to fix than missing error log. Bug: chromium:865725 Change-Id: I88608306cde54e5e6901c3caa9c8a053a4c4d243 Reviewed-on: https://chromium-review.googlesource.com/1145042 Commit-Queue: Ned Nguyen <nednguyen@google.com> Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> [modify] https://crrev.com/213eaf4f79bcdfc088cb7a6512948b7777403c7f/telemetry/telemetry/internal/results/page_test_results.py [modify] https://crrev.com/213eaf4f79bcdfc088cb7a6512948b7777403c7f/telemetry/telemetry/internal/story_runner.py [modify] https://crrev.com/213eaf4f79bcdfc088cb7a6512948b7777403c7f/telemetry/telemetry/internal/story_runner_unittest.py
,
Jul 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b241b3ee47b8f78e72ec61220aae5b5e4957a5a2 commit b241b3ee47b8f78e72ec61220aae5b5e4957a5a2 Author: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Date: Fri Jul 20 17:43:15 2018 Roll src/third_party/catapult 58189855d207..213eaf4f79bc (1 commits) https://chromium.googlesource.com/catapult.git/+log/58189855d207..213eaf4f79bc git log 58189855d207..213eaf4f79bc --date=short --no-merges --format='%ad %ae %s' 2018-07-20 nednguyen@google.com Make sure Telemetry log error recorded by results failure Created with: gclient setdep -r src/third_party/catapult@213eaf4f79bc The AutoRoll server is located here: https://catapult-roll.skia.org Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel BUG= chromium:865725 TBR=sullivan@chromium.org Change-Id: I76b5afa08638ea9431056a0a373c92edbdabd6e0 Reviewed-on: https://chromium-review.googlesource.com/1145441 Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#576914} [modify] https://crrev.com/b241b3ee47b8f78e72ec61220aae5b5e4957a5a2/DEPS
,
Jan 16
(6 days ago)
,
Jan 16
(6 days ago)
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by sullivan@chromium.org
, Jul 19