"Results DashboardUpload Failure" performance_test_suite |
|||
Issue description
,
Jan 17
(5 days ago)
Error:
Traceback (most recent call last):
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\bin\add_reserved_diagnostics", line 94, in <module>
sys.exit(main())
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\bin\add_reserved_diagnostics", line 82, in main
dicts, names_to_values)
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\tracing\value\diagnostics\add_reserved_diagnostics.py", line 168, in AddReservedDiagnostics
histograms.ImportDicts(dicts_across_names)
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\tracing\value\histogram_set.py", line 88, in ImportDicts
hist = histogram_module.Histogram.FromDict(d)
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\tracing\value\histogram.py", line 646, in FromDict
hist._bins[i].FromDict(bin_dct)
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\tracing\value\histogram.py", line 469, in FromDict
diagnostic_map_dict))
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\tracing\value\histogram.py", line 388, in FromDict
dm.AddDicts(dct)
File "C:\b\swarming\w\ir\cache\builder\src\third_party\catapult\tracing\tracing\value\histogram.py", line 392, in AddDicts
for name, diagnostic_dict in dct.items():
AttributeError: 'NoneType' object has no attribute 'items'
ERROR:root:Error uploading perf result of system_health.memory_desktop.reference
Traceback (most recent call last):
File "C:\b\swarming\w\ir\cache\builder\src\tools\perf\process_perf_results.py", line 425, in _upload_individual_benchmark
return _upload_individual(*params)
File "C:\b\swarming\w\ir\cache\builder\src\tools\perf\process_perf_results.py", line 414, in _upload_individual
service_account_file, oj)
File "C:\b\swarming\w\ir\cache\builder\src\tools\perf\process_perf_results.py", line 119, in _upload_perf_results
return upload_results_to_perf_dashboard.main(args)
File "C:\b\swarming\w\ir\cache\builder\src\tools\perf\core\upload_results_to_perf_dashboard.py", line 131, in main
dashboard_json = _GetDashboardHistogramData(options)
File "C:\b\swarming\w\ir\cache\builder\src\tools\perf\core\upload_results_to_perf_dashboard.py", line 83, in _GetDashboardHistogramData
perf_dashboard_machine_group=options.perf_dashboard_machine_group)
File "C:\b\swarming\w\ir\cache\builder\src\tools\perf\core\results_dashboard.py", line 183, in MakeHistogramSetWithDiagnostics
subprocess.check_call(cmd)
File "C:\b\swarming\w\ir\cipd_bin_packages\bin\Lib\subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['C:\\b\\swarming\\w\\ir\\cache\\vpython\\9649ae\\Scripts\\python.exe', 'C:\\b\\swarming\\w\\ir\\cache\\builder\\src\\third_party\\catapult\\tracing\\bin\\add_reserved_diagnostics', '--benchmarks', 'system_health.memory_desktop', '--bots', 'win-10-perf', '--builds', '1760', '--masters', 'ChromiumPerf', '--is_reference_build', 'true', '--log_urls_k', 'Buildbot stdio', '--log_urls_v', 'http://build.chromium.org/p/chromium/builders/win-10-perf/builds/1760/steps/system_health.memory_desktop/logs/stdio', '--build_urls_k', 'Build Status', '--build_urls_v', 'https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf/1760', '--webrtc_revisions', '455d27c49abfff587babed348ae90f1aba75969e', '--chromium_commit_positions', '621080', '--chromium_revisions', 'None', '--v8_revisions', '677a6110e4c5e25b35d36aba42b36741c8ce1ea2', 'c:\\b\\swarming\\w\\ir\\tmp\\t\\tmpa_tzhe\\system_health.memory_desktop.reference\\merged_perf_results.json', '--output_path', 'c:\\b\\swarming\\w\\ir\\tmp\\t\\tmp0al3qu']' returned non-zero exit status 1
from here: https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8924820682873154608/+/steps/performance_test_suite_on_Intel_GPU_on_Windows_on_Windows-10/0/logs/Merge_script_log/0
,
Jan 17
(5 days ago)
Still having trouble reproducing. The merge script log shows errors parsing results from system_health.memory_desktop.reference and memory.desktop.reference. https://logs.chromium.org/logs/chrome/buildbucket/cr-buildbucket.appspot.com/8924056898587315264/+/steps/performance_test_suite_on_Intel_GPU_on_Windows_on_Windows-10/0/logs/Merge_script_log/0 This isolate contains system_health.memory_desktop.reference: https://chrome-isolated.appspot.com/browse?namespace=default-gzip&hash=a60ece9f0502ff661340204478bc34cc85ca96f6 Here's the results: https://chrome-isolated.appspot.com/content?namespace=default-gzip&digest=b3977009b548120059aba54742dd6de81ade6601&as=system_health.memory_desktop.reference\perf_results.json I downloaded that and successfully parsed it.
,
Jan 18
(5 days ago)
I managed to repro even with the breakdown change rolled back. I also found a similar error on the JS side in merge_histograms. Current theory is that HistogramBin.addBin is somehow slipping undefined values into the "sampled stream" of sample diagnostic maps. https://github.com/catapult-project/catapult/blob/master/tracing/tracing/value/histogram.html#L96 I don't think there have been any recent changes around there, so I'm not sure. It might be worth it to try to simplify addBin/mergeSampledStreams regardless of whether it's necessary to fix this bug.
,
Jan 18
(4 days ago)
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c47537d42e1cfb7f3ff655a450fd0b026a15b33b commit c47537d42e1cfb7f3ff655a450fd0b026a15b33b Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Date: Fri Jan 18 11:33:56 2019 Roll src/third_party/catapult a0e5e3328cd7..8e77731b1363 (6 commits) https://chromium.googlesource.com/catapult.git/+log/a0e5e3328cd7..8e77731b1363 git log a0e5e3328cd7..8e77731b1363 --date=short --no-merges --format='%ad %ae %s' 2019-01-17 crouleau@chromium.org Revert "Stop splitting logcat output on carriage returns." 2019-01-17 crouleau@chromium.org Revert "Ignore zeros in Breakdowns." 2019-01-17 uwyiming@google.com Add an option in WPR replay to prefer previously unmatched responses. 2019-01-17 crouleau@chromium.org [Telemetry] Cleanup: Remove unused argument from FindAllBrowserTypes 2019-01-17 uwyiming@google.com Adding uwyiming (myself) as an owner to WPR. 2019-01-17 skym@chromium.org Stop splitting logcat output on carriage returns. Created with: gclient setdep -r src/third_party/catapult@8e77731b1363 The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel BUG=chromium:919223,chromium:919223,chromium:923125, chromium:922734 ,chromium:922828,chromium:919223 TBR=sullivan@chromium.org Change-Id: I7fc110d1697c8624cf391c634fa26204b3bef617 Reviewed-on: https://chromium-review.googlesource.com/c/1419365 Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#624077} [modify] https://crrev.com/c47537d42e1cfb7f3ff655a450fd0b026a15b33b/DEPS
,
Jan 18
(4 days ago)
The rollback fixed the issue. https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf/1855
,
Jan 18
(4 days ago)
,
Jan 19
(3 days ago)
Issue 923181 has been merged into this issue. |
|||
►
Sign in to add a comment |
|||
Comment 1 by crouleau@chromium.org
, Jan 17 (5 days ago)Status: Assigned (was: Untriaged)