Before r572268 landed, a large number of performance tests were failing on Android bots, e.g.:
https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus5%20Perf/1945 ~ 45 perf tests failed
https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus5%20Perf/1946 - only "the one" performance_test_suite suite failed.
How can I now debug and find information about which tests failed, and their logs?
The outputs (e.g stdout, swarming.summary) of this step are now (understandably) huge, so it's hard to find anything in there.
Also the "Merge script log" is showing an exception message:
Traceback (most recent call last):
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/process_perf_results.py", line 518, in <module>
sys.exit(main())
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/process_perf_results.py", line 514, in main
args.smoke_test_mode)
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/process_perf_results.py", line 280, in _process_perf_results
_merge_json_output(output_json, test_results_list, extra_links)
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/process_perf_results.py", line 109, in _merge_json_output
merged_results = results_merger.merge_test_results(jsons_to_merge)
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/core/results_merger.py", line 72, in merge_test_results
return _merge_json_test_result_format(shard_results_list)
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/core/results_merger.py", line 145, in _merge_json_test_result_format
merge('tests', merge_tries)
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/core/results_merger.py", line 141, in <lambda>
result_json, merged_results, key, merge_func)
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/core/results_merger.py", line 264, in merge_value
dest[key] = merge_func(source[key], dest[key])
File "/b/c/b/Android_Nexus5_Perf/src/tools/perf/core/results_merger.py", line 214, in merge_tries
prefix, k, v, curr_node, dest_node))
core.results_merger.MergeException: MergeFailure for tests
:loading.mobile:Locanto:actual: u'PASS' not mergable, curr_node: {u'actual': u'PASS', u'artifacts': {u'logs': [u'https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/cff486fc-7ef4-11e8-a8e6-0242ac110005', u'https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/d0c8b2f6-7ef4-11e8-a8e6-0242ac110005']}, u'is_unexpected': False, u'times': [30.875025033950806, 16.667243003845215], u'time': 30.875025033950806, u'expected': u'PASS'}
dest_node: {u'expected': u'PASS', u'is_unexpected': True, u'actual': u'SKIP', u'time': None, u'times': [None, None]}
Good! I think on the builds I looked there wasn't anything there because of the merge problem. But I see this is now fixed and the json.output is indeed useful. Thanks!
Pointed to this by the comment at the top of https://cs.chromium.org/chromium/build/scripts/slave/results_dashboard.py.
Are there any plans to move users of the deprecated build-side script to the src-side script? Anything that uses //build/scripts/slave/runtest.py that uploads to the dashboard (e.g. the sizes method in the Chromium recipe) is using the build-side script, which is causing some issues with LUCI migration.
runtest.py is currently only setup to upload chartjson, and while the build-side upload script can upload histograms, it'd be better to use the src-side script since it automagically handles oauth on LUCI.
Unfortunately, we can't just continue using chartjson with IP whitelisting during the migration since the IPs of the machines are going to change on a fairly regular basis.
So I know there is work underway for webrtc: crbug.com/862134
I don't think we ever officially went through and documented all the use cases of that script or put any plans into motion for a timeline to deprecate it.
can you add a pointer to the where the sizes script is still utilizing this code?
Anything that uses runtest.py (such as the sizes or runtest steps in the Chromium recipe API https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium/api.py?q=chromium/api.py&sq=package:chromium&dr&l=736) uses the deprecated code https://cs.chromium.org/chromium/build/scripts/slave/runtest.py?l=52.
What I'm currently hoping is that we can copy over the pretty small amount of oauth token generation code from the src-side script since the build-side script seems to support uploading histograms with an oauth token. This should be good enough to unblock LUCI migration, after which we can look into better long term solutions.
I'm currently adding histogram support to one of the tests that uses runtest.py for perf dashboard uploading so that I can actually test it out.
NO I am not actively driving this. The implementation from our side we would call complete, removing the build side scripts is what is left which I think would be a good task to enable easier maintenance going forward of this code.
I think a lot of these blocking/dependent bugs might be able to be closed given most of this work is wrapped up. I will clean that up before I remove myself as the owner.
It could be great to also clean up the src/testing/scripts scripts as well. The run_performance_test_wrapper.py script sometimes calls into run_performance_tests.py and sometimes calls into run_gtest_perf_test.py. Then again, run_performance_tests.py might call run_gtest_perf_test or it might call run_telemetry_benchmark_as_googletest.py. It's difficult to tell from the documentation what path can be removed now and what path is currently used.
Comment 1 by nedngu...@google.com
, Aug 22 2017