In https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py?l=1614&rcl=a5450a20ac2960e01de3cd444b666e9a153567bb there's a reference to chart JSON's 'enabled' flag, which is supposed to be subsumed by JSON test results's actual/expected skip. Ashley, could you help us replace this usage of enabled with the right incantation to look at JSON test results instead?
Spoke offline with Ashley; she's uploaded https://chromium-review.googlesource.com/c/chromium/tools/build/+/646566 which seems like it should fix this problem!
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/517ee6a47d425420b7927ce50ee80fecd9c02a70 commit 517ee6a47d425420b7927ce50ee80fecd9c02a70 Author: Ashley Enstad <ashleymarie@chromium.org> Date: Thu Sep 07 18:38:46 2017 Update swarming_isolated_script_tests to utilize the new json-test-results format Also uses the new format to decide if a benchmark is disabled or not BUG= chromium:760820 Change-Id: I65e8c584f25651c3c08dbb38c7fd3b468364410c Reviewed-on: https://chromium-review.googlesource.com/646566 Commit-Queue: Ashley Enstad <ashleymarie@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Ethan Kuefner <eakuefner@chromium.org> Reviewed-by: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test_empty.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_missing_all_shards.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_not_uploading.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test_no_chartjson.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_ignore_task_failure.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_disabled.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/test_utils/test_api.py [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_harness_failure.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_isolated_chartjson_test_missing_shard.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.py [add] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_simplified_ignore_task_failure.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_max_failures.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipes/chromium.expected/dynamic_swarmed_sharded_passed_isolated_script_perf_test_disabled.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.expected/chartjson_disabled.json [modify] https://crrev.com/517ee6a47d425420b7927ce50ee80fecd9c02a70/scripts/slave/recipe_modules/chromium_tests/tests/steps/swarming_isolated_script_test.py
Looks like this is complete at this point Feel free to reopen if you disagree :)
Comment 1 by eakuefner@chromium.org
, Sep 5 2017