Issue metadata
Sign in to add a comment
|
Fix test result output for run_gtest_perf_test.py |
||||||||||||||||||||||||
Issue descriptionTests run by https://cs.chromium.org/chromium/src/testing/scripts/run_gtest_perf_test.py don't output correct test results data. This causes test retries to be disabled I think, because it gives invalid test result data. I'm not sure exactly how it's invalid, but the recipe seems to think so. I do know that it doesn't give any individual tests to which failed, which the recipe should know to retry; that work is in bug 533481 . They also should be able to output valid test result data, since the tests they're running are gtests, which I think are capable of output-ing structured test result data. Here's an example build: https://ci.chromium.org/swarming/task/3e3cb14acdf87610?server=chromium-swarm.appspot.com Test results for components_perftests from that build: https://logs.chromium.org/v/?s=chromium%2Fled%2Fmartiniss_google.com%2F90e402f9428336f2581049eb26fc6dce09841e24b23a7e474b5dcf11b01e5235%2F%2B%2Fsteps%2Fcomponents_perftests__with_patch_%2F0%2Flogs%2Fjson.output%2F0
,
Jun 21 2018
As an aside, I tried running the test locally with './Debug_x64/components_perftests --gtest_output=json:test_output.json --gtest_filter=DiscardableSharedMemoryHeapTest.SearchFreeLists', but test_output.json gave me an xml file instead, which is strange...
,
Jun 22 2018
Right, these tests don't generate valid json test results so we have always created this result in run_gtest_perf_test.py and it doesn't get uploaded to the test results server:https://cs.chromium.org/chromium/src/testing/scripts/run_gtest_perf_test.py?q=run_gtest&sq=package:chromium&g=0&l=152 I am not familiar enough with the non-telemetry tests themselves, but I know for the perf results we actually redirect to stdout to a file to parse the results. Could we somehow re-write this here to be valid json since it is just one test each?
,
Jun 22 2018
I can rewrite it to be valid json. https://chromium-review.googlesource.com/c/chromium/src/+/1112635 is a attempt at this Technically it be more than one test each, since several of the test suites that this script runs have more than one test. https://chromium-swarm.appspot.com/task?id=3e0f53f2c04f7510&refresh=10&show_raw=1 for example, has several tests which fail. We really do want the ability to target individual tests in a test suite, which we can't do if we do what you suggest in test #3.
,
Jun 22 2018
Are the keys of the perf test output the tests run? E.g. the keys in this json object: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=5685733cf2434a0f6e7e0d6920ef9997b2f9eb06&as=perftest-output.json If that's the case, I can at least get a list of tests run. Not sure if I can figure out which of them failed.
,
Jun 22 2018
Also, in regards to #2, it looks like this is due to gtest running in multiple processes? Looks like our existing recipes which talk to gtests don't use that argument. They use --test-launcher-summary-output I could add support for parsing that, which already exists in https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/util.py?q=test_utils/uti&sq=package:chromium&g=0&l=189. Copy a version of that over to //testing/scripts/common.py and use that in run_gtest_perf_test.py to get a list of failed tests that we can return. It feels a bit hacky, but I think it could work.
,
Jun 25 2018
You are right that the test suite has multiple tests, I have never try to run one individually before so I can't comment on the xml file you received. I have added Charlie Harrision who is the owner of this test and maybe he can comment more. As far as #5 goes, I think they use an abbreviated name here. If you search for "search_free_list" from the perf json in the task https://chromium-swarm.appspot.com/task?id=3e0f53f2c04f7510&refresh=10&show_raw=1, you see it actually corresponds to DiscardableSharedMemoryHeapTest.SearchFreeLists which is the actual test. As far as #6 with the recipe goes, we are still launching these as IsolatedScript Tests. I am not sure if that makes a difference here since I am not familiar with the --test-launcher-summary-output flag. What makes you say they are running in multiple processes?
,
Jun 25 2018
The multiple processes thing might just be because i'm running them locally. Regardless, the --gtest_output doesn't seem to matter anymore, since I found the --test-launcher-summary-output flag. I was able to, locally, run components_perftests, force a test to fail, see that test in a json file, and re-run only that test. https://chromium-review.googlesource.com/c/chromium/src/+/1112635 should make this work for our recipes.
,
Jun 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3f38afc82b856e94d8f2d25334eee942f8255132 commit 3f38afc82b856e94d8f2d25334eee942f8255132 Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Jun 27 02:07:25 2018 //testing/scripts: Add gtest result parsing Adds a function to common.py which parses gtest results. Right now it only lists which tests fail, but it could be augmented in the future as needed. Bug: 855234 Change-Id: I327042f93683c429d46685225098e366e9de0cda Reviewed-on: https://chromium-review.googlesource.com/1112738 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#570632} [modify] https://crrev.com/3f38afc82b856e94d8f2d25334eee942f8255132/testing/scripts/common.py
,
Jun 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Jun 27 23:53:45 2018 gtest perf tests: Output well formed test results json This CL changes the run_gtest_perf_test.py script to output well formed test results json. Bug: 855234 Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 Reviewed-on: https://chromium-review.googlesource.com/1112635 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#570943} [modify] https://crrev.com/71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb/testing/scripts/run_gtest_perf_test.py
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9eeaba4e66b95a174d03ecc8caadb64fa3dcb890 commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890 Author: Ned Nguyen <nednguyen@google.com> Date: Thu Jun 28 11:42:00 2018 Revert "gtest perf tests: Output well formed test results json" This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) Original change's description: > gtest perf tests: Output well formed test results json > > This CL changes the run_gtest_perf_test.py script to output well formed > test results json. > > Bug: 855234 > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > Reviewed-on: https://chromium-review.googlesource.com/1112635 > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > Cr-Commit-Position: refs/heads/master@{#570943} TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1118238 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#571083} [modify] https://crrev.com/9eeaba4e66b95a174d03ecc8caadb64fa3dcb890/testing/scripts/run_gtest_perf_test.py
,
Jun 28 2018
My CL got reverted because it broke on some perf tests I didn't know about. It looks like they're somehow using a different version of googletest than the test suites I was testing on. They don't have the --test-launcher-summary-output option. I'm not sure why, ned or someone else, do you know why? Compare 'net_perftests' to 'components_perftests'. I looked at the BUILD entry for 'net_perftests' but I didn't see anything, although I don't know BUILD files that well.
,
Jun 28 2018
It looks like somehow net_perftests is not using https://cs.chromium.org/chromium/src/base/test/launcher/unit_test_launcher.cc?sq=package:chromium&g=0. I don't know enough to know why. I think I can reland this change, as long as I test for the presence of the result file, and correctly handle the case where it doesn't exist. As far as I can tell, the test suites on CQ all properly generate this file, so this should work fine.
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f02c92e1ae6b6b466f4d42b3d870ed070bb28b37 commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37 Author: Stephen Martinis <martiniss@chromium.org> Date: Thu Jun 28 20:27:38 2018 Reland "gtest perf tests: Output well formed test results json" This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890. Reason for revert: Will re-land with fix Original change's description: > Revert "gtest perf tests: Output well formed test results json" > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) > > Original change's description: > > gtest perf tests: Output well formed test results json > > > > This CL changes the run_gtest_perf_test.py script to output well formed > > test results json. > > > > Bug: 855234 > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > > Reviewed-on: https://chromium-review.googlesource.com/1112635 > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#570943} > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: 855234 > Reviewed-on: https://chromium-review.googlesource.com/1118238 > Reviewed-by: Ned Nguyen <nednguyen@google.com> > Commit-Queue: Ned Nguyen <nednguyen@google.com> > Cr-Commit-Position: refs/heads/master@{#571083} TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1118789 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#571245} [modify] https://crrev.com/f02c92e1ae6b6b466f4d42b3d870ed070bb28b37/testing/scripts/run_gtest_perf_test.py
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/89c5e7909f22e5cb9946f0cb7fcedf5494080d2b commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b Author: Ned Nguyen <nednguyen@google.com> Date: Thu Jun 28 23:20:16 2018 Revert "Reland "gtest perf tests: Output well formed test results json"" This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37. Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315 Original change's description: > Reland "gtest perf tests: Output well formed test results json" > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890. > > Reason for revert: Will re-land with fix > > Original change's description: > > Revert "gtest perf tests: Output well formed test results json" > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) > > > > Original change's description: > > > gtest perf tests: Output well formed test results json > > > > > > This CL changes the run_gtest_perf_test.py script to output well formed > > > test results json. > > > > > > Bug: 855234 > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > > > Reviewed-on: https://chromium-review.googlesource.com/1112635 > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > Cr-Commit-Position: refs/heads/master@{#570943} > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac > > No-Presubmit: true > > No-Tree-Checks: true > > No-Try: true > > Bug: 855234 > > Reviewed-on: https://chromium-review.googlesource.com/1118238 > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > Cr-Commit-Position: refs/heads/master@{#571083} > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: 855234 > Reviewed-on: https://chromium-review.googlesource.com/1118789 > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > Cr-Commit-Position: refs/heads/master@{#571245} TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1119445 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#571321} [modify] https://crrev.com/89c5e7909f22e5cb9946f0cb7fcedf5494080d2b/testing/scripts/run_gtest_perf_test.py
,
Jul 2
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4da4a0c81701d7acf80bd594b6bc00efece0e898 commit 4da4a0c81701d7acf80bd594b6bc00efece0e898 Author: Stephen Martinis <martiniss@chromium.org> Date: Mon Jul 02 19:36:13 2018 Reland "Reland "gtest perf tests: Output well formed test results json"" This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b. Reason for revert: Testing possible fixes Original change's description: > Revert "Reland "gtest perf tests: Output well formed test results json"" > > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37. > > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315 > > Original change's description: > > Reland "gtest perf tests: Output well formed test results json" > > > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890. > > > > Reason for revert: Will re-land with fix > > > > Original change's description: > > > Revert "gtest perf tests: Output well formed test results json" > > > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. > > > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) > > > > > > Original change's description: > > > > gtest perf tests: Output well formed test results json > > > > > > > > This CL changes the run_gtest_perf_test.py script to output well formed > > > > test results json. > > > > > > > > Bug: 855234 > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635 > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > > Cr-Commit-Position: refs/heads/master@{#570943} > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac > > > No-Presubmit: true > > > No-Tree-Checks: true > > > No-Try: true > > > Bug: 855234 > > > Reviewed-on: https://chromium-review.googlesource.com/1118238 > > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > > Cr-Commit-Position: refs/heads/master@{#571083} > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2 > > No-Presubmit: true > > No-Tree-Checks: true > > No-Try: true > > Bug: 855234 > > Reviewed-on: https://chromium-review.googlesource.com/1118789 > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#571245} > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: 855234 > Reviewed-on: https://chromium-review.googlesource.com/1119445 > Reviewed-by: Ned Nguyen <nednguyen@google.com> > Commit-Queue: Ned Nguyen <nednguyen@google.com> > Cr-Commit-Position: refs/heads/master@{#571321} TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1120798 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#571968} [modify] https://crrev.com/4da4a0c81701d7acf80bd594b6bc00efece0e898/testing/scripts/common.py [modify] https://crrev.com/4da4a0c81701d7acf80bd594b6bc00efece0e898/testing/scripts/run_gtest_perf_test.py [modify] https://crrev.com/4da4a0c81701d7acf80bd594b6bc00efece0e898/testing/scripts/run_performance_tests.py
,
Jul 3
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3cf9eadffa44ab79c557384ee166ef28bb071d25 commit 3cf9eadffa44ab79c557384ee166ef28bb071d25 Author: Emily Hanley <eyaich@chromium.org> Date: Tue Jul 03 12:58:25 2018 Revert "Reland "Reland "gtest perf tests: Output well formed test results json""" This reverts commit 4da4a0c81701d7acf80bd594b6bc00efece0e898. Reason for revert: Failing on mac and windows. I will add more comments to crbug.com/855234 Original change's description: > Reland "Reland "gtest perf tests: Output well formed test results json"" > > This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b. > > Reason for revert: Testing possible fixes > > Original change's description: > > Revert "Reland "gtest perf tests: Output well formed test results json"" > > > > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37. > > > > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315 > > > > Original change's description: > > > Reland "gtest perf tests: Output well formed test results json" > > > > > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890. > > > > > > Reason for revert: Will re-land with fix > > > > > > Original change's description: > > > > Revert "gtest perf tests: Output well formed test results json" > > > > > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. > > > > > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) > > > > > > > > Original change's description: > > > > > gtest perf tests: Output well formed test results json > > > > > > > > > > This CL changes the run_gtest_perf_test.py script to output well formed > > > > > test results json. > > > > > > > > > > Bug: 855234 > > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635 > > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > > > Cr-Commit-Position: refs/heads/master@{#570943} > > > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac > > > > No-Presubmit: true > > > > No-Tree-Checks: true > > > > No-Try: true > > > > Bug: 855234 > > > > Reviewed-on: https://chromium-review.googlesource.com/1118238 > > > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > > > Cr-Commit-Position: refs/heads/master@{#571083} > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2 > > > No-Presubmit: true > > > No-Tree-Checks: true > > > No-Try: true > > > Bug: 855234 > > > Reviewed-on: https://chromium-review.googlesource.com/1118789 > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > > > Cr-Commit-Position: refs/heads/master@{#571245} > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86 > > No-Presubmit: true > > No-Tree-Checks: true > > No-Try: true > > Bug: 855234 > > Reviewed-on: https://chromium-review.googlesource.com/1119445 > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > Cr-Commit-Position: refs/heads/master@{#571321} > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: 855234 > Reviewed-on: https://chromium-review.googlesource.com/1120798 > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Cr-Commit-Position: refs/heads/master@{#571968} TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org Change-Id: I031c5d6204e0b53fa2719168a38292455d676167 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1124539 Reviewed-by: Emily Hanley <eyaich@chromium.org> Commit-Queue: Emily Hanley <eyaich@chromium.org> Cr-Commit-Position: refs/heads/master@{#572183} [modify] https://crrev.com/3cf9eadffa44ab79c557384ee166ef28bb071d25/testing/scripts/common.py [modify] https://crrev.com/3cf9eadffa44ab79c557384ee166ef28bb071d25/testing/scripts/run_gtest_perf_test.py [modify] https://crrev.com/3cf9eadffa44ab79c557384ee166ef28bb071d25/testing/scripts/run_performance_tests.py
,
Jul 3
I reverted because this is failing on mac and windows: copmonents_perftests on windows 7 https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20Perf/builds/2700 views perftests on mac: https://uberchromegw.corp.google.com/i/chromium.perf/builders/mac-10_13_laptop_high_end-perf/builds/225 there are others on mac and windows.
,
Jul 3
The number of revert & reland are a bit too high. Stephen: I suggest you look into how to try run your change with pinpoint.
,
Jul 3
,
Jul 4
I've deployed a dev version that does C++ perf tests instead of Telemetry tests. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/
,
Jul 4
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12dc88d8a40000
,
Jul 4
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12dc88d8a40000 All of the runs failed. The most common error (10/20 runs) was: HTTPException: HTTP status code 400: {"error": {"message": "isolatedserver must be valid HTTPS URL, not chrome-isolated.appspot.com"}}
,
Jul 4
I was able to use the isolate provided by the pinpoint job to make this work. Ran the test and got an output, https://chrome-swarming.appspot.com/task?id=3e7bb0293bfee910&refresh=10&show_raw=1, which looks promising, although still has an error. It looks like it's harder to distinguish if a C++ test suite is "chromium" based or a regular gtest. Looks like we'll need a whitelist, which is what I was testing in the CL. I'll look at this more next week when I'm in office.
,
Jul 9
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14bc03faa40000
,
Jul 9
📍 Job complete. See results below. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14bc03faa40000
,
Jul 9
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14f8a3faa40000
,
Jul 10
📍 Job complete. See results below. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14f8a3faa40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ae9fb2a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/1481db14a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12bbd662a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ae6562a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/16f4b2dca40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ead562a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/178e2d14a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14cd921ca40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12e22162a40000
,
Jul 10
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/121f9ddaa40000
,
Jul 10
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14cd921ca40000 Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
,
Jul 10
📍 Job complete. See results below. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12bbd662a40000
,
Jul 10
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/121f9ddaa40000 Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
,
Jul 11
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/1481db14a40000 Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
,
Jul 11
📍 Job complete. See results below. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ead562a40000
,
Jul 11
📍 Job complete. See results below. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/16f4b2dca40000
,
Jul 11
📍 Job complete. See results below. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/178e2d14a40000
,
Jul 11
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ae6562a40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 11
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ae9fb2a40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 11
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12e22162a40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 11
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14cc62c2a40000
,
Jul 11
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/119dff1aa40000
,
Jul 12
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14cc62c2a40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 12
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/119dff1aa40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 12
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/16a2bec6a40000
,
Jul 12
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/17effa56a40000
,
Jul 12
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/17effa56a40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 12
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14a0966ea40000
,
Jul 12
📍 Pinpoint job started. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14f65e31a40000
,
Jul 12
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/16a2bec6a40000 All of the runs failed. The most common error (10/20 runs) was: SwarmingTestError: The test failed. No Python exception was found in the log.
,
Jul 12
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14a0966ea40000 All of the runs failed. The most common error (20/20 runs) was: Exception: Shard indicators must be present for perf tests
,
Jul 12
😿 Pinpoint job stopped with an error. https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14f65e31a40000 All of the runs failed. The most common error (20/20 runs) was: Exception: Shard indicators must be present for perf tests
,
Jul 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f53c3ea9907464bf25fef2add0536e402a129851 commit f53c3ea9907464bf25fef2add0536e402a129851 Author: Stephen Martinis <martiniss@chromium.org> Date: Tue Jul 17 19:10:57 2018 Reland #3 of "gtest perf tests: Output well formed test results json" This reverts commit 3cf9eadffa44ab79c557384ee166ef28bb071d25. Reason for revert: Will fix errors Original change's description: > Revert "Reland "Reland "gtest perf tests: Output well formed test results json""" > > This reverts commit 4da4a0c81701d7acf80bd594b6bc00efece0e898. > > Reason for revert: Failing on mac and windows. I will add more comments to crbug.com/855234 > > Original change's description: > > Reland "Reland "gtest perf tests: Output well formed test results json"" > > > > This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b. > > > > Reason for revert: Testing possible fixes > > > > Original change's description: > > > Revert "Reland "gtest perf tests: Output well formed test results json"" > > > > > > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37. > > > > > > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315 > > > > > > Original change's description: > > > > Reland "gtest perf tests: Output well formed test results json" > > > > > > > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890. > > > > > > > > Reason for revert: Will re-land with fix > > > > > > > > Original change's description: > > > > > Revert "gtest perf tests: Output well formed test results json" > > > > > > > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. > > > > > > > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) > > > > > > > > > > Original change's description: > > > > > > gtest perf tests: Output well formed test results json > > > > > > > > > > > > This CL changes the run_gtest_perf_test.py script to output well formed > > > > > > test results json. > > > > > > > > > > > > Bug: 855234 > > > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > > > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635 > > > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > > > > Cr-Commit-Position: refs/heads/master@{#570943} > > > > > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac > > > > > No-Presubmit: true > > > > > No-Tree-Checks: true > > > > > No-Try: true > > > > > Bug: 855234 > > > > > Reviewed-on: https://chromium-review.googlesource.com/1118238 > > > > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > > > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > > > > Cr-Commit-Position: refs/heads/master@{#571083} > > > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2 > > > > No-Presubmit: true > > > > No-Tree-Checks: true > > > > No-Try: true > > > > Bug: 855234 > > > > Reviewed-on: https://chromium-review.googlesource.com/1118789 > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > > > > Cr-Commit-Position: refs/heads/master@{#571245} > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86 > > > No-Presubmit: true > > > No-Tree-Checks: true > > > No-Try: true > > > Bug: 855234 > > > Reviewed-on: https://chromium-review.googlesource.com/1119445 > > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > > Cr-Commit-Position: refs/heads/master@{#571321} > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218 > > No-Presubmit: true > > No-Tree-Checks: true > > No-Try: true > > Bug: 855234 > > Reviewed-on: https://chromium-review.googlesource.com/1120798 > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#571968} > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > Change-Id: I031c5d6204e0b53fa2719168a38292455d676167 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: 855234 > Reviewed-on: https://chromium-review.googlesource.com/1124539 > Reviewed-by: Emily Hanley <eyaich@chromium.org> > Commit-Queue: Emily Hanley <eyaich@chromium.org> > Cr-Commit-Position: refs/heads/master@{#572183} Change-Id: I6c994148908ed9a223b8b59219b6b4a6957cd9ca Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1125420 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#575747} [modify] https://crrev.com/f53c3ea9907464bf25fef2add0536e402a129851/testing/scripts/common.py [modify] https://crrev.com/f53c3ea9907464bf25fef2add0536e402a129851/testing/scripts/run_gtest_perf_test.py [modify] https://crrev.com/f53c3ea9907464bf25fef2add0536e402a129851/testing/scripts/run_performance_tests.py
,
Jul 17
The CL seems to have landed ok. I'm looking at the perf bots and everything seems to be fine. I'll keep monitoring.
,
Jul 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0933c720c2c368df9988476875afbac2e2600bf8 commit 0933c720c2c368df9988476875afbac2e2600bf8 Author: Ned Nguyen <nednguyen@google.com> Date: Wed Jul 18 12:20:47 2018 Revert "Reland #3 of "gtest perf tests: Output well formed test results json"" This reverts commit f53c3ea9907464bf25fef2add0536e402a129851. Reason for revert: break media_perftest ( crbug.com/864942 ) Now I look at the code, I also have concerns about the approach of the CL. Please discuss with me on the approach to implement the feature. Original change's description: > Reland #3 of "gtest perf tests: Output well formed test results json" > > This reverts commit 3cf9eadffa44ab79c557384ee166ef28bb071d25. > > Reason for revert: Will fix errors > > Original change's description: > > Revert "Reland "Reland "gtest perf tests: Output well formed test results json""" > > > > This reverts commit 4da4a0c81701d7acf80bd594b6bc00efece0e898. > > > > Reason for revert: Failing on mac and windows. I will add more comments to crbug.com/855234 > > > > Original change's description: > > > Reland "Reland "gtest perf tests: Output well formed test results json"" > > > > > > This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b. > > > > > > Reason for revert: Testing possible fixes > > > > > > Original change's description: > > > > Revert "Reland "gtest perf tests: Output well formed test results json"" > > > > > > > > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37. > > > > > > > > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315 > > > > > > > > Original change's description: > > > > > Reland "gtest perf tests: Output well formed test results json" > > > > > > > > > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890. > > > > > > > > > > Reason for revert: Will re-land with fix > > > > > > > > > > Original change's description: > > > > > > Revert "gtest perf tests: Output well formed test results json" > > > > > > > > > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb. > > > > > > > > > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307) > > > > > > > > > > > > Original change's description: > > > > > > > gtest perf tests: Output well formed test results json > > > > > > > > > > > > > > This CL changes the run_gtest_perf_test.py script to output well formed > > > > > > > test results json. > > > > > > > > > > > > > > Bug: 855234 > > > > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796 > > > > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635 > > > > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > > > > > Cr-Commit-Position: refs/heads/master@{#570943} > > > > > > > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > > > > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac > > > > > > No-Presubmit: true > > > > > > No-Tree-Checks: true > > > > > > No-Try: true > > > > > > Bug: 855234 > > > > > > Reviewed-on: https://chromium-review.googlesource.com/1118238 > > > > > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > > > > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > > > > > Cr-Commit-Position: refs/heads/master@{#571083} > > > > > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > > > > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2 > > > > > No-Presubmit: true > > > > > No-Tree-Checks: true > > > > > No-Try: true > > > > > Bug: 855234 > > > > > Reviewed-on: https://chromium-review.googlesource.com/1118789 > > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > > > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > > > > > Cr-Commit-Position: refs/heads/master@{#571245} > > > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > > > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86 > > > > No-Presubmit: true > > > > No-Tree-Checks: true > > > > No-Try: true > > > > Bug: 855234 > > > > Reviewed-on: https://chromium-review.googlesource.com/1119445 > > > > Reviewed-by: Ned Nguyen <nednguyen@google.com> > > > > Commit-Queue: Ned Nguyen <nednguyen@google.com> > > > > Cr-Commit-Position: refs/heads/master@{#571321} > > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > > > Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218 > > > No-Presubmit: true > > > No-Tree-Checks: true > > > No-Try: true > > > Bug: 855234 > > > Reviewed-on: https://chromium-review.googlesource.com/1120798 > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > > > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > > > Cr-Commit-Position: refs/heads/master@{#571968} > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org > > > > Change-Id: I031c5d6204e0b53fa2719168a38292455d676167 > > No-Presubmit: true > > No-Tree-Checks: true > > No-Try: true > > Bug: 855234 > > Reviewed-on: https://chromium-review.googlesource.com/1124539 > > Reviewed-by: Emily Hanley <eyaich@chromium.org> > > Commit-Queue: Emily Hanley <eyaich@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#572183} > > Change-Id: I6c994148908ed9a223b8b59219b6b4a6957cd9ca > Bug: 855234 > Reviewed-on: https://chromium-review.googlesource.com/1125420 > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Reviewed-by: Dirk Pranke <dpranke@chromium.org> > Reviewed-by: Stephen Martinis <martiniss@chromium.org> > Cr-Commit-Position: refs/heads/master@{#575747} TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org Change-Id: I250655ba8a77cbee8ea46d7471c4691e761fa88e No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 855234 Reviewed-on: https://chromium-review.googlesource.com/1141824 Reviewed-by: Ned Nguyen <nednguyen@google.com> Commit-Queue: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#576013} [modify] https://crrev.com/0933c720c2c368df9988476875afbac2e2600bf8/testing/scripts/common.py [modify] https://crrev.com/0933c720c2c368df9988476875afbac2e2600bf8/testing/scripts/run_gtest_perf_test.py [modify] https://crrev.com/0933c720c2c368df9988476875afbac2e2600bf8/testing/scripts/run_performance_tests.py
,
Jul 20
,
Jul 20
,
Jul 20
Ned said that he would work on this. It's not blocking the other bug anymore I think.
,
Jul 21
|
|||||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||||
Comment 1 by martiniss@chromium.org
, Jun 21 2018Cc: eyaich@chromium.org nednguyen@chromium.org
Components: Speed>Benchmarks
Labels: -Pri-3 Pri-2
Status: Untriaged (was: Unconfirmed)