New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 855234 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 866124
Owner:
Last visit > 30 days ago
Closed: Jul 21
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Fix test result output for run_gtest_perf_test.py

Project Member Reported by martiniss@chromium.org, Jun 21 2018

Issue description

Tests run by https://cs.chromium.org/chromium/src/testing/scripts/run_gtest_perf_test.py don't output correct test results data. This causes test retries to be disabled I think, because it gives invalid test result data.

I'm not sure exactly how it's invalid, but the recipe seems to think so. I do know that it doesn't give any individual tests to which failed, which the recipe should know to retry; that work is in  bug 533481 .

They also should be able to output valid test result data, since the tests they're running are gtests, which I think are capable of output-ing structured test result data. 

Here's an example build: https://ci.chromium.org/swarming/task/3e3cb14acdf87610?server=chromium-swarm.appspot.com

Test results for components_perftests from that build: https://logs.chromium.org/v/?s=chromium%2Fled%2Fmartiniss_google.com%2F90e402f9428336f2581049eb26fc6dce09841e24b23a7e474b5dcf11b01e5235%2F%2B%2Fsteps%2Fcomponents_perftests__with_patch_%2F0%2Flogs%2Fjson.output%2F0
 
Blocking: 533481
Cc: eyaich@chromium.org nednguyen@chromium.org
Components: Speed>Benchmarks
Labels: -Pri-3 Pri-2
Status: Untriaged (was: Unconfirmed)
cc-ing perf people; do you know why this script always outputs no test results?
As an aside, I tried running the test locally with './Debug_x64/components_perftests --gtest_output=json:test_output.json --gtest_filter=DiscardableSharedMemoryHeapTest.SearchFreeLists', but test_output.json gave me an xml file instead, which is strange...

Comment 3 by eyaich@chromium.org, Jun 22 2018

Right, these tests don't generate valid json test results so we have always created this result in run_gtest_perf_test.py and it doesn't get uploaded to the test results server:https://cs.chromium.org/chromium/src/testing/scripts/run_gtest_perf_test.py?q=run_gtest&sq=package:chromium&g=0&l=152

I am not familiar enough with the non-telemetry tests themselves, but I know for the perf results we actually redirect to stdout to a file to parse the results.  

Could we somehow re-write this here to be valid json since it is just one test each?


I can rewrite it to be valid json. https://chromium-review.googlesource.com/c/chromium/src/+/1112635 is a attempt at this

Technically it be more than one test each, since several of the test suites that this script runs have more than one test. https://chromium-swarm.appspot.com/task?id=3e0f53f2c04f7510&refresh=10&show_raw=1 for example, has several tests which fail. We really do want the ability to target individual tests in a test suite, which we can't do if we do what you suggest in test #3. 
Are the keys of the perf test output the tests run? E.g. the keys in this json object: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=5685733cf2434a0f6e7e0d6920ef9997b2f9eb06&as=perftest-output.json

If that's the case, I can at least get a list of tests run. Not sure if I can figure out which of them failed.
Also, in regards to #2, it looks like this is due to gtest running in multiple processes? 

Looks like our existing recipes which talk to gtests don't use that argument. They use --test-launcher-summary-output

I could add support for parsing that, which already exists in https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/test_utils/util.py?q=test_utils/uti&sq=package:chromium&g=0&l=189. Copy a version of that over to //testing/scripts/common.py and use that in run_gtest_perf_test.py to get a list of failed tests that we can return. It feels a bit hacky, but I think it could work.

Comment 7 by eyaich@chromium.org, Jun 25 2018

Cc: csharrison@chromium.org
You are right that the test suite has multiple tests, I have never try to run one individually before so I can't comment on the xml file you received.  I have added Charlie Harrision who is the owner of this test and maybe he can comment more. 

As far as #5 goes, I think they use an abbreviated name here.  If you search for "search_free_list" from the perf json in the task https://chromium-swarm.appspot.com/task?id=3e0f53f2c04f7510&refresh=10&show_raw=1, you see it actually corresponds to DiscardableSharedMemoryHeapTest.SearchFreeLists which is the actual test.

As far as #6 with the recipe goes, we are still launching these as IsolatedScript Tests.  I am not sure if that makes a difference here since I am not familiar with the --test-launcher-summary-output flag.  What makes you say they are running in multiple processes?

The multiple processes thing might just be because i'm running them locally. Regardless, the --gtest_output doesn't seem to matter anymore, since I found the --test-launcher-summary-output flag.

I was able to, locally, run components_perftests, force a test to fail, see that test in a json file, and re-run only that test. https://chromium-review.googlesource.com/c/chromium/src/+/1112635 should make this work for our recipes.
Project Member

Comment 9 by bugdroid1@chromium.org, Jun 27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3f38afc82b856e94d8f2d25334eee942f8255132

commit 3f38afc82b856e94d8f2d25334eee942f8255132
Author: Stephen Martinis <martiniss@chromium.org>
Date: Wed Jun 27 02:07:25 2018

//testing/scripts: Add gtest result parsing

Adds a function to common.py which parses gtest results. Right now it
only lists which tests fail, but it could be augmented in the future as
needed.

Bug:  855234 
Change-Id: I327042f93683c429d46685225098e366e9de0cda
Reviewed-on: https://chromium-review.googlesource.com/1112738
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#570632}
[modify] https://crrev.com/3f38afc82b856e94d8f2d25334eee942f8255132/testing/scripts/common.py

Project Member

Comment 10 by bugdroid1@chromium.org, Jun 27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb

commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb
Author: Stephen Martinis <martiniss@chromium.org>
Date: Wed Jun 27 23:53:45 2018

gtest perf tests: Output well formed test results json

This CL changes the run_gtest_perf_test.py script to output well formed
test results json.

Bug:  855234 
Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
Reviewed-on: https://chromium-review.googlesource.com/1112635
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#570943}
[modify] https://crrev.com/71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb/testing/scripts/run_gtest_perf_test.py

Project Member

Comment 11 by bugdroid1@chromium.org, Jun 28 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9eeaba4e66b95a174d03ecc8caadb64fa3dcb890

commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890
Author: Ned Nguyen <nednguyen@google.com>
Date: Thu Jun 28 11:42:00 2018

Revert "gtest perf tests: Output well formed test results json"

This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.

Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)

Original change's description:
> gtest perf tests: Output well formed test results json
> 
> This CL changes the run_gtest_perf_test.py script to output well formed
> test results json.
> 
> Bug:  855234 
> Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> Reviewed-on: https://chromium-review.googlesource.com/1112635
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#570943}

TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org

Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1118238
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#571083}
[modify] https://crrev.com/9eeaba4e66b95a174d03ecc8caadb64fa3dcb890/testing/scripts/run_gtest_perf_test.py

My CL got reverted because it broke on some perf tests I didn't know about. It looks like they're somehow using a different version of googletest than the test suites I was testing on. They don't have the --test-launcher-summary-output option. I'm not sure why, ned or someone else, do you know why?

Compare 'net_perftests' to 'components_perftests'. I looked at the BUILD entry for 'net_perftests' but I didn't see anything, although I don't know BUILD files that well.
It looks like somehow net_perftests is not using https://cs.chromium.org/chromium/src/base/test/launcher/unit_test_launcher.cc?sq=package:chromium&g=0. I don't know enough to know why.

I think I can reland this change, as long as I test for the presence of the result file, and correctly handle the case where it doesn't exist. As far as I can tell, the test suites on CQ all properly generate this file, so this should work fine.
Project Member

Comment 14 by bugdroid1@chromium.org, Jun 28 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f02c92e1ae6b6b466f4d42b3d870ed070bb28b37

commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Jun 28 20:27:38 2018

Reland "gtest perf tests: Output well formed test results json"

This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890.

Reason for revert: Will re-land with fix

Original change's description:
> Revert "gtest perf tests: Output well formed test results json"
> 
> This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.
> 
> Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)
> 
> Original change's description:
> > gtest perf tests: Output well formed test results json
> > 
> > This CL changes the run_gtest_perf_test.py script to output well formed
> > test results json.
> > 
> > Bug:  855234 
> > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> > Reviewed-on: https://chromium-review.googlesource.com/1112635
> > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#570943}
> 
> TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> 
> Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  855234 
> Reviewed-on: https://chromium-review.googlesource.com/1118238
> Reviewed-by: Ned Nguyen <nednguyen@google.com>
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Cr-Commit-Position: refs/heads/master@{#571083}

TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org

Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1118789
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#571245}
[modify] https://crrev.com/f02c92e1ae6b6b466f4d42b3d870ed070bb28b37/testing/scripts/run_gtest_perf_test.py

Project Member

Comment 15 by bugdroid1@chromium.org, Jun 28 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/89c5e7909f22e5cb9946f0cb7fcedf5494080d2b

commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b
Author: Ned Nguyen <nednguyen@google.com>
Date: Thu Jun 28 23:20:16 2018

Revert "Reland "gtest perf tests: Output well formed test results json""

This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37.

Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315

Original change's description:
> Reland "gtest perf tests: Output well formed test results json"
> 
> This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890.
> 
> Reason for revert: Will re-land with fix
> 
> Original change's description:
> > Revert "gtest perf tests: Output well formed test results json"
> > 
> > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.
> > 
> > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)
> > 
> > Original change's description:
> > > gtest perf tests: Output well formed test results json
> > > 
> > > This CL changes the run_gtest_perf_test.py script to output well formed
> > > test results json.
> > > 
> > > Bug:  855234 
> > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> > > Reviewed-on: https://chromium-review.googlesource.com/1112635
> > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > > Cr-Commit-Position: refs/heads/master@{#570943}
> > 
> > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > 
> > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Bug:  855234 
> > Reviewed-on: https://chromium-review.googlesource.com/1118238
> > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > Cr-Commit-Position: refs/heads/master@{#571083}
> 
> TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> 
> Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  855234 
> Reviewed-on: https://chromium-review.googlesource.com/1118789
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#571245}

TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org

Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1119445
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#571321}
[modify] https://crrev.com/89c5e7909f22e5cb9946f0cb7fcedf5494080d2b/testing/scripts/run_gtest_perf_test.py

Project Member

Comment 16 by bugdroid1@chromium.org, Jul 2

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4da4a0c81701d7acf80bd594b6bc00efece0e898

commit 4da4a0c81701d7acf80bd594b6bc00efece0e898
Author: Stephen Martinis <martiniss@chromium.org>
Date: Mon Jul 02 19:36:13 2018

Reland "Reland "gtest perf tests: Output well formed test results json""

This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b.

Reason for revert: Testing possible fixes

Original change's description:
> Revert "Reland "gtest perf tests: Output well formed test results json""
> 
> This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37.
> 
> Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315
> 
> Original change's description:
> > Reland "gtest perf tests: Output well formed test results json"
> > 
> > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890.
> > 
> > Reason for revert: Will re-land with fix
> > 
> > Original change's description:
> > > Revert "gtest perf tests: Output well formed test results json"
> > > 
> > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.
> > > 
> > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)
> > > 
> > > Original change's description:
> > > > gtest perf tests: Output well formed test results json
> > > > 
> > > > This CL changes the run_gtest_perf_test.py script to output well formed
> > > > test results json.
> > > > 
> > > > Bug:  855234 
> > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> > > > Reviewed-on: https://chromium-review.googlesource.com/1112635
> > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > > > Cr-Commit-Position: refs/heads/master@{#570943}
> > > 
> > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > 
> > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
> > > No-Presubmit: true
> > > No-Tree-Checks: true
> > > No-Try: true
> > > Bug:  855234 
> > > Reviewed-on: https://chromium-review.googlesource.com/1118238
> > > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > > Cr-Commit-Position: refs/heads/master@{#571083}
> > 
> > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > 
> > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Bug:  855234 
> > Reviewed-on: https://chromium-review.googlesource.com/1118789
> > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#571245}
> 
> TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> 
> Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  855234 
> Reviewed-on: https://chromium-review.googlesource.com/1119445
> Reviewed-by: Ned Nguyen <nednguyen@google.com>
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Cr-Commit-Position: refs/heads/master@{#571321}

TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org

Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1120798
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#571968}
[modify] https://crrev.com/4da4a0c81701d7acf80bd594b6bc00efece0e898/testing/scripts/common.py
[modify] https://crrev.com/4da4a0c81701d7acf80bd594b6bc00efece0e898/testing/scripts/run_gtest_perf_test.py
[modify] https://crrev.com/4da4a0c81701d7acf80bd594b6bc00efece0e898/testing/scripts/run_performance_tests.py

Project Member

Comment 17 by bugdroid1@chromium.org, Jul 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3cf9eadffa44ab79c557384ee166ef28bb071d25

commit 3cf9eadffa44ab79c557384ee166ef28bb071d25
Author: Emily Hanley <eyaich@chromium.org>
Date: Tue Jul 03 12:58:25 2018

Revert "Reland "Reland "gtest perf tests: Output well formed test results json"""

This reverts commit 4da4a0c81701d7acf80bd594b6bc00efece0e898.

Reason for revert: Failing on mac and windows.  I will add more comments to  crbug.com/855234 

Original change's description:
> Reland "Reland "gtest perf tests: Output well formed test results json""
> 
> This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b.
> 
> Reason for revert: Testing possible fixes
> 
> Original change's description:
> > Revert "Reland "gtest perf tests: Output well formed test results json""
> > 
> > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37.
> > 
> > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315
> > 
> > Original change's description:
> > > Reland "gtest perf tests: Output well formed test results json"
> > > 
> > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890.
> > > 
> > > Reason for revert: Will re-land with fix
> > > 
> > > Original change's description:
> > > > Revert "gtest perf tests: Output well formed test results json"
> > > > 
> > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.
> > > > 
> > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)
> > > > 
> > > > Original change's description:
> > > > > gtest perf tests: Output well formed test results json
> > > > > 
> > > > > This CL changes the run_gtest_perf_test.py script to output well formed
> > > > > test results json.
> > > > > 
> > > > > Bug:  855234 
> > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635
> > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > > > > Cr-Commit-Position: refs/heads/master@{#570943}
> > > > 
> > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > > 
> > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
> > > > No-Presubmit: true
> > > > No-Tree-Checks: true
> > > > No-Try: true
> > > > Bug:  855234 
> > > > Reviewed-on: https://chromium-review.googlesource.com/1118238
> > > > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > > > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > > > Cr-Commit-Position: refs/heads/master@{#571083}
> > > 
> > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > 
> > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2
> > > No-Presubmit: true
> > > No-Tree-Checks: true
> > > No-Try: true
> > > Bug:  855234 
> > > Reviewed-on: https://chromium-review.googlesource.com/1118789
> > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> > > Cr-Commit-Position: refs/heads/master@{#571245}
> > 
> > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > 
> > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Bug:  855234 
> > Reviewed-on: https://chromium-review.googlesource.com/1119445
> > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > Cr-Commit-Position: refs/heads/master@{#571321}
> 
> TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> 
> Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  855234 
> Reviewed-on: https://chromium-review.googlesource.com/1120798
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#571968}

TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org

Change-Id: I031c5d6204e0b53fa2719168a38292455d676167
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1124539
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Commit-Queue: Emily Hanley <eyaich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#572183}
[modify] https://crrev.com/3cf9eadffa44ab79c557384ee166ef28bb071d25/testing/scripts/common.py
[modify] https://crrev.com/3cf9eadffa44ab79c557384ee166ef28bb071d25/testing/scripts/run_gtest_perf_test.py
[modify] https://crrev.com/3cf9eadffa44ab79c557384ee166ef28bb071d25/testing/scripts/run_performance_tests.py

I reverted because this is failing on mac and windows:

copmonents_perftests on windows 7
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Win%207%20Perf/builds/2700

views perftests on mac: 
https://uberchromegw.corp.google.com/i/chromium.perf/builders/mac-10_13_laptop_high_end-perf/builds/225

there are others on mac and windows.
Cc: dtu@chromium.org
The number of revert & reland are a bit too high. Stephen: I suggest you look into how to try run your change with pinpoint.
Owner: martiniss@chromium.org
Status: Assigned (was: Untriaged)
I've deployed a dev version that does C++ perf tests instead of Telemetry tests.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12dc88d8a40000

All of the runs failed. The most common error (10/20 runs) was:
HTTPException: HTTP status code 400: {"error": {"message": "isolatedserver must be valid HTTPS URL, not chrome-isolated.appspot.com"}}
I was able to use the isolate provided by the pinpoint job to make this work. Ran the test and got an output, https://chrome-swarming.appspot.com/task?id=3e7bb0293bfee910&refresh=10&show_raw=1, which looks promising, although still has an error.

It looks like it's harder to distinguish if a C++ test suite is "chromium" based or a regular gtest. Looks like we'll need a whitelist, which is what I was testing in the CL. I'll look at this more next week when I'm in office.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14cd921ca40000

Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/121f9ddaa40000

Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/1481db14a40000

Buildbucket says the build completed successfully, but Pinpoint can't find the isolate hash.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ae6562a40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14ae9fb2a40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/12e22162a40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14cc62c2a40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/119dff1aa40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/17effa56a40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/16a2bec6a40000

All of the runs failed. The most common error (10/20 runs) was:
SwarmingTestError: The test failed. No Python exception was found in the log.
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14a0966ea40000

All of the runs failed. The most common error (20/20 runs) was:
Exception: Shard indicators must be present for perf tests
😿 Pinpoint job stopped with an error.
https://dev-dtu-8cd47289-dot-pinpoint-dot-chromeperf.appspot.com/job/14f65e31a40000

All of the runs failed. The most common error (20/20 runs) was:
Exception: Shard indicators must be present for perf tests
Project Member

Comment 61 by bugdroid1@chromium.org, Jul 17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f53c3ea9907464bf25fef2add0536e402a129851

commit f53c3ea9907464bf25fef2add0536e402a129851
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Jul 17 19:10:57 2018

Reland #3 of "gtest perf tests: Output well formed test results json"

This reverts commit 3cf9eadffa44ab79c557384ee166ef28bb071d25.

Reason for revert: Will fix errors

Original change's description:
> Revert "Reland "Reland "gtest perf tests: Output well formed test results json"""
>
> This reverts commit 4da4a0c81701d7acf80bd594b6bc00efece0e898.
>
> Reason for revert: Failing on mac and windows.  I will add more comments to  crbug.com/855234 
>
> Original change's description:
> > Reland "Reland "gtest perf tests: Output well formed test results json""
> >
> > This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b.
> >
> > Reason for revert: Testing possible fixes
> >
> > Original change's description:
> > > Revert "Reland "gtest perf tests: Output well formed test results json""
> > >
> > > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37.
> > >
> > > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315
> > >
> > > Original change's description:
> > > > Reland "gtest perf tests: Output well formed test results json"
> > > >
> > > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890.
> > > >
> > > > Reason for revert: Will re-land with fix
> > > >
> > > > Original change's description:
> > > > > Revert "gtest perf tests: Output well formed test results json"
> > > > >
> > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.
> > > > >
> > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)
> > > > >
> > > > > Original change's description:
> > > > > > gtest perf tests: Output well formed test results json
> > > > > >
> > > > > > This CL changes the run_gtest_perf_test.py script to output well formed
> > > > > > test results json.
> > > > > >
> > > > > > Bug:  855234 
> > > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> > > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635
> > > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > > > > > Cr-Commit-Position: refs/heads/master@{#570943}
> > > > >
> > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > > >
> > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
> > > > > No-Presubmit: true
> > > > > No-Tree-Checks: true
> > > > > No-Try: true
> > > > > Bug:  855234 
> > > > > Reviewed-on: https://chromium-review.googlesource.com/1118238
> > > > > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > > > > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > > > > Cr-Commit-Position: refs/heads/master@{#571083}
> > > >
> > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > >
> > > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2
> > > > No-Presubmit: true
> > > > No-Tree-Checks: true
> > > > No-Try: true
> > > > Bug:  855234 
> > > > Reviewed-on: https://chromium-review.googlesource.com/1118789
> > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > > Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> > > > Cr-Commit-Position: refs/heads/master@{#571245}
> > >
> > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > >
> > > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86
> > > No-Presubmit: true
> > > No-Tree-Checks: true
> > > No-Try: true
> > > Bug:  855234 
> > > Reviewed-on: https://chromium-review.googlesource.com/1119445
> > > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > > Cr-Commit-Position: refs/heads/master@{#571321}
> >
> > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> >
> > Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Bug:  855234 
> > Reviewed-on: https://chromium-review.googlesource.com/1120798
> > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#571968}
>
> TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
>
> Change-Id: I031c5d6204e0b53fa2719168a38292455d676167
> No-Presubmit: true
> No-Tree-Checks: true
> No-Try: true
> Bug:  855234 
> Reviewed-on: https://chromium-review.googlesource.com/1124539
> Reviewed-by: Emily Hanley <eyaich@chromium.org>
> Commit-Queue: Emily Hanley <eyaich@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#572183}

Change-Id: I6c994148908ed9a223b8b59219b6b4a6957cd9ca
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1125420
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#575747}
[modify] https://crrev.com/f53c3ea9907464bf25fef2add0536e402a129851/testing/scripts/common.py
[modify] https://crrev.com/f53c3ea9907464bf25fef2add0536e402a129851/testing/scripts/run_gtest_perf_test.py
[modify] https://crrev.com/f53c3ea9907464bf25fef2add0536e402a129851/testing/scripts/run_performance_tests.py

The CL seems to have landed ok. I'm looking at the perf bots and everything seems to be fine. I'll keep monitoring.
Project Member

Comment 63 by bugdroid1@chromium.org, Jul 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0933c720c2c368df9988476875afbac2e2600bf8

commit 0933c720c2c368df9988476875afbac2e2600bf8
Author: Ned Nguyen <nednguyen@google.com>
Date: Wed Jul 18 12:20:47 2018

Revert "Reland #3 of "gtest perf tests: Output well formed test results json""

This reverts commit f53c3ea9907464bf25fef2add0536e402a129851.

Reason for revert: break media_perftest ( crbug.com/864942 )


Now I look at the code, I also have concerns about the approach of the CL. Please discuss with me on the approach to implement the feature.

Original change's description:
> Reland #3 of "gtest perf tests: Output well formed test results json"
> 
> This reverts commit 3cf9eadffa44ab79c557384ee166ef28bb071d25.
> 
> Reason for revert: Will fix errors
> 
> Original change's description:
> > Revert "Reland "Reland "gtest perf tests: Output well formed test results json"""
> >
> > This reverts commit 4da4a0c81701d7acf80bd594b6bc00efece0e898.
> >
> > Reason for revert: Failing on mac and windows.  I will add more comments to  crbug.com/855234 
> >
> > Original change's description:
> > > Reland "Reland "gtest perf tests: Output well formed test results json""
> > >
> > > This reverts commit 89c5e7909f22e5cb9946f0cb7fcedf5494080d2b.
> > >
> > > Reason for revert: Testing possible fixes
> > >
> > > Original change's description:
> > > > Revert "Reland "gtest perf tests: Output well formed test results json""
> > > >
> > > > This reverts commit f02c92e1ae6b6b466f4d42b3d870ed070bb28b37.
> > > >
> > > > Reason for revert: still breaking perf waterfall https://ci.chromium.org/buildbot/chromium.perf/linux-perf/315
> > > >
> > > > Original change's description:
> > > > > Reland "gtest perf tests: Output well formed test results json"
> > > > >
> > > > > This reverts commit 9eeaba4e66b95a174d03ecc8caadb64fa3dcb890.
> > > > >
> > > > > Reason for revert: Will re-land with fix
> > > > >
> > > > > Original change's description:
> > > > > > Revert "gtest perf tests: Output well formed test results json"
> > > > > >
> > > > > > This reverts commit 71b865f40dfdaf80f466e94bc4c98ffeb5fd89fb.
> > > > > >
> > > > > > Reason for revert: break many perf tests (see https://ci.chromium.org/buildbot/chromium.perf/linux-perf/307)
> > > > > >
> > > > > > Original change's description:
> > > > > > > gtest perf tests: Output well formed test results json
> > > > > > >
> > > > > > > This CL changes the run_gtest_perf_test.py script to output well formed
> > > > > > > test results json.
> > > > > > >
> > > > > > > Bug:  855234 
> > > > > > > Change-Id: Ia5ce6e258865e34a2aa37635c014cd54efa62796
> > > > > > > Reviewed-on: https://chromium-review.googlesource.com/1112635
> > > > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > > > > > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > > > > > > Cr-Commit-Position: refs/heads/master@{#570943}
> > > > > >
> > > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > > > >
> > > > > > Change-Id: Ia0aaff923014438005ebc0a8677c3bbbebb808ac
> > > > > > No-Presubmit: true
> > > > > > No-Tree-Checks: true
> > > > > > No-Try: true
> > > > > > Bug:  855234 
> > > > > > Reviewed-on: https://chromium-review.googlesource.com/1118238
> > > > > > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > > > > > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > > > > > Cr-Commit-Position: refs/heads/master@{#571083}
> > > > >
> > > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > > >
> > > > > Change-Id: Idb255b1fafb7f254fd44d6554f93ff27608279f2
> > > > > No-Presubmit: true
> > > > > No-Tree-Checks: true
> > > > > No-Try: true
> > > > > Bug:  855234 
> > > > > Reviewed-on: https://chromium-review.googlesource.com/1118789
> > > > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > > > Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> > > > > Cr-Commit-Position: refs/heads/master@{#571245}
> > > >
> > > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > > >
> > > > Change-Id: I6bc0869caa6a763dc8bb542fd5a1b3d25853ac86
> > > > No-Presubmit: true
> > > > No-Tree-Checks: true
> > > > No-Try: true
> > > > Bug:  855234 
> > > > Reviewed-on: https://chromium-review.googlesource.com/1119445
> > > > Reviewed-by: Ned Nguyen <nednguyen@google.com>
> > > > Commit-Queue: Ned Nguyen <nednguyen@google.com>
> > > > Cr-Commit-Position: refs/heads/master@{#571321}
> > >
> > > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> > >
> > > Change-Id: I25bc4bb1f77031d95dd70347bfed886112b76218
> > > No-Presubmit: true
> > > No-Tree-Checks: true
> > > No-Try: true
> > > Bug:  855234 
> > > Reviewed-on: https://chromium-review.googlesource.com/1120798
> > > Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> > > Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> > > Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> > > Cr-Commit-Position: refs/heads/master@{#571968}
> >
> > TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org
> >
> > Change-Id: I031c5d6204e0b53fa2719168a38292455d676167
> > No-Presubmit: true
> > No-Tree-Checks: true
> > No-Try: true
> > Bug:  855234 
> > Reviewed-on: https://chromium-review.googlesource.com/1124539
> > Reviewed-by: Emily Hanley <eyaich@chromium.org>
> > Commit-Queue: Emily Hanley <eyaich@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#572183}
> 
> Change-Id: I6c994148908ed9a223b8b59219b6b4a6957cd9ca
> Bug:  855234 
> Reviewed-on: https://chromium-review.googlesource.com/1125420
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#575747}

TBR=dpranke@chromium.org,nednguyen@google.com,martiniss@chromium.org,eyaich@chromium.org

Change-Id: I250655ba8a77cbee8ea46d7471c4691e761fa88e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  855234 
Reviewed-on: https://chromium-review.googlesource.com/1141824
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#576013}
[modify] https://crrev.com/0933c720c2c368df9988476875afbac2e2600bf8/testing/scripts/common.py
[modify] https://crrev.com/0933c720c2c368df9988476875afbac2e2600bf8/testing/scripts/run_gtest_perf_test.py
[modify] https://crrev.com/0933c720c2c368df9988476875afbac2e2600bf8/testing/scripts/run_performance_tests.py

Blocking: -533481
Status: Ned (was: Assigned)
Owner: nednguyen@chromium.org
Status: Assigned (was: Ned)
Ned said that he would work on this. It's not blocking the other bug anymore I think.
Mergedinto: 866124
Status: Duplicate (was: Assigned)

Sign in to add a comment