New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 894261 link

Starred by 2 users
Status: Assigned
Owner:
crouleau@chromium.org
Cc:
█ kbr@chromium.org
dpranke@chromium.org
erikc...@chromium.org
crouleau@chromium.org
█ robert...@chromium.org
chanli@chromium.org
lijeffrey@chromium.org
liaoyuke@chromium.org
st...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocking:
issue 894251
issue 894258



Sign in to add a comment

Supporting telemetry_perf_unittests in FindIt - support Findit required cmd flags

Project Member Reported by chanli@chromium.org, Oct 10 
To support telemetry-based tests in Findit, test runners need to support the following cmd flags:
1. --isolated-script-test-filter=A::B::C
2. --isolated-script-test-repeat=30
  a. To be consistent with currently supported test types, the exec 
     order should be ABCABCABC...
3. --isolated-script-test-launcher-retry-limit=0
4. --isolated-script-test-also-run-disabled-tests
 
Context: https://docs.google.com/document/d/1XZrAecLT33sJng9JTHsWCNpcBVU_BT7weLE76nwsTXA/edit?usp=sharing

Comment 1 by nedngu...@google.com, Oct 10

Cc: erikc...@chromium.org
Labels: -Pri-3 Pri-1
Owner: nedngu...@google.com
Status: Assigned (was: Available)
Want to get this done soon for:
1) Reducing telemetry_perf_unittest flakes
2) Pave the way for implementing this in benchmarking harness & GPU test harnesses
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 12 

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/a65eb1970e6c5e6bec40648ac97d3f1aa8ed5509

commit a65eb1970e6c5e6bec40648ac97d3f1aa8ed5509
Author: Nghia Nguyen <nednguyen@google.com>
Date: Fri Oct 12 15:24:34 2018

[Telemetry] Add --test-filter flag that support exact matching of multiple tests

*Besides the added unittest, one can also verify that this works by running:
./telemetry/bin/run_tests --browser=system --test-filter=telemetry.decorators_unittest.TestEnableDecorators.testEnabledStringOnMethod::telemetry.decorators_unittest.TestEnableDecorators.testEnabledStringOnSubClass

Bug: chromium:894261
Change-Id: I956919531f53f6f15e47e8457e7112f54d6f6a1c
Reviewed-on: https://chromium-review.googlesource.com/c/1277773
Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/a65eb1970e6c5e6bec40648ac97d3f1aa8ed5509/telemetry/telemetry/testing/run_tests_unittest.py
[modify] https://crrev.com/a65eb1970e6c5e6bec40648ac97d3f1aa8ed5509/telemetry/telemetry/testing/run_tests.py
Project Member

Comment 3 by bugdroid1@chromium.org, Oct 12 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bb11cbc66e56a475bdf0f5bf3d962cf80c3afe27

commit bb11cbc66e56a475bdf0f5bf3d962cf80c3afe27
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Oct 12 22:54:58 2018

Roll src/third_party/catapult c8b97e37ec9c..99b88b0a2d58 (21 commits)

https://chromium.googlesource.com/catapult.git/+log/c8b97e37ec9c..99b88b0a2d58


git log c8b97e37ec9c..99b88b0a2d58 --date=short --no-merges --format='%ad %ae %s'
2018-10-12 simonhatch@chromium.org Pinpoint - Merge bugs with same root cause.
2018-10-12 benjhayden@chromium.org Change deprecation warnings in /api/alerts to errors.
2018-10-12 benjhayden@chromium.org Cache report names, test suites, and descriptors in v2spa service worker
2018-10-12 chiniforooshan@chromium.org Telemetry: process all rendering pipeline events
2018-10-12 benjhayden@chromium.org Cache session ids in v2spa service worker.
2018-10-12 benjhayden@chromium.org Fix links in rollback.md
2018-10-12 benjhayden@chromium.org Add KeyValueCacheRequest for v2spa service worker.
2018-10-12 benjhayden@chromium.org Add ts_mon metrics to common/timing.py
2018-10-12 benjhayden@chromium.org Add CacheRequestBase for v2spa service worker.
2018-10-12 nednguyen@google.com [Telemetry] Add --test-filter flag that support exact matching of multiple tests
2018-10-12 sadrul@chromium.org rendering: Generate metrics at the 95%ile.
2018-10-12 pasko@chromium.org androidStartupMetric: re-introduce First Contentful Paint
2018-10-12 chrishtr@chromium.org Clean up categories to match current tracing in Blink.
2018-10-11 jbudorick@chromium.org Revert "Enable orderfile memory optimization in benchmarks"
2018-10-11 benjhayden@chromium.org Add task queue for v2spa service worker.
2018-10-11 benjhayden@chromium.org Remove unnecessary file dashboard/test/index.html
2018-10-11 perezju@chromium.org [soundwave] Allow points with missing r_chromium
2018-10-11 chiniforooshan@chromium.org Telemetry: migrate frame_times & percentage_smooth
2018-10-11 ulan@chromium.org [tracing] Restore the minimum mutator utilization metric.
2018-10-11 seanmccullough@chromium.org [chromeperf] add prefix to wct script so tests get found.
2018-10-11 pasko@chromium.org Enable orderfile memory optimization in benchmarks


Created with:
  gclient setdep -r src/third_party/catapult@99b88b0a2d58

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:840872, chromium:893199 ,chromium:867060,chromium:894261,chromium:894287, chromium:886621 ,chromium:893514,chromium:758566, chromium:879526 , chromium:890757 , chromium:892385 , chromium:877660 ,chromium:758566
TBR=sullivan@chromium.org

Change-Id: I469ed20d66790fa12e290ad65e2ea51c16cfdba7
Reviewed-on: https://chromium-review.googlesource.com/c/1279078
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#599392}
[modify] https://crrev.com/bb11cbc66e56a475bdf0f5bf3d962cf80c3afe27/DEPS
Project Member

Comment 4 by bugdroid1@chromium.org, Oct 13 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eec69cb5080617a5b731198fa5673dbc25f60806

commit eec69cb5080617a5b731198fa5673dbc25f60806
Author: Ned Nguyen <nednguyen@google.com>
Date: Sat Oct 13 00:42:19 2018

Update run_telemetry_as_googletest.py to directly use Telemetry unittest harness's --test-filter flag

Note: this depends on https://chromium-review.googlesource.com/c/catapult/+/1277773 being rolled to Chromium/src/

Bug: chromium:894261
Change-Id: I946403a16adfa44f0dcf9aa099c57a69749b5dff
Reviewed-on: https://chromium-review.googlesource.com/c/1277817
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#599445}
[modify] https://crrev.com/eec69cb5080617a5b731198fa5673dbc25f60806/testing/scripts/run_telemetry_as_googletest.py
Project Member

Comment 7 by bugdroid1@chromium.org, Oct 16 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c15e89cc4cb03b98e335d29c08039320609bdb84

commit c15e89cc4cb03b98e335d29c08039320609bdb84
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Oct 16 00:43:56 2018

Roll src/third_party/catapult 5aac72d05c7e..2be20fdd2d70 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/5aac72d05c7e..2be20fdd2d70


git log 5aac72d05c7e..2be20fdd2d70 --date=short --no-merges --format='%ad %ae %s'
2018-10-15 nednguyen@google.com Remove --exact-test-filter flag


Created with:
  gclient setdep -r src/third_party/catapult@2be20fdd2d70

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG=chromium:894261
TBR=sullivan@chromium.org

Change-Id: I618e12507c58e9a214232aa3866780f6b13ef8bf
Reviewed-on: https://chromium-review.googlesource.com/c/1281264
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#599798}
[modify] https://crrev.com/c15e89cc4cb03b98e335d29c08039320609bdb84/DEPS
Project Member

Comment 8 by bugdroid1@chromium.org, Oct 16 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cf17a972c6633f00f5c941da41a210c66b85f28e

commit cf17a972c6633f00f5c941da41a210c66b85f28e
Author: John Budorick <jbudorick@chromium.org>
Date: Tue Oct 16 01:01:30 2018

Revert "[testing/scripts] Add base adapter class that enables more code reuse for all isolated script test"

This reverts commit 18e40bdceea9a7a147d99e35cf138c7872d9d229.

Reason for revert: Spec revert for https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20Tests%20%281%29/84707 etc

Original change's description:
> [testing/scripts] Add base adapter class that enables more code reuse for all isolated script test
> 
> BUG: 894261
> Change-Id: Id707ba8a8782a23a41a7dc4680cbc2538e1d3df1
> Reviewed-on: https://chromium-review.googlesource.com/c/1278156
> Reviewed-by: David Trainor <dtrainor@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Ned Nguyen <nednguyen@google.com>
> Cr-Commit-Position: refs/heads/master@{#599669}

TBR=nyquist@chromium.org,dpranke@chromium.org,kbr@chromium.org,dtrainor@chromium.org,nednguyen@google.com,martiniss@chromium.org

Change-Id: Ib67b3565568dec728a78fd7d95342d4762a8ecbb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/1282215
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#599803}
[modify] https://crrev.com/cf17a972c6633f00f5c941da41a210c66b85f28e/chrome/android/monochrome/BUILD.gn
[modify] https://crrev.com/cf17a972c6633f00f5c941da41a210c66b85f28e/testing/scripts/common.py
[modify] https://crrev.com/cf17a972c6633f00f5c941da41a210c66b85f28e/testing/scripts/run_chromedriver_tests.py
[modify] https://crrev.com/cf17a972c6633f00f5c941da41a210c66b85f28e/testing/scripts/run_gpu_integration_test_as_googletest.py
[modify] https://crrev.com/cf17a972c6633f00f5c941da41a210c66b85f28e/testing/scripts/run_isolated_script_test.py
[modify] https://crrev.com/cf17a972c6633f00f5c941da41a210c66b85f28e/testing/scripts/run_telemetry_as_googletest.py
Project Member

Comment 9 by bugdroid1@chromium.org, Oct 16 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/09dcd00f7f832753594286999666ddb14c796f7a

commit 09dcd00f7f832753594286999666ddb14c796f7a
Author: Ned Nguyen <nednguyen@google.com>
Date: Tue Oct 16 06:01:50 2018

Reland "[testing/scripts] Add base adapter class that enables more code reuse for all isolated script test"

This is a reland of 18e40bdceea9a7a147d99e35cf138c7872d9d229


It also includes the fix to only generate filter file for isolated_script_test if --isolated-script-test-filter flag is passed

This will make the behavior of testing/scripts/run_isolated_script_test.py matches
with what happened before the refactoring in https://chromium-review.googlesource.com/c/chromium/src/+/1278156

The bug happened because when  --isolated-script-test-filter is not passed,
TypUnittestAdapter.generate_test_filter_args() isn't called, which means the _temp_filter_file file
handle is not closed. Hence attempting to remove the file will cause a crash on Windows platform
(removing a file whose file handle opened).

In this fix, we create _temp_filter_file within TypUnittestAdapter.generate_test_filter_args() and
only clean up the file if it was created.

This also fix generate_test_filter_args to generate flags with self._temp_filter_file.name (used to beself._temp_filter_file)

TBR=kbr@chromium.org, dpranke@chromium.org, dtrainor@chromium.org

Bug: 894261,  895586 ,  895601 
Change-Id: I74255ba2bc7598db9df18cff4bff4407c2a927f8
Reviewed-on: https://chromium-review.googlesource.com/c/1281902
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#599880}
[modify] https://crrev.com/09dcd00f7f832753594286999666ddb14c796f7a/chrome/android/monochrome/BUILD.gn
[modify] https://crrev.com/09dcd00f7f832753594286999666ddb14c796f7a/testing/scripts/common.py
[modify] https://crrev.com/09dcd00f7f832753594286999666ddb14c796f7a/testing/scripts/run_chromedriver_tests.py
[modify] https://crrev.com/09dcd00f7f832753594286999666ddb14c796f7a/testing/scripts/run_gpu_integration_test_as_googletest.py
[modify] https://crrev.com/09dcd00f7f832753594286999666ddb14c796f7a/testing/scripts/run_isolated_script_test.py
[modify] https://crrev.com/09dcd00f7f832753594286999666ddb14c796f7a/testing/scripts/run_telemetry_as_googletest.py
Project Member

Comment 10 by bugdroid1@chromium.org, Oct 16 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/18e19acb36361a59e929845cdf6fc77c737f61e3

commit 18e19acb36361a59e929845cdf6fc77c737f61e3
Author: Ned Nguyen <nednguyen@google.com>
Date: Tue Oct 16 13:51:53 2018

Implement generate_test_launcher_retry_limit_args for Telemetry unittest script

Bug: 894261
Change-Id: I6d161b6e1c7b7cbfe039fc15747f71acb3042f70
Reviewed-on: https://chromium-review.googlesource.com/c/1281170
Reviewed-by: Caleb Rouleau <crouleau@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#599962}
[modify] https://crrev.com/18e19acb36361a59e929845cdf6fc77c737f61e3/testing/scripts/common.py
[modify] https://crrev.com/18e19acb36361a59e929845cdf6fc77c737f61e3/testing/scripts/run_telemetry_as_googletest.py
Project Member

Comment 12 by bugdroid1@chromium.org, Oct 17 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d78223532dd4dc471195262fdcdcca227861f1b9

commit d78223532dd4dc471195262fdcdcca227861f1b9
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Wed Oct 17 20:07:40 2018

Roll src/third_party/catapult b273e0cd217d..519565187c85 (4 commits)

https://chromium.googlesource.com/catapult.git/+log/b273e0cd217d..519565187c85


git log b273e0cd217d..519565187c85 --date=short --no-merges --format='%ad %ae %s'
2018-10-17 benjhayden@chromium.org Revert "Migrate trace_viewer.gypi to gni"
2018-10-17 nednguyen@google.com Add support for --repeat flag in typ
2018-10-17 chiniforooshan@chromium.org aura: filter out frames not submitted from browser
2018-10-17 benjhayden@chromium.org Migrate trace_viewer.gypi to gni


Created with:
  gclient setdep -r src/third_party/catapult@519565187c85

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:895699 ,chromium:894261, chromium:883894 , chromium:895699 
TBR=sullivan@chromium.org

Change-Id: I42ab97661d390f5ea0e56de3f6196dd4ac19a925
Reviewed-on: https://chromium-review.googlesource.com/c/1286751
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#600532}
[modify] https://crrev.com/d78223532dd4dc471195262fdcdcca227861f1b9/DEPS

Comment 13 by nedngu...@google.com, Oct 18

Status: Fixed (was: Assigned)
The last 3 CLs are in https://bugs.chromium.org/p/chromium/issues/detail?id=893618#c5, #c6, #c7. This bug should be fixed.

Feel free to file any bug against me if telemetry_perf_unittests Findit integration has problems

Comment 14 by nedngu...@google.com, Oct 18

Cc: -nednguyen@chromium.org

Comment 15 by kbr@chromium.org, Oct 29

Blocking: 894258

Comment 16 by nedngu...@google.com, Nov 9

Cc: dpranke@chromium.org
 Issue 781746  has been merged into this issue.

Comment 17 by chanli@chromium.org, Jan 18 (4 days ago)

Owner: crouleau@chromium.org
Status: Assigned (was: Fixed)
reopen this bug.

I had a sample task https://chromium-swarm.appspot.com/task?id=42638d5c9993b210&refresh=10&show_raw=1 not working.

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/b/s/w/ir/tools/perf/test_results/analysis_unittest.py", line 7, in <module>
    from test_results import analysis
  File "/b/s/w/ir/tools/perf/test_results/analysis.py", line 8, in <module>
    import pandas  # pylint: disable=import-error
ImportError: No module named pandas

**Non zero exit code**
If you don't see any error stack, this could have been a native crash. Consider installing faulthandler (https://faulthandler.readthedocs.io/) for more useful error message


FIT, this task is referencing task https://chromium-swarm.appspot.com/task?id= 4263442bca4a6b10&refresh=10&show_raw=1 from build https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests/74909
►

Sign in to add a comment