New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 13
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocked on:
issue 904019

Blocking:
issue 698902
issue 807751
issue 905426



Sign in to add a comment
link

Issue 835690: Add support for test expectations to TYP

Reported by dpranke@chromium.org, Apr 23 2018 Project Member

Issue description

TYP needs to support some sort of test expectations format so that we can
switch the layout tests to it and also ensure that the GPU tests, the Telemetry
tests, and the other Python-based tests all also support the functionality.
 

Comment 1 by dpranke@chromium.org, Apr 23 2018

Blocking: 807751

Comment 2 by nednguyen@chromium.org, Apr 23 2018

Cc: charliea@chromium.org
Dirk: one thing I just remember is we have a generalized expectation parser in https://cs.chromium.org/chromium/src/third_party/catapult/common/py_utils/py_utils/expectations_parser.py?rcl=219bbf110903207c2112a382cdf4741a089993e3&l=12 

Maybe we can just move this from catapult/common/py_utils/py_utils/ to catapult/third_party/typ/?

Comment 3 by nednguyen@chromium.org, Apr 23 2018

Cc: -nedngu...@google.com nednguyen@chromium.org

Comment 4 by dpranke@chromium.org, Apr 23 2018

Thanks for the pointer, it looks promising and I'll take a look.

Comment 5 by dpranke@chromium.org, Jun 11 2018

Blocking: 698902

Comment 6 by dpranke@chromium.org, Aug 11

I think I have this mostly ready in https://crrev.com/c/1023650 now, but I'm going to split that up into pieces to make it easier to review:

1. copy the existing parser over 
2. reformat to use the same (pep8) style as the other typ code
3. change the parser to use the latest version of the syntax.
4. integrate into the rest of typ.
5. add glob support.
6. add support for multiple expectations files.

The GPU tests will need a couple of other features that are related but won't be a direct part of this chain of CLs:

7. Support for a "strict" mode for retries, wherein TYP only retries tests that fail and that are marked as flaky (rather than all failures).

8. Possibly add support for a flag to classify "Fail" as including crashes and timeouts, so that you don't have to use Crash and Timeout as expected results, but can still report the crash and timeout back to TYP properly so that they show up as such in the json results (I don't think you can do this as cleanly in the test code itself).

Comment 7 by kbr@chromium.org, Aug 11

Did we determine that (6) (support for multiple expectations files) wasn't needed?

Comment 8 by dpranke@chromium.org, Aug 11

We determined that multiple expectations files are not needed for the GPU tests, but they are for webkit_layout_tests.

I've actually only coded (1) - (5) so far and plan to land them ASAP. (i.e., my ordering is slightly misleading).

(7) and (8) are easy; I'll get to them shortly after getting (1) - (5) up.

(6) is fairly straightforward but a little bit more involved. In particular, I need to decide if I want to support multiple ways to handle tests with expectations in multiple files, or if I can just start by supporting only the semantics we need in webkit_layout_tests. 

But, I don't want (6) to block your GPU work, so I'm going to try and get everything else done first.

Comment 9 by kbr@chromium.org, Aug 11

Right, forgot that layout tests require multiple expectations files. Thank you in advance for unblocking us! Will switch the GPU tests over to the new mechanism as soon as it's available.

Comment 10 by kbr@chromium.org, Oct 4

Any updates on progress? Our team would very much like to switch to using typ's test expectations so that we can see whether tests marked flaky are still flaky. Thanks!

Comment 11 by dpranke@chromium.org, Oct 4

No real updates over comment #8, sadly. Hard to believe I'm this far behind, but the dates are right. I uploaded a bunch of CLs on 8/10 and needed to do one more pass over them before I sent them out for the actual review to land, and haven't gotten to that yet :(.

Comment 12 by kbr@chromium.org, Oct 4

OK, no worries, please keep this bug updated as things land. Again, we're excited to collaborate with you on this. Thanks!

Comment 13 by bugdroid1@chromium.org, Nov 2

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/1e976e1c6dbf7f06adfae9a1439cbbdaa807e9e4

commit 1e976e1c6dbf7f06adfae9a1439cbbdaa807e9e4
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Nov 02 21:58:32 2018

Copy existing expectations parser into typ.

We want to add support for handling the bit.ly/chromium-test-list-format
for test expectations to TYP.

This CL is the first step in that process; it simply copies over the
existing parser from py_utils unchanged. Subsequent CLs will update
the parser to handle the correct format, clean up the coding style,
and integrate it into TYP itself.

Bug:  chromium:835690 
Change-Id: I6d61975d34db868a6b551c001ad26e45928cf671
Reviewed-on: https://chromium-review.googlesource.com/c/1171862
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[add] https://crrev.com/1e976e1c6dbf7f06adfae9a1439cbbdaa807e9e4/third_party/typ/typ/tests/expectations_parser_test.py
[add] https://crrev.com/1e976e1c6dbf7f06adfae9a1439cbbdaa807e9e4/third_party/typ/typ/expectations_parser.py

Comment 14 by bugdroid1@chromium.org, Nov 2

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/c02ed04700f771ba63f90e9044f7bfbbcbbe5bec

commit c02ed04700f771ba63f90e9044f7bfbbcbbe5bec
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Nov 02 22:28:56 2018

Reformat typ expectation parser code to match the surrounding style.

This reformats the expectation_parser code to be PEP-8 compliant
(4-space indents, using lower_case_with_underscores for method
names) to match TYP, even though that's not what the rest of
catapult uses.

The reformatting was done using `yapf -i --style pep8`.

Bug:  chromium:835690 
Change-Id: I894054522bc0afbb97cd4810d7fe3d8912456fd1
Reviewed-on: https://chromium-review.googlesource.com/c/1171865
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/c02ed04700f771ba63f90e9044f7bfbbcbbe5bec/third_party/typ/typ/tests/expectations_parser_test.py
[modify] https://crrev.com/c02ed04700f771ba63f90e9044f7bfbbcbbe5bec/third_party/typ/typ/expectations_parser.py

Comment 15 by bugdroid1@chromium.org, Nov 3

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/093f35ab3a60e50020ee8388c24ddd54204e81cc

commit 093f35ab3a60e50020ee8388c24ddd54204e81cc
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Sat Nov 03 01:23:33 2018

Roll src/third_party/catapult 468ff3e5a030..c02ed04700f7 (2 commits)

https://chromium.googlesource.com/catapult.git/+log/468ff3e5a030..c02ed04700f7


git log 468ff3e5a030..c02ed04700f7 --date=short --no-merges --format='%ad %ae %s'
2018-11-02 dpranke@chromium.org Reformat typ expectation parser code to match the surrounding style.
2018-11-02 dpranke@chromium.org Copy existing expectations parser into typ.


Created with:
  gclient setdep -r src/third_party/catapult@c02ed04700f7

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 , chromium:835690 
TBR=sullivan@chromium.org

Change-Id: Ica23fbe593b0950a5e061374af37497b7bc9e782
Reviewed-on: https://chromium-review.googlesource.com/c/1316548
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#605117}
[modify] https://crrev.com/093f35ab3a60e50020ee8388c24ddd54204e81cc/DEPS

Comment 16 by bugdroid1@chromium.org, Nov 6

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/70df4ea4898ed2478b3e0b9e50f2667a58a78aaa

commit 70df4ea4898ed2478b3e0b9e50f2667a58a78aaa
Author: Dirk Pranke <dpranke@chromium.org>
Date: Tue Nov 06 20:46:53 2018

Address review comments from https://crrev.com/c/1171862.

This addresses the feedback from the initial CL, which was
just copying files around. The comments suggested improvements
to type safety and error messaging.

Bug:  chromium:835690 
Change-Id: Ia5303bdcbcb9b80e1c4abb85e28277c948f39fbd
Reviewed-on: https://chromium-review.googlesource.com/c/1316567
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/70df4ea4898ed2478b3e0b9e50f2667a58a78aaa/third_party/typ/typ/expectations_parser.py

Comment 17 by bugdroid1@chromium.org, Nov 6

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/2e722b007d34e61aad0b5f13d74e5a5d9616c1a0

commit 2e722b007d34e61aad0b5f13d74e5a5d9616c1a0
Author: Dirk Pranke <dpranke@chromium.org>
Date: Tue Nov 06 21:42:51 2018

Rename 'conditions' to 'tags' in the parser.

This better matches the terminology in the spec (and the file
format itself).

Change-Id: Ib2885b0b31d5e3236f0032ad825945e13e2dab1a
Bug:  chromium:835690 
Reviewed-on: https://chromium-review.googlesource.com/c/1313261
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/2e722b007d34e61aad0b5f13d74e5a5d9616c1a0/third_party/typ/typ/tests/expectations_parser_test.py
[modify] https://crrev.com/2e722b007d34e61aad0b5f13d74e5a5d9616c1a0/third_party/typ/typ/expectations_parser.py

Comment 18 by bugdroid1@chromium.org, Nov 6

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d31efeca7c0554367d4ba3a293a0a81b6fe872e0

commit d31efeca7c0554367d4ba3a293a0a81b6fe872e0
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Nov 06 22:22:42 2018

Roll src/third_party/catapult 0273c5e3c23a..70df4ea4898e (2 commits)

https://chromium.googlesource.com/catapult.git/+log/0273c5e3c23a..70df4ea4898e


git log 0273c5e3c23a..70df4ea4898e --date=short --no-merges --format='%ad %ae %s'
2018-11-06 dpranke@chromium.org Address review comments from https://crrev.com/c/1171862.
2018-11-06 eroman@chromium.org Add --proxy-bypass-list parameter to explicitly allow proxying of localhost.


Created with:
  gclient setdep -r src/third_party/catapult@70df4ea4898e

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 ,chromium:901896
TBR=sullivan@chromium.org

Change-Id: Icea4759cfbfe703010fc8136b82755bd9be74c18
Reviewed-on: https://chromium-review.googlesource.com/c/1320614
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#605847}
[modify] https://crrev.com/d31efeca7c0554367d4ba3a293a0a81b6fe872e0/DEPS

Comment 19 by bugdroid1@chromium.org, Nov 7

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d1e66b95379e80b0795f0242d7c372d7651deb8

commit 1d1e66b95379e80b0795f0242d7c372d7651deb8
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Wed Nov 07 00:58:11 2018

Roll src/third_party/catapult 70df4ea4898e..2e722b007d34 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/70df4ea4898e..2e722b007d34


git log 70df4ea4898e..2e722b007d34 --date=short --no-merges --format='%ad %ae %s'
2018-11-06 dpranke@chromium.org Rename 'conditions' to 'tags' in the parser.


Created with:
  gclient setdep -r src/third_party/catapult@2e722b007d34

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 
TBR=sullivan@chromium.org

Change-Id: I0b55520ce5c2b2a2d55691e151a0f54ec369be43
Reviewed-on: https://chromium-review.googlesource.com/c/1321329
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#605894}
[modify] https://crrev.com/1d1e66b95379e80b0795f0242d7c372d7651deb8/DEPS

Comment 20 by bugdroid1@chromium.org, Nov 8

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/f04a3a61ad90b65450499cfca9ae2f7b57054d03

commit f04a3a61ad90b65450499cfca9ae2f7b57054d03
Author: Dirk Pranke <dpranke@chromium.org>
Date: Thu Nov 08 03:40:18 2018

Update the expectation parser in TYP to match the new format.

The "tagged list" file format in bit.ly/chromium-test-list-format
drifted a bit from what was implemented in common/py_utils. Most
notably, we switched to using "[" and "]" to enclose the list of
tags, so that the list of tags could span multiple lines.

This CL also updates the json_results.py class to be consistent
with the spec by using "PASS" instead of "Pass", etc.

Bug:  chromium:835690 
Change-Id: I533d801b05d61d4f3c8dc15ef9c3f22b10308f6f
Reviewed-on: https://chromium-review.googlesource.com/c/1171875
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/f04a3a61ad90b65450499cfca9ae2f7b57054d03/third_party/typ/typ/tests/main_test.py
[modify] https://crrev.com/f04a3a61ad90b65450499cfca9ae2f7b57054d03/third_party/typ/typ/tests/expectations_parser_test.py
[modify] https://crrev.com/f04a3a61ad90b65450499cfca9ae2f7b57054d03/third_party/typ/typ/json_results.py
[modify] https://crrev.com/f04a3a61ad90b65450499cfca9ae2f7b57054d03/third_party/typ/typ/tests/json_results_test.py
[modify] https://crrev.com/f04a3a61ad90b65450499cfca9ae2f7b57054d03/third_party/typ/typ/expectations_parser.py

Comment 21 by bugdroid1@chromium.org, Nov 8

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/275d211360a55814de5717ca120e82af264f796b

commit 275d211360a55814de5717ca120e82af264f796b
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Thu Nov 08 09:15:21 2018

Roll src/third_party/catapult 026f83d49289..f04a3a61ad90 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/026f83d49289..f04a3a61ad90


git log 026f83d49289..f04a3a61ad90 --date=short --no-merges --format='%ad %ae %s'
2018-11-08 dpranke@chromium.org Update the expectation parser in TYP to match the new format.


Created with:
  gclient setdep -r src/third_party/catapult@f04a3a61ad90

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 
TBR=sullivan@chromium.org

Change-Id: I342f7d89d9cd55f70eaad8659ac9d56450d4ddf5
Reviewed-on: https://chromium-review.googlesource.com/c/1325653
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#606391}
[modify] https://crrev.com/275d211360a55814de5717ca120e82af264f796b/DEPS

Comment 22 by bugdroid1@chromium.org, Nov 8

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/4b687df1594d83d8876e6ff84b3dc9fc1a439d78

commit 4b687df1594d83d8876e6ff84b3dc9fc1a439d78
Author: Dirk Pranke <dpranke@chromium.org>
Date: Thu Nov 08 21:33:15 2018

Integrate test expectation-parsing code into TYP.

As noted in earlier CLs on this bug, this adds support for
the "tagged test list" variant of bit.ly/chromium-test-list-format
to TYP.

Programmatically, this is done by adding support for a single file
argument (-X/--expectations-file) and a list of tags (-x/--tags),
both specified as args to the runner like most other parameters.

The "tags" must be the subset of tags specified in the
file that should be applied to the tests in this run. For example,
if the tags are [ Debug Release ] and [ Mac Win Linux ], then
you might specify `-x Mac -x Debug` for a debug Mac build.

Bug:  chromium:835690 
Change-Id: I165964e4f4f6403f01669500d0ea90a99e2e8b81
Reviewed-on: https://chromium-review.googlesource.com/c/1171876
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/4b687df1594d83d8876e6ff84b3dc9fc1a439d78/third_party/typ/typ/tests/main_test.py
[modify] https://crrev.com/4b687df1594d83d8876e6ff84b3dc9fc1a439d78/third_party/typ/typ/arg_parser.py
[modify] https://crrev.com/4b687df1594d83d8876e6ff84b3dc9fc1a439d78/third_party/typ/typ/__init__.py
[modify] https://crrev.com/4b687df1594d83d8876e6ff84b3dc9fc1a439d78/third_party/typ/typ/runner.py

Comment 23 by kbr@chromium.org, Nov 10

Blockedon: 904019

Comment 24 by bugdroid1@chromium.org, Nov 10

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/33c10c78715457aac9937c0a5b71b477233cc9da

commit 33c10c78715457aac9937c0a5b71b477233cc9da
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Sat Nov 10 09:22:50 2018

Roll src/third_party/catapult 5d5091665700..f2c3502f7788 (9 commits)

https://chromium.googlesource.com/catapult.git/+log/5d5091665700..f2c3502f7788


git log 5d5091665700..f2c3502f7788 --date=short --no-merges --format='%ad %ae %s'
2018-11-10 nednguyen@google.com [browser_test_runner] Ensure that skipped tests have skipped expectation if typ expectation file is not used
2018-11-09 benjhayden@chromium.org Add BatchIterator for v2spa
2018-11-09 benjhayden@chromium.org Merge test suite descriptors in v2spa
2018-11-09 benjhayden@chromium.org Add RecommendedOptions to v2spa
2018-11-09 benjhayden@chromium.org Support uploading to dev_appserver.py
2018-11-09 benjhayden@chromium.org Generalize complex test cases in Descriptor.
2018-11-09 benjhayden@chromium.org Expose annotations via /api/timeseries2
2018-11-09 benjhayden@chromium.org Add tag-filter for v2spa
2018-11-08 dpranke@chromium.org Integrate test expectation-parsing code into TYP.


Created with:
  gclient setdep -r src/third_party/catapult@f2c3502f7788

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:904019 ,chromium:898552,chromium:900218, chromium:835690 
TBR=sullivan@chromium.org

Change-Id: Ic8d2d5b5c618bef91ecd2c6051a9e85ab340dab9
Reviewed-on: https://chromium-review.googlesource.com/c/1330771
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#607117}
[modify] https://crrev.com/33c10c78715457aac9937c0a5b71b477233cc9da/DEPS

Comment 25 by nedngu...@google.com, Nov 14

Blockedon: 905426

Comment 26 by bugdroid1@chromium.org, Nov 30

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/1b98245e3cb73fd67d1e2e50a016f4d2af76dd30

commit 1b98245e3cb73fd67d1e2e50a016f4d2af76dd30
Author: Dirk Pranke <dpranke@chromium.org>
Date: Fri Nov 30 02:42:29 2018

Move typ's expectation-processing code into expectations_parser.py.

Prior to this CL, the logic for handling test expectations in typ
was mostly split between runner.py and expectations_parser.py.
runner.py was a fine location for actually loading the file contents,
but most of the logic of actually parsing and handling the expectations
isn't all that germane to running the tests, so it was bloating the
code. Plus, having a file dedicated only to parsing the tagged test list
format was a bit excessive.

So, this CL moves the logic for figuring out the expected results for
a given test into expectations_parser.py and renames some of the classes
a bit to be more generic. This simplifies runner.py a bit and complicates
expectations_parser.py, but that's probably the right tradeoff.

It also makes expectations_parser.py a less-accurate name for the file.
It should probably be renamed to expectations.py, but I avoid doing that
in this CL to minimize the size of the diff to review.

Bug:  chromium:835690 
Change-Id: I4c8cc16547489c409d9cab6fa7b10910582ad83e
Reviewed-on: https://chromium-review.googlesource.com/c/1352282
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/1b98245e3cb73fd67d1e2e50a016f4d2af76dd30/third_party/typ/typ/tests/expectations_parser_test.py
[modify] https://crrev.com/1b98245e3cb73fd67d1e2e50a016f4d2af76dd30/third_party/typ/typ/runner.py
[modify] https://crrev.com/1b98245e3cb73fd67d1e2e50a016f4d2af76dd30/third_party/typ/typ/__init__.py
[modify] https://crrev.com/1b98245e3cb73fd67d1e2e50a016f4d2af76dd30/third_party/typ/typ/expectations_parser.py

Comment 27 by bugdroid1@chromium.org, Nov 30

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e672e897dc8463f099bc533aa77aa48239acbd1

commit 1e672e897dc8463f099bc533aa77aa48239acbd1
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Nov 30 04:11:34 2018

Roll src/third_party/catapult 63d69d7792c1..1b98245e3cb7 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/63d69d7792c1..1b98245e3cb7


git log 63d69d7792c1..1b98245e3cb7 --date=short --no-merges --format='%ad %ae %s'
2018-11-30 dpranke@chromium.org Move typ's expectation-processing code into expectations_parser.py.


Created with:
  gclient setdep -r src/third_party/catapult@1b98245e3cb7

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 
TBR=sullivan@chromium.org

Change-Id: I55c7cfc5493c487cb0e0a0eebb63b2bf0ce9e35f
Reviewed-on: https://chromium-review.googlesource.com/c/1356236
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#612558}
[modify] https://crrev.com/1e672e897dc8463f099bc533aa77aa48239acbd1/DEPS

Comment 28 by bugdroid1@chromium.org, Dec 4

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/bec19e3e3951392e8e3173ab3ae75d9a6c7f745b

commit bec19e3e3951392e8e3173ab3ae75d9a6c7f745b
Author: Dirk Pranke <dpranke@chromium.org>
Date: Tue Dec 04 00:25:30 2018

Add glob support to test expectations files in typ.

Bug:  chromium:835690 
Change-Id: Ia5820d18ccd90a159a8f2973ef48ee81ff68a6bd
Reviewed-on: https://chromium-review.googlesource.com/c/1171877
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/bec19e3e3951392e8e3173ab3ae75d9a6c7f745b/third_party/typ/typ/tests/main_test.py
[modify] https://crrev.com/bec19e3e3951392e8e3173ab3ae75d9a6c7f745b/third_party/typ/typ/expectations_parser.py

Comment 29 by bugdroid1@chromium.org, Dec 4

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4b203448b6c739b933c340dadfc46b692ff60d26

commit 4b203448b6c739b933c340dadfc46b692ff60d26
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Dec 04 05:21:20 2018

Roll src/third_party/catapult 4feee5840de2..bec19e3e3951 (2 commits)

https://chromium.googlesource.com/catapult.git/+log/4feee5840de2..bec19e3e3951


git log 4feee5840de2..bec19e3e3951 --date=short --no-merges --format='%ad %ae %s'
2018-12-04 dpranke@chromium.org Add glob support to test expectations files in typ.
2018-12-04 bsheedy@chromium.org devil: Workaround adb keyevent issue when dismissing ANR dialogs


Created with:
  gclient setdep -r src/third_party/catapult@bec19e3e3951

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 , chromium:908917 
TBR=sullivan@chromium.org

Change-Id: I39f2d54f018f6455404f518e5cb75e4d2eccd50d
Reviewed-on: https://chromium-review.googlesource.com/c/1360157
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#613432}
[modify] https://crrev.com/4b203448b6c739b933c340dadfc46b692ff60d26/DEPS

Comment 30 by dpranke@chromium.org, Dec 13

Status: Fixed (was: Started)
I think this is complete now.

Comment 31 by bugdroid1@chromium.org, Dec 13

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/b0086b6061429297fc52c8da73ff09f96558dbcb

commit b0086b6061429297fc52c8da73ff09f96558dbcb
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Thu Dec 13 21:41:10 2018

Modifying telemetry so that test expectations files can be passed to Typ

Currently there is no way to pass test expectations files from gpu tests
to Typ. This CL will allow a developer to add test expectation files to
the gpu_project_config.py CONFIG data structure, which can be passed
to Typ through browser_test_runner.py.

R=nednguyen@google.com

Bug:  chromium:835690 
Change-Id: Icdb39715cd8945fddc3e63406664489c54793f2b
Reviewed-on: https://chromium-review.googlesource.com/c/1375232
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Kenneth Russell <kbr@chromium.org>

[modify] https://crrev.com/b0086b6061429297fc52c8da73ff09f96558dbcb/telemetry/telemetry/testing/browser_test_context.py
[modify] https://crrev.com/b0086b6061429297fc52c8da73ff09f96558dbcb/telemetry/telemetry/project_config.py
[modify] https://crrev.com/b0086b6061429297fc52c8da73ff09f96558dbcb/telemetry/telemetry/testing/run_browser_tests.py
[modify] https://crrev.com/b0086b6061429297fc52c8da73ff09f96558dbcb/telemetry/telemetry/testing/browser_test_runner.py

Comment 32 by kbr@chromium.org, Dec 13

Rakib: could you please file a new bug and block it on this one, for follow-on work regarding using typ's test expectations in other harnesses? Thanks.

Comment 33 by kbr@chromium.org, Dec 13

Blockedon: -905426
Blocking: 905426

Comment 34 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/afefc1c51fc4536ee0c1bfc7d07dda5225dceaac

commit afefc1c51fc4536ee0c1bfc7d07dda5225dceaac
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Fri Dec 14 03:41:36 2018

fixing exceptions raised in benchmark_runner.py

These exceptions are caused by the merge of 1375232. In the CL
I changed the getter function, expectations_file to expectations_files.
Since expectations_file is used in benchmark_runner.py, exceptions
were being thrown.

R=dpranke@google.com, nednguyen@google.com

Bug:  chromium:835690 
Change-Id: I9d34047d561023c250625af015754c50c3eb51ba
Reviewed-on: https://chromium-review.googlesource.com/c/1377534
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Commit-Queue: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/afefc1c51fc4536ee0c1bfc7d07dda5225dceaac/telemetry/telemetry/benchmark_runner.py

Comment 35 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2faf59d54deaa08001ac5e07f5e7dfa8bf29132e

commit 2faf59d54deaa08001ac5e07f5e7dfa8bf29132e
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Dec 14 06:03:22 2018

Roll src/third_party/catapult 203676d4b186..afefc1c51fc4 (2 commits)

https://chromium.googlesource.com/catapult.git/+log/203676d4b186..afefc1c51fc4


git log 203676d4b186..afefc1c51fc4 --date=short --no-merges --format='%ad %ae %s'
2018-12-14 rmhasan@google.com fixing exceptions raised in benchmark_runner.py
2018-12-13 rmhasan@google.com Modifying telemetry so that test expectations files can be passed to Typ


Created with:
  gclient setdep -r src/third_party/catapult@afefc1c51fc4

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 , chromium:835690 
TBR=sullivan@chromium.org

Change-Id: If55409ab5070ba6cb02281e698a3f30f54ea9a89
Reviewed-on: https://chromium-review.googlesource.com/c/1377301
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#616595}
[modify] https://crrev.com/2faf59d54deaa08001ac5e07f5e7dfa8bf29132e/DEPS

Comment 36 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/2ce7ac89a3e96e501f512ce1367e00f76b0d209a

commit 2ce7ac89a3e96e501f512ce1367e00f76b0d209a
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Fri Dec 14 21:04:31 2018

Fixing potential issues I found after merging CL 1375232

R=nednguyen@google.com

Bug:  chromium:835690 
Change-Id: I0f264c5f3ac664f132259d9a38a032cf95cffc24
Reviewed-on: https://chromium-review.googlesource.com/c/1378207
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/2ce7ac89a3e96e501f512ce1367e00f76b0d209a/telemetry/telemetry/benchmark_runner.py
[modify] https://crrev.com/2ce7ac89a3e96e501f512ce1367e00f76b0d209a/telemetry/telemetry/project_config.py

Comment 37 by bugdroid1@chromium.org, Dec 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d518daea1ca0178713d2551dacd56f5bc66d6c7c

commit d518daea1ca0178713d2551dacd56f5bc66d6c7c
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Sat Dec 15 00:19:46 2018

Roll src/third_party/catapult 8964ef5af77b..2ce7ac89a3e9 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/8964ef5af77b..2ce7ac89a3e9


git log 8964ef5af77b..2ce7ac89a3e9 --date=short --no-merges --format='%ad %ae %s'
2018-12-14 rmhasan@google.com Fixing potential issues I found after merging CL 1375232


Created with:
  gclient setdep -r src/third_party/catapult@2ce7ac89a3e9

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 
TBR=sullivan@chromium.org

Change-Id: I2108c0e09ea37bd469162d90db4472914bfc41b8
Reviewed-on: https://chromium-review.googlesource.com/c/1378833
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#616888}
[modify] https://crrev.com/d518daea1ca0178713d2551dacd56f5bc66d6c7c/DEPS

Comment 38 by bugdroid1@chromium.org, Dec 15

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a9d4c8681880b1e71695210e2206d8bd605b5b4f

commit a9d4c8681880b1e71695210e2206d8bd605b5b4f
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Sat Dec 15 09:12:51 2018

Change parameter used to pass test expectations file paths

We are planning to remove the expectations_file parameter of the
telemetry.project_config.ProjectConfig class. We will be using the
new parameter, expectations_files instead.

R=nednguyen@google.com

Bug:  chromium:835690 
Change-Id: Id1137016018ff6fecee2022d4bc40c63915da0ff
Reviewed-on: https://chromium-review.googlesource.com/c/1377179
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Cr-Commit-Position: refs/heads/master@{#616971}
[modify] https://crrev.com/a9d4c8681880b1e71695210e2206d8bd605b5b4f/tools/perf/chrome_telemetry_build/chromium_config.py
[modify] https://crrev.com/a9d4c8681880b1e71695210e2206d8bd605b5b4f/tools/perf/run_benchmark

Comment 39 by bugdroid1@chromium.org, Dec 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/c8440966d73b13b1979849f434faf559a257044c

commit c8440966d73b13b1979849f434faf559a257044c
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Fri Dec 21 01:12:50 2018

Adds ability to pass tags to Typ from browser test suites

This CL will allow browser test suites to generate tags and
then pass them to Typ. These tags will be used to get
expectations for a test. This CL also adds several test cases in
browser_test_runner_unittest.py which passes an expectations
file to Typ through browser_runner_test.Run(). This test also
checks that generated tags can be passed from a browser test
class.

R=nednguyen@google.com

Bug:  chromium:835690 
Change-Id: I2efe4c7561a327fbf5839084ccedf6c42a7c8f9f
Reviewed-on: https://chromium-review.googlesource.com/c/1378692
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/c8440966d73b13b1979849f434faf559a257044c/telemetry/telemetry/testing/run_browser_tests.py
[add] https://crrev.com/c8440966d73b13b1979849f434faf559a257044c/telemetry/examples/browser_tests/generate_tags_test.py
[modify] https://crrev.com/c8440966d73b13b1979849f434faf559a257044c/telemetry/telemetry/testing/serially_executed_browser_test_case.py
[modify] https://crrev.com/c8440966d73b13b1979849f434faf559a257044c/telemetry/telemetry/testing/browser_test_runner_unittest.py

Comment 40 by bugdroid1@chromium.org, Dec 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/4ee958e1f5215809e5d2df1ad308331c3ac67ff5

commit 4ee958e1f5215809e5d2df1ad308331c3ac67ff5
Author: Juan Antonio Navarro Pérez <perezju@chromium.org>
Date: Fri Dec 21 12:51:27 2018

Revert "Adds ability to pass tags to Typ from browser test suites"

This reverts commit c8440966d73b13b1979849f434faf559a257044c.

Reason for revert: Failing in catapult autoroller, e.g.:
https://chromium-review.googlesource.com/c/chromium/src/+/1387773

Original change's description:
> Adds ability to pass tags to Typ from browser test suites
> 
> This CL will allow browser test suites to generate tags and
> then pass them to Typ. These tags will be used to get
> expectations for a test. This CL also adds several test cases in
> browser_test_runner_unittest.py which passes an expectations
> file to Typ through browser_runner_test.Run(). This test also
> checks that generated tags can be passed from a browser test
> class.
> 
> R=​nednguyen@google.com
> 
> Bug:  chromium:835690 
> Change-Id: I2efe4c7561a327fbf5839084ccedf6c42a7c8f9f
> Reviewed-on: https://chromium-review.googlesource.com/c/1378692
> Commit-Queue: Rakib Hasan <rmhasan@google.com>
> Reviewed-by: Ned Nguyen <nednguyen@google.com>

TBR=nednguyen@google.com,rmhasan@google.com

Change-Id: I891c65ac5ace10e76026083601cff602d2297063
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  chromium:835690 
Reviewed-on: https://chromium-review.googlesource.com/c/1387913
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>

[modify] https://crrev.com/4ee958e1f5215809e5d2df1ad308331c3ac67ff5/telemetry/telemetry/testing/run_browser_tests.py
[delete] https://crrev.com/d6891dfeca7882a9d5398a730e4150091e10a4dd/telemetry/examples/browser_tests/generate_tags_test.py
[modify] https://crrev.com/4ee958e1f5215809e5d2df1ad308331c3ac67ff5/telemetry/telemetry/testing/serially_executed_browser_test_case.py
[modify] https://crrev.com/4ee958e1f5215809e5d2df1ad308331c3ac67ff5/telemetry/telemetry/testing/browser_test_runner_unittest.py

Comment 41 by bugdroid1@chromium.org, Dec 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6de01bf77212792a1307576105fd75c0ae652e2f

commit 6de01bf77212792a1307576105fd75c0ae652e2f
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Dec 21 16:30:09 2018

Roll src/third_party/catapult 2f6c18c2d5aa..4ee958e1f521 (5 commits)

https://chromium.googlesource.com/catapult.git/+log/2f6c18c2d5aa..4ee958e1f521


git log 2f6c18c2d5aa..4ee958e1f521 --date=short --no-merges --format='%ad %ae %s'
2018-12-21 perezju@chromium.org Revert "Adds ability to pass tags to Typ from browser test suites"
2018-12-21 crouleau@chromium.org [Telemetry] Remove deprecated --smoke-test-mode flag
2018-12-21 perezju@chromium.org [Telemetry] Remove LegacyPageTest.TabForPage
2018-12-21 lwsong@google.com Fixes for Metrics Visualization
2018-12-21 rmhasan@google.com Adds ability to pass tags to Typ from browser test suites


Created with:
  gclient setdep -r src/third_party/catapult@4ee958e1f521

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 ,chromium:894254, chromium:851948 , chromium:835690 
TBR=sullivan@chromium.org

Change-Id: Ic0ce5e1219d7c4582c1ac05862f18feb95b401cf
Reviewed-on: https://chromium-review.googlesource.com/c/1388379
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#618545}
[modify] https://crrev.com/6de01bf77212792a1307576105fd75c0ae652e2f/DEPS

Comment 42 by bugdroid1@chromium.org, Dec 21

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/d7d78924ccedc33db62fb368ddce8cceccb22552

commit d7d78924ccedc33db62fb368ddce8cceccb22552
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Fri Dec 21 23:34:26 2018

Adds ability to pass tags to Typ from browser test suites

This CL will allow browser test suites to generate tags and
then pass them to Typ. These tags will be used to get
expectations for a test. This CL also adds several test cases in
browser_test_runner_unittest.py which passes an expectations
file to Typ through browser_runner_test.Run(). This test also
checks that generated tags can be passed from a browser test
class.

R=nednguyen@google.com

Bug:  chromium:835690 
Change-Id: I9235b5c7b40b2db9bccb0871e254e221971d9187
Reviewed-on: https://chromium-review.googlesource.com/c/1387920
Commit-Queue: Rakib Hasan <rmhasan@google.com>
Reviewed-by: Ned Nguyen <nednguyen@google.com>

[modify] https://crrev.com/d7d78924ccedc33db62fb368ddce8cceccb22552/telemetry/telemetry/testing/run_browser_tests.py
[add] https://crrev.com/d7d78924ccedc33db62fb368ddce8cceccb22552/telemetry/examples/browser_tests/generate_tags_test.py
[modify] https://crrev.com/d7d78924ccedc33db62fb368ddce8cceccb22552/telemetry/telemetry/testing/serially_executed_browser_test_case.py
[modify] https://crrev.com/d7d78924ccedc33db62fb368ddce8cceccb22552/telemetry/telemetry/testing/browser_test_runner_unittest.py

Comment 43 by bugdroid1@chromium.org, Dec 22

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b67b934895af5bd8d4635ee094178dfb7ac12b28

commit b67b934895af5bd8d4635ee094178dfb7ac12b28
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Sat Dec 22 01:29:38 2018

Roll src/third_party/catapult 442e7b273107..d7d78924cced (1 commits)

https://chromium.googlesource.com/catapult.git/+log/442e7b273107..d7d78924cced


git log 442e7b273107..d7d78924cced --date=short --no-merges --format='%ad %ae %s'
2018-12-21 rmhasan@google.com Adds ability to pass tags to Typ from browser test suites


Created with:
  gclient setdep -r src/third_party/catapult@d7d78924cced

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 
TBR=sullivan@chromium.org

Change-Id: Ica6cd56da591a27dc436714726306f34dd310c58
Reviewed-on: https://chromium-review.googlesource.com/c/1388961
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#618706}
[modify] https://crrev.com/b67b934895af5bd8d4635ee094178dfb7ac12b28/DEPS

Comment 44 by bugdroid1@chromium.org, Jan 4

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/865a64df6576494643d2eec97f9abbd89b37ed9c

commit 865a64df6576494643d2eec97f9abbd89b37ed9c
Author: Rakib M. Hasan <rmhasan@google.com>
Date: Fri Jan 04 18:31:59 2019

Removes expectations_file named argument from constructor of ProjectConfig

This CL is part of the effort to rename the expectations_file argument
of the telemetry.project_config.ProjectConfig constructor. In a previous
CL we removed the use of the expectations_file argument in
tools.perf.chrome_telemetry_build.ChromiumConfig constructor. In this
CL we will remove the argument from the ProjectConfig's classes constructor.

R=nednguyen@google.com

Bug:  chromium:835690 
Change-Id: I42a454228c8574f8a7700cc2e576b524727a7a23
Reviewed-on: https://chromium-review.googlesource.com/c/1394810
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Rakib Hasan <rmhasan@google.com>

[modify] https://crrev.com/865a64df6576494643d2eec97f9abbd89b37ed9c/telemetry/telemetry/project_config.py

Comment 45 by bugdroid1@chromium.org, Jan 4

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7f984ff61c9036482586af2405d34f772183f270

commit 7f984ff61c9036482586af2405d34f772183f270
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Fri Jan 04 20:46:33 2019

Roll src/third_party/catapult 51771a7cad87..865a64df6576 (1 commits)

https://chromium.googlesource.com/catapult.git/+log/51771a7cad87..865a64df6576


git log 51771a7cad87..865a64df6576 --date=short --no-merges --format='%ad %ae %s'
2019-01-04 rmhasan@google.com Removes expectations_file named argument from constructor of ProjectConfig


Created with:
  gclient setdep -r src/third_party/catapult@865a64df6576

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:835690 
TBR=sullivan@chromium.org

Change-Id: I6b46e480bc5490aa7ed2aa20efba15d664f538e8
Reviewed-on: https://chromium-review.googlesource.com/c/1396206
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#620056}
[modify] https://crrev.com/7f984ff61c9036482586af2405d34f772183f270/DEPS

Sign in to add a comment