I'll take a look at this.

Looks like the first failed roll (https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_swarming_rel/builds/382860) was a flake.

However, the second failed roll looks like the real thing (https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/126415), with lots of failing pixel tests. I don't know much about these pixel tests, but it does seem like they import Telemetry code, so this is where I'm going to focus.

This is the list of CLs in that roll that I pointed out: https://chromium.googlesource.com/catapult.git/+log/734f737c6b57..df3154eb043f

Of those CLs, the most likely culprit seems to be https://chromium.googlesource.com/catapult.git/+/f9a1ae1b32a8cbbe45936ad46ef0704320cd9fa0. It seems to be changing something to do with compositor rectangles, which corresponds nicely to the failure:

Traceback (most recent call last):
    _RunGpuTest at content\test\gpu\gpu_tests\gpu_integration_test.py:132
      self.RunActualGpuTest(url, *args)
    RunActualGpuTest at content\test\gpu\gpu_tests\pixel_integration_test.py:141
      tab, page.name, screenshot, page.expected_colors, dpr)
    _ValidateScreenshotSamples at content\test\gpu\gpu_tests\cloud_storage_integration_test_base.py:367
      self.GetParsedCommandLineOptions().test_machine_name)
    _CompareScreenshotSamples at content\test\gpu\gpu_tests\cloud_storage_integration_test_base.py:172
      str(actual_color.b) + "]")
    fail at .swarming_module\bin\Lib\unittest\case.py:410
      raise self.failureException(msg)
  AssertionError: Expected pixel at [1, 1] (actual pixel (1, 1))  to be [102, 77, 0] but got [94, 71, 0]
 
Just noting this for now: next I want to figure out what's going on with the Android device also.

Urgh, it looks like there are probably three problems: the PRESUBMIT problem, the GPU test problem, and the Android device problem.

Isn't this a tracing UI CL? https://chromium.googlesource.com/catapult.git/+/f9a1ae1b32a8cbbe45936ad46ef0704320cd9fa0

Does the tracing/tracing/ui/extras JS code get run during telemetry tests?

Good point. (I split that issue off into  issue 825210 , as it seems separate from the other CQ problems.)

Here's a link to the Catapult PRESUBMIT builder: https://luci-milo.appspot.com/buildbot/tryserver.client.catapult/Catapult%20Presubmit/

Here's a link to the Catapult Android builder: https://luci-milo.appspot.com/buildbot/tryserver.client.catapult/Catapult%20Android%20Tryserver/

It looks like the problem is a pylint failure:

** Presubmit ERRORS **
Pylint (477 files using ['--disable=cyclic-import'] on 8 cores) (16.80s) failed
************* Module benchmarks.simple_story_set
F:  5, 0: Unable to import 'telemetry' (import-error)
F:  6, 0: Unable to import 'telemetry' (import-error)
E: 11, 2: Use of super on an old style class (super-on-old-class)
E: 29, 2: Use of super on an old style class (super-on-old-class)
************* Module benchmarks.tbm_benchmark
F:  4, 0: Unable to import 'telemetry' (import-error)
F:  5, 0: Unable to import 'telemetry.web_perf' (import-error)
W:  9, 0: Class has no __init__ method (no-init)
W: 11,27: Unused argument 'options' (unused-argument)
************* Module browser_tests.failed_tests
F:  7, 0: Unable to import 'telemetry.testing' (import-error)
W: 10, 0: Class has no __init__ method (no-init)
W: 27, 0: Class has no __init__ method (no-init)
************* Module browser_tests.process_tests
F:  7, 0: Unable to import 'telemetry.testing' (import-error)
W: 10, 0: Class has no __init__ method (no-init)
W: 30, 0: Class has no __init__ method (no-init)
...

(this goes on for a very long time)

This is the first failure that I'm seeing: https://luci-milo.appspot.com/buildbot/tryserver.client.catapult/Catapult%20Presubmit/11357 around 5:40pm last night.

Interestingly, both this CL that passed the presubmit (https://luci-milo.appspot.com/buildbot/tryserver.client.catapult/Catapult%20Presubmit/11356) and this CL that failed (https://luci-milo.appspot.com/buildbot/tryserver.client.catapult/Catapult%20Presubmit/11358) seem to have the same got_revision (diffbase), indicating that something infrastructure-wise changed in between those CLs. I wonder if infra rolled the version of pylint that they use and pylint added more default-enforced lint rules?

I don't think my previous hypothesis of "pylint got rolled, more default lint rules" makes any sense. Some of these lint rules have been around for a long time, like import-error.

So it looks like our pylint version is provided by infra's depot_tools, and it looks like around 5:30 last night, depot_tools made a change that *prevented* depot_tools virtualenv from leaking into other repos (https://chromium.googlesource.com/chromium/tools/depot_tools/+/5025893cb6061b8a219750155cccd67b55079311). I wonder if Catapult's PRESUBMIT depended on that leakage to work correctly. If so, we probably want to find a way to specify for ourselves whatever version of pylint we were implicitly getting due to depot_tools.

(Also worth noting: if I run `git cl presubmit` on my local workstation, it also passes, which seems to suggest that there's some sort of dependency mismatch between what's running on the CQ and other machines. This seems to suggest that our dependencies might be a little too loosy-goosy.)

I just stumbled across https://www.google.com/url?q=https://groups.google.com/a/chromium.org/forum/?utm_medium%3Demail%26utm_source%3Dfooter%23!msg/chromium-dev/-D9E1V64NNY/RV-oWJVWBgAJ&sa=D&source=hangouts&ust=1521905931432000&usg=AFQjCNHy9_f2LFENvuDh460UvsbdWNRIBQ: it looks like we probably just have to figure out how to specify the version of pylint to use in https://chromium.googlesource.com/chromium/src.git/+/master/.vpython

It looks like in order to make this work, we probably need a pylint "wheel" though (package available from cipd) that we can specify in that vpython file. 

+Robbie, do you agree that this is the right path to head down? FWIW, this is passing on my local workstation with pylint 1.4.1.

CC'ing some other reviewers of https://chromium.googlesource.com/chromium/tools/depot_tools/+/5025893cb6061b8a219750155cccd67b55079311 but I don't think this warrants P0. Chromium's CQ is not broken. https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/?limit=200 is running fine. Some Catapult-side change has triggered this issue, thereby blocking rolls forward. Downgrading to P1.

Also CC'ing dnj@ because he provided many or most of the vpython wheels.

inclined to agree w/ prioritization in #21.

Sounds good. Thanks for the adjustment. Talking to Robbie now about how we might be able to fix this.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/23bd73535c4ca820910393e088ba62f868ad5fec

commit 23bd73535c4ca820910393e088ba62f868ad5fec
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Mar 23 19:05:27 2018

Clear $VPYTHON_CLEAR_PYTHONPATH when setting $PYTHONPATH.

On buildbot (only), we don't trust PYTHONPATH, so we have set
VPYTHON_CLEAR_PYTHONPATH to make the first invocation of vpython in a
given chain of python invocations clear PYTHONPATH.

This has the unfortunate side-effect that python scripts that set
PYTHONPATH and then call into vpython (like presubmit) end up
having their PYTHONPATH modifications removed.

As a workaround (until I clean up PYTHONPATH in buildbot for real), fix
presubmit_canned_checks so that it clears VPYTHON_CLEAR_PYTHONPATH when it
adjusts PYTHONPATH.

R=charlie@chromium.org, nodir@chromium.org, vadimsh@chromium.org

Bug:  825290 , 825174 
Change-Id: Ib5f73add1726fdf3c335d26fc0af76cfe3b747b2
Reviewed-on: https://chromium-review.googlesource.com/978632
Reviewed-by: Charlie Andrews <charliea@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/23bd73535c4ca820910393e088ba62f868ad5fec/presubmit_canned_checks.py
[modify] https://crrev.com/23bd73535c4ca820910393e088ba62f868ad5fec/recipes/trigger_recipe_roller.txt

Assigning this over to Ethan, as time is running out over here on the east coast. 

It looks like there are still PRESUBMIT failures, but the number is decreased drastically - down to just 5 or 6.

https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Presubmit/builds/11382/steps/presubmit/logs/stdio

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/325503ce9c14a4685fb36be0c7df9053e7b5ab9b

commit 325503ce9c14a4685fb36be0c7df9053e7b5ab9b
Author: depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Mar 23 23:25:39 2018

Roll src/third_party/depot_tools/ 728244f70..0d9ecc925 (3 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/728244f70a8b..0d9ecc925d38

$ git log 728244f70..0d9ecc925 --date=short --no-merges --format='%ad %ae %s'
2018-03-23 ehmaldonado Reland "gclient eval: Expand vars while parsing DEPS files"
2018-03-23 sergiyb Correct documentation for the --no-referenced-issues argument
2018-03-23 iannucci Clear $VPYTHON_CLEAR_PYTHONPATH when setting $PYTHONPATH.

Created with:
  roll-dep src/third_party/depot_tools
BUG= chromium:821199 , chromium:825290 , chromium:825174 


The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=agable@chromium.org

Change-Id: I58cbb0ca469176bb99a5a86d7504c986fa9b162b
Reviewed-on: https://chromium-review.googlesource.com/978865
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#545622}
[modify] https://crrev.com/325503ce9c14a4685fb36be0c7df9053e7b5ab9b/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/82a6480a3a0d67c1d8ad99809eafcbf87468fb52

commit 82a6480a3a0d67c1d8ad99809eafcbf87468fb52
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Mar 23 23:45:49 2018

[presubmit_canned_checks] Stop exporting sys.path into PYTHONPATH.

Again, sys.path shouldn't ever be exported into PYTHONPATH :)

R=eakuefner@chromium.org, nodir@chromium.org, vadimsh@chromium.org

Bug:  825290 , 825174 
Change-Id: Ia0594da3ff25972a08fdf74ae2aef5be79cbf3af
Reviewed-on: https://chromium-review.googlesource.com/978594
Reviewed-by: Ethan Kuefner <eakuefner@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/82a6480a3a0d67c1d8ad99809eafcbf87468fb52/presubmit_canned_checks.py
[modify] https://crrev.com/82a6480a3a0d67c1d8ad99809eafcbf87468fb52/recipes/trigger_recipe_roller.txt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/b4f1d4498c1ceed46868f003bbf6dae2097be5a6

commit b4f1d4498c1ceed46868f003bbf6dae2097be5a6
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Mar 23 23:53:59 2018

Fix actual pylint errors.

TBR=eakuefner@chromium.org, nodir@chromium.org, vadimsh@chromium.org

Bug:  825290 , 825174 
Change-Id: Ide4f03b0f9100a2110bad9d510921b46b2d4e43a
Reviewed-on: https://chromium-review.googlesource.com/979112
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/b4f1d4498c1ceed46868f003bbf6dae2097be5a6/tests/gclient_eval_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/dbbf350a34ab7991e635f91a70bd2d211eb921be

commit dbbf350a34ab7991e635f91a70bd2d211eb921be
Author: Robert Iannucci <iannucci@chromium.org>
Date: Sat Mar 24 00:01:34 2018

Fix trailing whitespace test.

And write the test better to avoid overzealous editor plugins :)

TBR=eakuefner@chromium.org, nodir@chromium.org, vadimsh@chromium.org

Bug:  825290 , 825174 
Change-Id: I0541d4d70bcb256c24cb659bae256fa0aacb6806
Reviewed-on: https://chromium-review.googlesource.com/979116
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/dbbf350a34ab7991e635f91a70bd2d211eb921be/tests/gclient_eval_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/fe230b67399bc68274f053fbb3e9706eda31cb10

commit fe230b67399bc68274f053fbb3e9706eda31cb10
Author: Ethan Kuefner <eakuefner@chromium.org>
Date: Sat Mar 24 00:52:56 2018

[Build] Fix tracing Python tests

Because Pylint now runs under vpython, and due to other changes pertaining to
how PYTHONPATH is set up before Pylint is is invoked, we need to now explicitly
include those third_party paths on which tracing depends when running Pylint.

Bug:  chromium:825174 
Change-Id: Ibe755229db77edfbed721d88d3338817a8210a44
TBR=nednguyen,sullivan,charliea
Reviewed-on: https://chromium-review.googlesource.com/978593
Reviewed-by: Ethan Kuefner <eakuefner@chromium.org>
Commit-Queue: Ethan Kuefner <eakuefner@chromium.org>

[modify] https://crrev.com/fe230b67399bc68274f053fbb3e9706eda31cb10/tracing/PRESUBMIT.py

This should be fixed now :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e6be243fdc31f9bda03caf3c206a8982b45e5a84

commit e6be243fdc31f9bda03caf3c206a8982b45e5a84
Author: depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Mar 24 01:38:36 2018

Roll src/third_party/depot_tools/ 0d9ecc925..82a6480a3 (1 commit)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/0d9ecc925d38..82a6480a3a0d

$ git log 0d9ecc925..82a6480a3 --date=short --no-merges --format='%ad %ae %s'
2018-03-23 iannucci [presubmit_canned_checks] Stop exporting sys.path into PYTHONPATH.

Created with:
  roll-dep src/third_party/depot_tools
BUG= chromium:825290 , chromium:825174 


The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=agable@chromium.org

Change-Id: Ic28a910f52c64051a17e2a9f94a3886f7bb237f2
Reviewed-on: https://chromium-review.googlesource.com/979114
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#545655}
[modify] https://crrev.com/e6be243fdc31f9bda03caf3c206a8982b45e5a84/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4b62356e5c9750122f05e4e15e9d9732bead6e70

commit 4b62356e5c9750122f05e4e15e9d9732bead6e70
Author: catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Mar 24 02:56:11 2018

Roll src/third_party/catapult/ c4e9b1332..fe230b673 (1 commit)

https://chromium.googlesource.com/catapult.git/+log/c4e9b1332818..fe230b67399b

$ git log c4e9b1332..fe230b673 --date=short --no-merges --format='%ad %ae %s'
2018-03-23 eakuefner [Build] Fix tracing Python tests

Created with:
  roll-dep src/third_party/catapult
BUG= chromium:825174 


The AutoRoll server is located here: https://catapult-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=sullivan@chromium.org

Change-Id: Ic39c3da2c175d4b435cc3cc7cb6c4989a3d812d9
Reviewed-on: https://chromium-review.googlesource.com/979170
Commit-Queue: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Reviewed-by: catapult-chromium-autoroll <catapult-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#545665}
[modify] https://crrev.com/4b62356e5c9750122f05e4e15e9d9732bead6e70/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cd97a8c02119790f363670851c2417fbb9236112

commit cd97a8c02119790f363670851c2417fbb9236112
Author: depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Sat Mar 24 04:59:02 2018

Roll src/third_party/depot_tools/ 82a6480a3..dbbf350a3 (2 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/82a6480a3a0d..dbbf350a34ab

$ git log 82a6480a3..dbbf350a3 --date=short --no-merges --format='%ad %ae %s'
2018-03-23 iannucci Fix trailing whitespace test.
2018-03-23 iannucci Fix actual pylint errors.

Created with:
  roll-dep src/third_party/depot_tools
BUG= chromium:825290 , chromium:825174 , chromium:825290 , chromium:825174 


The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=agable@chromium.org

Change-Id: If3ca536bc4f21f1f692b3c6ebdffae05203db93b
Reviewed-on: https://chromium-review.googlesource.com/979293
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#545672}
[modify] https://crrev.com/cd97a8c02119790f363670851c2417fbb9236112/DEPS