+1

I discovered that infra.git/appengine/PRESUBMIT.py forks depot_tools's pylint stuff. I'm not sure why though.

Additionally, there's something in that PRESUBMIT.py file which puts sys.path on PYTHONPATH, which is a bug.

This is almost 100% guaranteed to be a PYTHONPATH bug in PRESUBMIT.py that was previously papered over by depot_tools exporting every path it could think of onto PYTHONPATH.

This only affects Findit and not other python GAE apps?

This is not specific to Findit.
At least it affected chromium-try-flakes too in presubmit checking https://chromium-review.googlesource.com/c/infra/infra/+/996015

blocking work on monorail too

Robbie, I think this is related to your changes to vpython/sys.path/PYTHONPATH and you understand the problem the most, thus I'll assign this to you. 

I'm disabling pylint for appengine/* for now, to unblock development. Please follow up on the bug, or we risk end up without pylint forever...

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6747afeb41062a3231ac6709641852f66c66e28d

commit 6747afeb41062a3231ac6709641852f66c66e28d
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 04 22:25:22 2018

Disable pylint for appengine/*, it is non-trivially broken.

R=tandrii@chromium.org, iannucci@chromium.org
BUG= 828677 

Change-Id: Id6397ee98fec57f5dab4bda5647de8b50ad24b53
Reviewed-on: https://chromium-review.googlesource.com/996567
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/6747afeb41062a3231ac6709641852f66c66e28d/appengine/PRESUBMIT.py

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Is there someone else who can take this on?

This is making Monorail team have no presubmit.

A P0 with OOO owner for >= 1 month makes me a sad girl.

From comment 12 it looks like development is unblocked so is this actually a P1 now? Andrii, you're now our vpython expert (sorry), do you think this is something you can take a look at?

1. It's not correct that there is no presubmit. Only pylint isn't run on appengine/* files.

2. This is purely incorrect PYTHONPATH issue as Robbie wrote, vpython broke a hack here, likely this one: https://chromium-review.googlesource.com/c/infra/infra/+/996567/2/appengine/PRESUBMIT.py#62
Solution is likely to adopt smth like this: https://chromium.googlesource.com/infra/infra/+/refs/changes/67/996567/2/PRESUBMIT.py#102 
But it'll take some trial and error :( 

3. expectations: I treat it as Pri2 because till ~Thursday I have more important issues to resolve.

It has been much longer than ~Thursday.

Can someone from foundations please pick this up?

Andrii, how much work do you think this is to fix? It would be good to get linting back for the appengine directory since it frequently changes.

Anywhere from lucky 1 hour to 2 days of untangling PYTHONPATH assumptions in each layer of subprocess call stack :(

Foundations team, can you let us know when you plan a fix for this? 

Jao-ke offered to take a look.

This is definitely still a p2 though. Can you clarify why this is actually a p1 now?

+monorail engs

My understanding is that a Monorail release had to be rolled back due to an issue this would have caught, if running on presubmit as the team expected.  They know to run manually right now, but doing this for ~months is error prone.

Ok, thanks for clarifying, let's keep it at p1 then.

The main reason this is taking a while is that Robbie has the most knowledge of how to untangle this code and he's not back from leave until next week. Jao-ke will have a look but if it looks like it will take several days to fix versus a few hours for Robbie we might just have to wait until next week anyway.

Jao-ke, would you mind posting an update tomorrow with anything you've discovered? Thanks!

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/7895379d203f8064b8b206a3cd7f28ebb7fea8da

commit 7895379d203f8064b8b206a3cd7f28ebb7fea8da
Author: Jao-ke Chin-Lee <jchinlee@chromium.org>
Date: Thu May 31 04:18:33 2018

Revert "Disable pylint for appengine/*, it is non-trivially broken."

This reverts commit 6747afeb41062a3231ac6709641852f66c66e28d.

Reason for revert: presubmit no longer failing

Original change's description:
> Disable pylint for appengine/*, it is non-trivially broken.
> 
> R=​tandrii@chromium.org, iannucci@chromium.org
> BUG= 828677 
> 
> Change-Id: Id6397ee98fec57f5dab4bda5647de8b50ad24b53
> Reviewed-on: https://chromium-review.googlesource.com/996567
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Reviewed-by: Shuotao Gao <stgao@chromium.org>
> Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

TBR=iannucci@chromium.org,vadimsh@chromium.org,stgao@chromium.org,tandrii@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  828677 
Change-Id: Id740edeff406f71e4a662494b04062d55d33ed20
Reviewed-on: https://chromium-review.googlesource.com/1079731
Reviewed-by: Jao-ke Chin-Lee <jchinlee@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Jao-ke Chin-Lee <jchinlee@chromium.org>

[modify] https://crrev.com/7895379d203f8064b8b206a3cd7f28ebb7fea8da/appengine/PRESUBMIT.py

Pylint doesn't seem to fail any more, so I've reenabled it in presubmit and will keep an eye on the builder.

The NextAction date has arrived: 2018-05-31

Pylint is up and running again. There are some failing jobs due to pylint errors that snuck in while pylint was disabled, which is expected.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/190f8a76875e8dfecd57f8326217a32edbc26d7c

commit 190f8a76875e8dfecd57f8326217a32edbc26d7c
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Thu May 31 18:54:11 2018

[monorail] Make linter shut up.

Bug:  828677 
Change-Id: Ie26178a1084fbf7c35bc0fda8f2e1ad58d5ad0e0
Reviewed-on: https://chromium-review.googlesource.com/1081151
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>
Reviewed-by: Jason Robbins <jrobbins@chromium.org>
Reviewed-by: Jeff Carpenter <jeffcarp@chromium.org>

[modify] https://crrev.com/190f8a76875e8dfecd57f8326217a32edbc26d7c/appengine/monorail/services/test/issue_svc_test.py
[modify] https://crrev.com/190f8a76875e8dfecd57f8326217a32edbc26d7c/appengine/monorail/tracker/test/issuedetail_test.py
[modify] https://crrev.com/190f8a76875e8dfecd57f8326217a32edbc26d7c/appengine/monorail/services/issue_svc.py
[modify] https://crrev.com/190f8a76875e8dfecd57f8326217a32edbc26d7c/appengine/monorail/tracker/issueentry.py
[modify] https://crrev.com/190f8a76875e8dfecd57f8326217a32edbc26d7c/appengine/monorail/businesslogic/test/work_env_test.py
[modify] https://crrev.com/190f8a76875e8dfecd57f8326217a32edbc26d7c/appengine/monorail/businesslogic/work_env.py

...and have confirmed that CLs touching python files do get correctly linted.

Great! Glad things are running again even if we don't know why it was broken for a period.