This is causing a large CQ backlog on https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/linux-chromeos-rel/, can we revert a CL? This is P0 because it's blocking every developer in Chrome.

Thanks!

Looks like an attempt to disable those tests was made in r537747.

Looks like that missed at least DirectoryTreeContextMenu/FileManagerBrowserTest.Test/13 which just flaked for me.

#5 Yes, an attempt was made on r537747 to help deal with  issue 804413 , and I'm semi-glad that not all of the cases of that issue were disabled ... tried looking back in time to determine when these flakes started

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests%20(with%20patch)&showAllRuns=true&tests=VideoPlayerBrowserTestInGuestMode.OpenSingleVideoOnDownloads

Result is "Aw Snap" for me, filed  issue 826959  about that browser crash.

@kbr, thanks for the report, and nod, I see your CL https://chromium-review.googlesource.com/952778 linux-chromeos-rel trys failed 4 times in a row.  2 of those were due FileManagerBrowserTests

== kbr@'s first try run (log attached) ==

ken try job #1, failure case 1
DirectoryTreeContextMenu/FileManagerBrowserTest.Test/4 FAILURE
[10230:10230:0405/023425.510967:INFO:CONSOLE(0)] "[FAIL] copyFromDirectoryTreeWithoutChaningCurrentDirectory: API Test Error in copyFromDirectoryTreeWithoutChaningCurrentDirectory: The error count is not 0.
Actual: 1 Expected: 0
- current bug reports it, multiple failures (see log)

ken try #1, failure case 2
DirectoryTreeContextMenu/FileManagerBrowserTest.Test/4 CRASHED
[10636:10636:0405/023427.047095:FATAL:private_api_file_system.cc(810)] Check failed: external_backend->CanHandleType(file_system_url.type()). GURL: type: -1
-  issue 804413  on file for that case

Deleted: DirectoryTreeContextMenu-FileManagerBrowserTest-Test:4.log.txt 1.1 MB

DirectoryTreeContextMenu-FileManagerBrowserTest-Test:4.log.txt 1.1 MB View Download

== kbr@'s fourth try run (log attached) ==

ken try #4, case 1
KeyboardOperations/FileManagerBrowserTest.Test/1 FAILED
[7934:7934:0405/155739.172339:INFO:CONSOLE(0)] "[FAIL] keyboardDeleteDownloads: API Test Error in keyboardDeleteDownloads: The error count is not 0.
Actual: 1 Expected: 0
- current bug reports it, multiple failures (see log)

ken try #4, case 2
KeyboardOperations/FileManagerBrowserTest.Test/1 CRASHED
29532:29532:0405/155636.254824:FATAL:private_api_file_system.cc(810)] Check failed: external_backend->CanHandleType(file_system_url.type()). GURL: type: -1
-  issue 804413  on file for this case

Deleted: KeyboardOperations-FileManagerBrowserTest.Test-1.log.txt 1.1 MB

KeyboardOperations-FileManagerBrowserTest.Test-1.log.txt 1.1 MB View Download

DriveSpecific/FileManagerBrowserTest.Test/4
DriveSpecific/FileManagerBrowserTest.Test/5

are both flaky on

https://ci.chromium.org/buildbot/chromium.chromiumos/linux-chromeos-rel/

Looking at the flakiness dashboard for DriveSpecific/FileManagerBrowserTest.Test*, I agree with #9. I see a little too much flakiness in these tests [1]. Sent a patch to disable them ...

[1] https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests%20(with%20patch)&tests=DriveSpecific%2FFileManagerBrowserTest.Test*

I have confirmed these tests in #0 were affected by  Issue 804413 .
- DirectoryTreeContextMenu/FileManagerBrowserTest.Test/4
- DriveSpecific/FileManagerBrowserTest.Test/4
- FileDisplay/FileManagerBrowserTest.Test/0
- KeyboardOperations/FileManagerBrowserTest.Test/8
- ]OpenAudioFiles/FileManagerBrowserTest.Test/3
- OpenAudioFiles/FileManagerBrowserTest.Test/6

 However, the others in #0 and #9 seemes to have failed by a tentative error with test infrastrucre not relevant to FileManagerBrowserTest.
- DriveSpecific/FileManagerBrowserTest.Test/5
- TabindexFocusDirectorySelected/FileManagerBrowserTestWithLegacyEventDispatch.Test/0
- TabindexFocusDownloads/FileManagerBrowserTestWithLegacyEventDispatch.Test/0

DriveSpecific/FileManagerBrowserTest.Test/5:
> [28742:28742:0411/033009.041057:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.

TabindexFocusDirectorySelected/FileManagerBrowserTestWithLegacyEventDispatch.Test/0:
> [981:981:0404/235029.861774:ERROR:broker_posix.cc(42)] Invalid node channel message
> [16422:16422:0404/235338.758645:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.

TabindexFocusDownloads/FileManagerBrowserTestWithLegacyEventDispatch.Test/0:
> [16653:16653:0405/001711.183341:ERROR:broker_posix.cc(42)] Invalid node channel message
> [16031:16031:0404/235334.790771:ERROR:network_type_pattern.cc(134)] NetworkTypePattern: wifi: Can not match empty type.

Are these tests disabled? I'm still seeing failures on my CQ jobs.

I looked at https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/linux-chromeos-rel/98963 (oshima@) which failed due to a linux-chromeos-rel exception.

Not sure if that is was the CQ job you were referring to, but it was about 5 hours ago.  Looking at linux-chromeos-rel for the last 200 runs, it has been the color purple state for ~10 hours:

https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/linux-chromeos-rel/?limit=200

The problem appears to be swarming shards failing with error:

./browser_tests: error while loading shared libraries: libwayland-egl.so.1: cannot open shared object file: No such file or directory.

And as an FIY, I am prepared to disable FileManagerBrowserTests if they flake too much, for some reasonable definition of that (see #10).

I spoke with the FileManagerBrowserTests owners, asking them if they'd be OK with me disabling all of their tests (my idea).  They where very polite about it (they are from Japan) but were obviously uncomfortable with my idea.  The better compromise I sought was to selectively disable tests should they flake too much, and the owners were happy with this approach.

They wanted to avoid what happened on  issue 804413 : ad hoc FileManagerBrowserTests disablement patches landed for tests that do not even have the symptom of that bug.  I found these ad hoc patches and sent patches to undo them: they had nothing to do with  issue 804413  in my analysis and testing, done prior to landing my patches that re-enable tests from the ad hoc bucket.

 Issue 813477  suggested the fix to deal with this test flakiness is to rewrite the FileManager test system.  I don't see how that solves the problem at hand; now I see gerrit 1007115 that looks to me like it'll change the behavior of common code used by FileManagerBrowserTests.  In the middle of this investigation, I'm not clear how that change helps the resolve the situation, I do worry that it'll make thing worse (@slangley).

kbr@ said it best:

> There are a large number of bugs filed about these tests which have sat apparently uninvestigated for years.

The fix is too investigate.  Find out why they are flaking, and resolve.  And we are moving toward resolve in  issue 831074 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d65b43d69f13e93abcb8a8d73a56abf6fc2d2cd6

commit d65b43d69f13e93abcb8a8d73a56abf6fc2d2cd6
Author: Noel Gordon <noel@chromium.org>
Date: Thu Apr 12 11:17:48 2018

[ChromeOS] Disable DriveSpecific/FileManagerBrowserTest/Test.*

This test group (5 tests in all) flakes more often than desired on chromium
trys (the CQ). Disable them while the flake cause is being investigated.

Note: these tests are only built and run on ChromeOS. TODO(noel) once issue
829310 is resolved, this change can be reverted.

Bug:  829310 
Change-Id: If6e30690d9076debfd988a1fa0d7c93575099a14
Reviewed-on: https://chromium-review.googlesource.com/1006746
Commit-Queue: Noel Gordon <noel@chromium.org>
Reviewed-by: Henrik Grunell <grunell@chromium.org>
Reviewed-by: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#550143}
[modify] https://crrev.com/d65b43d69f13e93abcb8a8d73a56abf6fc2d2cd6/chrome/browser/chromeos/file_manager/file_manager_browsertest.cc

 Issue 831717  has been merged into this issue.

I just had DirectoryTreeContextMenu/FileManagerBrowserTest.Test/2 fail on an unrelated change in https://ci.chromium.org/buildbot/tryserver.chromium.chromiumos/linux-chromeos-rel/100628,
so this is not only for DriveSpecific* tests.

Right, thank you for the report.  Saving logs, multiple failures, but not issue  Issue 804413 .

Deleted: DirectoryTreeContextMenu:FileManagerBrowserTest.Test:2.log.txt 928 KB

DirectoryTreeContextMenu:FileManagerBrowserTest.Test:2.log.txt 928 KB View Download

DirectoryTreeContextMenu:FileManagerBrowserTest.Test:2 https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests%20(with%20patch)&tests=DirectoryTreeContextMenu%2FFileManagerBrowserTest.Test%2F2%20

Read 20 build logs showing gray black, once instance of a flake found, flake rate estimate: 1 chance in 50 builds.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ce92ada91522695758a70d18e960194e5f071895

commit ce92ada91522695758a70d18e960194e5f071895
Author: Noel Gordon <noel@chromium.org>
Date: Tue Apr 17 08:30:34 2018

Rid our system of persistent FileManagerBrowserTest flakiness

Postpone the creation of the FilesApp component extensions AudioPlayer
FileManager, Gallery, and VideoPlayer in browser testing. Not doing so
leads to systemic browser test failures (flakiness) per the bugs.

Fix: prepare test resources first, load the component extensions, then
start testing. Failure was the component extensions were active before
the file resources needed by the tests were ready, causing flake-fails
when the component extensions tried to access those resources.

Bug:  831074 ,  804413 ,  829310 
Cq-Include-Trybots: master.tryserver.chromium.linux:closure_compilation
Change-Id: Ie794bac638f48c3855e0c1f1609e59aa615c31dd
Reviewed-on: https://chromium-review.googlesource.com/1013823
Commit-Queue: Noel Gordon <noel@chromium.org>
Reviewed-by: Naoki Fukino <fukino@chromium.org>
Reviewed-by: Tomasz Mikolajewski <mtomasz@chromium.org>
Reviewed-by: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551279}
[modify] https://crrev.com/ce92ada91522695758a70d18e960194e5f071895/chrome/browser/chromeos/file_manager/file_manager_browsertest_base.cc

Check failed: external_backend->CanHandleType(file_system_url.type()). GURL: type: -1, the most common symptom of this bug (see #7 #8) has not seen for 24 hours on the bots since the fix #20 landed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5ddea8cc359644ebd78aad045619906f3f4c41cc

commit 5ddea8cc359644ebd78aad045619906f3f4c41cc
Author: Noel Gordon <noel@chromium.org>
Date: Wed Apr 18 13:55:33 2018

Revert "[ChromeOS] Disable DriveSpecific/FileManagerBrowserTest/Test.*"

This reverts commit d65b43d69f13e93abcb8a8d73a56abf6fc2d2cd6.

Reason for revert: fixed by https://crrev.com/551279

Tbr: yamaguchi-san
Bug:  829310 

Original change's description:
> [ChromeOS] Disable DriveSpecific/FileManagerBrowserTest/Test.*
>
> This test group (5 tests in all) flakes more often than desired on chromium
> trys (the CQ). Disable them while the flake cause is being investigated.
>
> Note: these tests are only built and run on ChromeOS. TODO(noel) once issue
> 829310 is resolved, this change can be reverted.
>
> Bug:  829310 
> Change-Id: If6e30690d9076debfd988a1fa0d7c93575099a14
> Reviewed-on: https://chromium-review.googlesource.com/1006746
> Commit-Queue: Noel Gordon <noel@chromium.org>
> Reviewed-by: Henrik Grunell <grunell@chromium.org>
> Reviewed-by: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#550143}

TBR=grunell@chromium.org,noel@chromium.org,yamaguchi@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  829310 
Change-Id: I0156d9ac746cdd005da07bc9599dc117795941cc
Reviewed-on: https://chromium-review.googlesource.com/1016880
Commit-Queue: Noel Gordon <noel@chromium.org>
Reviewed-by: Noel Gordon <noel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551661}
[modify] https://crrev.com/5ddea8cc359644ebd78aad045619906f3f4c41cc/chrome/browser/chromeos/file_manager/file_manager_browsertest.cc

#11 yamaguchi@++, thank you for that list, very helpful.  I went through them in the groups you gave.

First group
- DirectoryTreeContextMenu/FileManagerBrowserTest.Test/4
 no flakes Test/0...Test/24 https://bit.ly/2qIV9GX
 re-enabled Test/2 Test/7 Test/10 Test/23 Test/24
- DriveSpecific/FileManagerBrowserTest.Test/4
 I re-enabled all of them on https://crrev.com/550143
 awaiting more data https://bit.ly/2qGUre9
 Test/4 Test/5 need more work, more on that later.
- FileDisplay/FileManagerBrowserTest.Test/0
 no flakes Test/0...Test/6 https://bit.ly/2HuyKYi
- KeyboardOperations/FileManagerBrowserTest.Test/8
 no flakes Test/0...Test/11 https://bit.ly/2Hx9ocg
 re-enabled Test/7 Test/10
- OpenAudioFiles/FileManagerBrowserTest.Test/3
 no flakes Test/0...Test/8 https://bit.ly/2vpjUgP
- FileDisplay/FileManagerBrowserTest.Test
 no flakes Test/0...Test/6 https://bit.ly/2HuyKYi

Apart from DriveSpecific/FileManagerBrowserTest.Test/4 & 5, these all look like they are headed in the right direction (fixed).

Note some in first group and related (had the symptom of this bug) got disabled while the fix was working its way through the bots.  I backed that out with various patches and reverts.

- DefaultTaskDialog/FileManagerBrowserTest.Test/0
 closed  issue 832742  fixed
- DirectoryTreeContextMenu/FileManagerBrowserTest.Test/10
 closed  issue 832798  fixed
- DirectoryTreeContextMenu/FileManagerBrowserTest.Test/7
 closed  issue 832799  fixed
- Transfer/FileManagerBrowserTest.Test/3
 closed  issue 831211  fixed, re-enabled Test/4 Test/5
- KeyboardOperations/FileManagerBrowserTest.Test/2
 closed  issue 833049  fixed
- KeyboardOperations/FileManagerBrowserTest.Test/7
 closed  issue 832192  fixed
- KeyboardOperations/FileManagerBrowserTest.Test/10
 closed  issue 833752  fixed

Second group.

- DriveSpecific/FileManagerBrowserTest.Test/5
  I re-enabled, does not look so great https://bit.ly/2EXofHy but none of these
  flakes made a chromium developer's try job red.
  Error is the a File Manager test extension though: 'message.replace is not a
  function.'  Not related to the current bug.  Test/4 has the same error, so we
  could move them off into another bug.

-TabindexFocusDirectorySelected/FileManagerBrowserTestWithLegacyEventDispatch.Test/0
 fixed.
-TabindexFocusDownloads/FileManagerBrowserTestWithLegacyEventDispatch.Test/0
-TabindexFocusDownloads/FileManagerBrowserTestWithLegacyEventDispatch.Test/2
 fixed.
-TabindexFocusDirectorySelected/FileManagerBrowserTestWithLegacyEventDispatch.Test/0
 fixed.
-TabindexFocusDownloads/FileManagerBrowserTestWithLegacyEventDispatch.Test/0
-TabindexFocusDownloads/FileManagerBrowserTestWithLegacyEventDispatch.Test/1
 fixed.

Finally, I linux-chromeos-rel flaking due to other factors as I write, and some of those hit the FileManagerBrowserTests and can fail them.

Issue 832981, seems common.

 Issue 829306 , can hit a PASSED FileManagerBrowserTests, being worked on.

Issue XXXXXX, check stop when extensions unload. FileManagerBrowserTests (since we use lots of extensions) sometimes get hit.  A group of bugs around

Check failed: !browser_client || browser_client->IsShuttingDown() || did_respond() || ignore_all_did_respond_for_testing_do_not_use 

https://bugs.chromium.org/p/chromium/issues/list?can=2&q=%21browser_client+%7C%7C+browser_client-%3EIsShuttingDown%28%29+%7C%7C+did_respond%28%29+%7C%7C+ignore_all_did_respond_for_testing_do_not_use&colspec=ID+Pri+M+Stars+ReleaseBlock+Component+Status+Owner+Summary+OS+Modified&x=m&y=releaseblock&cells=ids

Seems to affect more than just FileManager, has been around for near on two years.

Can we revert this CL https://chromium-review.googlesource.com/c/chromium/src/+/1016880/3/chrome/browser/chromeos/file_manager/file_manager_browsertest.cc. The tests should be re-disabled until additional fix is in place (https://ci.chromium.org/buildbot/chromium.chromiumos/linux-chromeos-rel/)

#28 Thanks for the ping.  Those particular tests were re-enabled to gather data about how they fail (I really don't know how to fix things without data about how said things breaks :).  We got 10 hours worth of data, tests

 DriveSpecific/FileManagerBrowserTest.Test/2
 DriveSpecific/FileManagerBrowserTest.Test/4
 DriveSpecific/FileManagerBrowserTest.Test/5

need more work.  Mentioned that in #25, unrelated bug, needs a new bug, so filed  issue 834507  about it.  Sending a patch to revert just these 3 tests only.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6abcea2f6e43e43c3fa4048007ddd461f747eb9b

commit 6abcea2f6e43e43c3fa4048007ddd461f747eb9b
Author: Noel Gordon <noel@chromium.org>
Date: Thu Apr 19 00:35:45 2018

Disable DriveSpecific/FileManagerBrowserTest.Test/2/4/5 browser tests

See https://bugs.chromium.org/p/chromium/issues/detail?id=829310#c29

Bug:  834507 ,  829310 
Change-Id: Ifd37d589110f6fabd7048c231884b8c007d9ecc0
Reviewed-on: https://chromium-review.googlesource.com/1018260
Commit-Queue: Noel Gordon <noel@chromium.org>
Reviewed-by: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551899}
[modify] https://crrev.com/6abcea2f6e43e43c3fa4048007ddd461f747eb9b/chrome/browser/chromeos/file_manager/file_manager_browsertest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fbc016f8498abf4d05e725bb439f5f37d1b5ddd7

commit fbc016f8498abf4d05e725bb439f5f37d1b5ddd7
Author: Noel Gordon <noel@chromium.org>
Date: Thu Apr 19 07:35:28 2018

Revert "Disable DriveSpecific/FileManagerBrowserTest.Test/2/4/5 browser tests"

This reverts commit 6abcea2f6e43e43c3fa4048007ddd461f747eb9b.

Reason for revert: Data collected overnight pointed out the problem (was due to a very recent change made while these tests were disabled). Fixed by patches landed on  issue 834507 . revert here to re-enable

  DriveSpecific/FileManagerBrowserTest.Test/2
  DriveSpecific/FileManagerBrowserTest.Test/4
  DriveSpecific/FileManagerBrowserTest.Test/5

and to find out if anything else causes them to flake.

Tbr: lucmult@
Bug:  834507 ,  829310 

Original change's description:
> Disable DriveSpecific/FileManagerBrowserTest.Test/2/4/5 browser tests
> 
> See https://bugs.chromium.org/p/chromium/issues/detail?id=829310#c29
> 
> Bug:  834507 ,  829310 
> Change-Id: Ifd37d589110f6fabd7048c231884b8c007d9ecc0
> Reviewed-on: https://chromium-review.googlesource.com/1018260
> Commit-Queue: Noel Gordon <noel@chromium.org>
> Reviewed-by: Tatsuhisa Yamaguchi <yamaguchi@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#551899}

TBR=noel@chromium.org,yamaguchi@chromium.org

Change-Id: I284f3ae85d90916d1f9ad7485c3061419ace7565
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  834507 ,  829310 
Reviewed-on: https://chromium-review.googlesource.com/1018563
Reviewed-by: Noel Gordon <noel@chromium.org>
Commit-Queue: Noel Gordon <noel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551964}
[modify] https://crrev.com/fbc016f8498abf4d05e725bb439f5f37d1b5ddd7/chrome/browser/chromeos/file_manager/file_manager_browsertest.cc

re revert in #32, - DriveSpecific/FileManagerBrowserTest.Test/* all tests in that group looking good to me right now, https://bit.ly/2EXofHy

#36 ahem, test are:

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests%20(with%20patch)&tests=DriveSpecific%2FFileManagerBrowserTest

Re: #26  Check failed: !browser_client || browser_client->IsShuttingDown() || did_respond() || ignore_all_did_respond_for_testing_do_not_use

That problem now has it own (separate)  issue 668680 .

Check failed: external_backend->CanHandleType(file_system_url.type()). GURL: type: -1, has not seen for 48 hours on the bots since the fix #20 landed.

And now I need a lucky number ...

.. and a decent meal, and also a decent sleep :)