New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 725379 link

Starred by 3 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Several tests in org.chromium.components.crash.browser.CrashDumpManagerTest are flaky

Project Member Reported by chromium...@appspot.gserviceaccount.com, May 23 2017

Issue description

"org.chromium.components.crash.browser.CrashDumpManagerTest#testUploadMinidump_NullMinidumpPath" is flaky.

This issue was created automatically by the chromium-try-flakes app. Please find the right owner to fix the respective test/step and assign this issue to them. If the step/test is infrastructure-related, please add Infra-Troopers label and change issue status to Untriaged. When done, please remove the issue from Sheriff Bug Queue by removing the Sheriff-Chromium label.

We have detected 3 recent flakes. List of all flakes can be found at https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyaQsSBUZsYWtlIl5vcmcuY2hyb21pdW0uY29tcG9uZW50cy5jcmFzaC5icm93c2VyLkNyYXNoRHVtcE1hbmFnZXJUZXN0I3Rlc3RVcGxvYWRNaW5pZHVtcF9OdWxsTWluaWR1bXBQYXRoDA.

Flaky tests should be disabled within 30 minutes unless culprit CL is found and reverted. Please see more details here: https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriffing-bug-queues#triaging-auto-filed-flakiness-bugs
 
The log is:
============
java.lang.AssertionError
	at org.chromium.base.test.util.TestFileUtil.deleteFile(TestFileUtil.java:55)
	at org.chromium.components.crash.browser.CrashDumpManagerTest.tearDown(CrashDumpManagerTest.java:40)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:129)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
===============
Owner: isherman@chromium.org
Status: Assigned (was: Untriaged)
The test was added recently by isherman@.
Labels: -Sheriff-Chromium
I am disabling the test, the CL is https://codereview.chromium.org/2902103004.
Cc: gsennton@chromium.org
It looks like we're trying to delete all the files in the cache-directory - maybe there are files (or directories) in the cache-directory, that aren't cleaned before running this test?

Ilya: if it makes sense I think we should use a temporary directory for the tests instead of using the cache-directory directly.
Project Member

Comment 5 by bugdroid1@chromium.org, May 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/406f05e73b87ed54dd6b6ace45cea9385208c5ac

commit 406f05e73b87ed54dd6b6ace45cea9385208c5ac
Author: vitaliii <vitaliii@chromium.org>
Date: Wed May 24 15:08:26 2017

Disable CrashDumpManagerTest#testUploadMinidump_NullMinidumpPath (flaky)

TBR=gsennton@chromium.org,isherman@chromium.org

BUG=725379

Review-Url: https://codereview.chromium.org/2902103004
Cr-Commit-Position: refs/heads/master@{#474289}

[modify] https://crrev.com/406f05e73b87ed54dd6b6ace45cea9385208c5ac/components/crash/android/javatests/src/org/chromium/components/crash/browser/CrashDumpManagerTest.java

Comment 6 by mgiuca@chromium.org, May 26 2017

 Issue 726604  has been merged into this issue.

Comment 7 by mgiuca@chromium.org, May 26 2017

testUploadMinidump_NoCallback is also flaky (see  Issue 726604 ). Disabling and duping that bug into this.
Project Member

Comment 9 by bugdroid1@chromium.org, May 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/73fce28b25fd97073601ecb04ee5f88c16cc4ad2

commit 73fce28b25fd97073601ecb04ee5f88c16cc4ad2
Author: Matt Giuca <mgiuca@chromium.org>
Date: Fri May 26 07:44:45 2017

Disable CrashDumpManagerTest#testUploadMinidump_NoCallback (flaky).

TBR=gsennton@chromium.org,isherman@chromium.org

Bug: 725379
Change-Id: I20682c13346285b427b94f04300eb2056f0b5835
Reviewed-on: https://chromium-review.googlesource.com/517003
Reviewed-by: Matt Giuca <mgiuca@chromium.org>
Commit-Queue: Matt Giuca <mgiuca@chromium.org>
Cr-Commit-Position: refs/heads/master@{#474957}
[modify] https://crrev.com/73fce28b25fd97073601ecb04ee5f88c16cc4ad2/components/crash/android/javatests/src/org/chromium/components/crash/browser/CrashDumpManagerTest.java

Cc: isherman@chromium.org
 Issue 726976  has been merged into this issue.
Status: Started (was: Assigned)
 Issue 727506  has been merged into this issue.
Project Member

Comment 13 by bugdroid1@chromium.org, May 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6b0e73063c6d65e7172cd247e05006078a60cf23

commit 6b0e73063c6d65e7172cd247e05006078a60cf23
Author: isherman <isherman@chromium.org>
Date: Wed May 31 15:52:15 2017

Fix some flaky tests: Use a dedicated temp dir.

BUG=725379
TEST=CrashDumpManagerTest.*
R=gsennton@chromium.org

Review-Url: https://codereview.chromium.org/2911983008
Cr-Commit-Position: refs/heads/master@{#475924}

[modify] https://crrev.com/6b0e73063c6d65e7172cd247e05006078a60cf23/components/crash/android/javatests/src/org/chromium/components/crash/browser/CrashDumpManagerTest.java

Project Member

Comment 14 by bugdroid1@chromium.org, May 31 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f6c031634d0e361e90534d410121d25c991b3a80

commit f6c031634d0e361e90534d410121d25c991b3a80
Author: pkasting <pkasting@chromium.org>
Date: Wed May 31 20:02:39 2017

Revert of Fix some flaky tests: Use a dedicated temp dir. (patchset #1 id:1 of https://codereview.chromium.org/2911983008/ )

Reason for revert:
Looks to be causing consistent failures on Android, e.g.:

https://luci-milo.appspot.com/buildbot/chromium.linux/Android%20Tests/42292

Original issue's description:
> Fix some flaky tests: Use a dedicated temp dir.
>
> BUG=725379
> TEST=CrashDumpManagerTest.*
> R=gsennton@chromium.org
>
> Review-Url: https://codereview.chromium.org/2911983008
> Cr-Commit-Position: refs/heads/master@{#475924}
> Committed: https://chromium.googlesource.com/chromium/src/+/6b0e73063c6d65e7172cd247e05006078a60cf23

TBR=gsennton@chromium.org,isherman@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=725379

Review-Url: https://codereview.chromium.org/2915883003
Cr-Commit-Position: refs/heads/master@{#475991}

[modify] https://crrev.com/f6c031634d0e361e90534d410121d25c991b3a80/components/crash/android/javatests/src/org/chromium/components/crash/browser/CrashDumpManagerTest.java

 Issue 726138  has been merged into this issue.
Summary: Several tests in org.chromium.components.crash.browser.CrashDumpManagerTest are flaky (was: "org.chromium.components.crash.browser.CrashDumpManagerTest#testUploadMinidump_NullMinidumpPath" is flaky)
Hmm, I'm not sure why the revert in #14 was necessary -- that is, I'm not sure why these tests are still failing in some situations.  Gustav, any ideas?
Components: Internals>CrashReporting
I couldn't repro this on an O device.
The failures (at least here: https://chromium-swarm.appspot.com/task?id=3678480a2cb49510&refresh=10&show_raw=1 ) are running hammerheads on KTU84P (KitKat MR2), I'll flash one of those and see if I can repro.

I would say that the different failures here:
https://00e9e64bac2af4321a0dfa9ff8ca56d0e4f36a9588adea6da0-apidata.googleusercontent.com/download/storage/v1/b/chromium-result-details/o/html%2Fchrome_public_test_apk_Android%20Tests_42292_2017_05_31_T17_22_29-UTC?qk=AD5uMEvLa4a-rdo-67M3KerbsggFKuFZsjI7GuR5l6Nd4ttewrriTvZErrFz8AWxXvr6DI7mFMQeDI-RevS5-Hk_TK7v-ajPEFtYJ6WqZMhdgil88ZC91uccskLHx6UN1v8xlYCrOquDhbUcDbu6TnNY6Pf4ZHOAfMW8jgXzuKO6f-qEYkxx5wffbMr-wtHJLrT_axFonof3f2gPtA5NFJxDhnI-vuTsxJcEZak0td6_oD1BjsD8j8wZqejyeMh5Hs4Pr6lzEBfYvFZ6SViz7iQC_8RTYZcOtfeczVUV1ZmrA9JJkwdTSljbuNNO6JNVoEQ0YPXmj8Yg2KT8Y27FecCcH6AKJR5E2-m9XlJYbECFd0EdeCv9FLQVxUaBeuNkyxQcB8Gmw0oGuvpctKvNhKdwdVsZD7VO_NgQ6VFv6JVmJHXtgR2JAsXxm76vM1yYA-NArPV47d8rH2Qa4e1lCT564BK8CpbkS0hfyAU5TESd8JHbtL4f1_QI6yizQ9AfxejC6D_qXKp_ERhHUGk-IxVX1yU43iFFVguaawGMenYCfYGQHAGpakQrSbrktlg7fPM4sk2GTnjWEqcx2wKkZFmQRE5uO3grNQXq3EUhZ-7zasMhXll8PcS7HZIMASo144Tw43O8wulxZftBrOLczStQuf0kmFOVOIkORimD8lZHAuHBrRXZXuySyrcEDkr-bZJT7CENlk-GsWECSSMlZfpLG99LzKHJ-8ADwjymrzl5pBm-mOJ-egKygLZ9k81XL8xXjKZam6mB_HYDzWZ0zfON1Kp_5IbcHPpOlK5jVkpJn85QMss9r-zzBZBfD-AkZrSFLPkR7-wssmoItetIDEANDU6V8rSnQqCzqczF9h4xD728cy9Il9E


suggest that the temporary directory is never created.

java.io.IOException: open failed: ENOENT (No such file or directory)
	at java.io.File.createNewFile(File.java:946)
	at org.chromium.components.crash.browser.CrashDumpManagerTest.testUploadMinidump_MinidumpPathIsValid(CrashDumpManagerTest.java:99)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:129)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
Caused by: libcore.io.ErrnoException: open failed: ENOENT (No such file or directory)
	at libcore.io.Posix.open(Native Method)
	at libcore.io.BlockGuardOs.open(BlockGuardOs.java:110)
	at java.io.File.createNewFile(File.java:939)
	... 15 more


junit.framework.AssertionFailedError
	at org.chromium.components.crash.browser.CrashDumpManagerTest.testUploadMinidump_MinidumpPathIsDirectory(CrashDumpManagerTest.java:83)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:129)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)



java.io.IOException: open failed: ENOENT (No such file or directory)
	at java.io.File.createNewFile(File.java:946)
	at org.chromium.components.crash.browser.CrashDumpManagerTest.testUploadMinidump_NoCallback(CrashDumpManagerTest.java:47)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:129)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
Caused by: libcore.io.ErrnoException: open failed: ENOENT (No such file or directory)
	at libcore.io.Posix.open(Native Method)
	at libcore.io.BlockGuardOs.open(BlockGuardOs.java:110)
	at java.io.File.createNewFile(File.java:939)
	... 15 more
Ran these tests on a hammerhead with KTU84P and repeated 20 times - the tests pass each time. I don't know what's different on the tryserver bots. Do we have a way of running these tests on those bots?
#20: try jobs :) but please don't kick a bunch of try jobs on linux_android_rel_ng

There are indeed other ways to run tests on the same hardware. I'll follow up off-thread; it's kinda complicated at the moment.
Project Member

Comment 22 by chromium...@appspot.gserviceaccount.com, Jun 1 2017

Labels: Sheriff-Chromium
Detected 7 new flakes for test/step "org.chromium.components.crash.browser.CrashDumpManagerTest#testUploadMinidump_MinidumpPathIsDirectory". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNycAsSBUZsYWtlImVvcmcuY2hyb21pdW0uY29tcG9uZW50cy5jcmFzaC5icm93c2VyLkNyYXNoRHVtcE1hbmFnZXJUZXN0I3Rlc3RVcGxvYWRNaW5pZHVtcF9NaW5pZHVtcFBhdGhJc0RpcmVjdG9yeQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
I agree that it seems the temp directory is failing to be created.  I'm not sure what might cause that -- file permissions? Full disk?

Thanks for helping to investigate!  Please include me on off-thread discussion as well -- if nothing else, I might learn something =)
Labels: -Sheriff-Chromium Infra-Troopers
Labels: -Infra-Troopers
Not a trooper bug, fwiw. Writing up the instructions now.
Ugh, I just realized that the bot pkasting@ is referring to (that's failing with Ilya's CL) is 'Android Tests' on 'chromium.linux'. John, is it OK to use your instructions on that builder? (I tried running on linux_android_rel_ng / tryserver.chromium.android and that run succeeds).
Or is it possible to see runs from 31st of May for linux_android_rel_ng?
From here:
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng
I can only see the 200 most recent runs, and that's not enough to see results from when Ilya landed his CL.
#26: Yeah, that uses the same devices (hammerhead/KTU84P) as linux_android_rel_ng.

#27: Again, I'll follow up offline.
Project Member

Comment 29 by chromium...@appspot.gserviceaccount.com, Jun 3 2017

Labels: Sheriff-Chromium
Detected 3 new flakes for test/step "org.chromium.components.crash.browser.CrashDumpManagerTest#testUploadMinidump_MinidumpPathIsDirectory". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNycAsSBUZsYWtlImVvcmcuY2hyb21pdW0uY29tcG9uZW50cy5jcmFzaC5icm93c2VyLkNyYXNoRHVtcE1hbmFnZXJUZXN0I3Rlc3RVcGxvYWRNaW5pZHVtcF9NaW5pZHVtcFBhdGhJc0RpcmVjdG9yeQw. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).
Labels: -Sheriff-Chromium
https://chromium-review.googlesource.com/c/524042/ is out to disable the remaining tests.
Project Member

Comment 31 by bugdroid1@chromium.org, Jun 5 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/119740b795871b7c4fc374c3e13d15ebf20ced4f

commit 119740b795871b7c4fc374c3e13d15ebf20ced4f
Author: Colin Blundell <blundell@chromium.org>
Date: Mon Jun 05 09:51:28 2017

Disable remaining CrashDumpManager tests due to flake

At least one of these tests continues to exhibit flake, and there's no
reason to suppose that the other isn't subject to the same problems as
the rest of the tests in this suite, as it looks like it's going
through the same basic flow.

TBR=isherman@chromium.org

Bug: 725379
Change-Id: I1fa0747d9554e64b67f4c6ee11777d9096ff5572
Reviewed-on: https://chromium-review.googlesource.com/524042
Reviewed-by: Colin Blundell <blundell@chromium.org>
Commit-Queue: Colin Blundell <blundell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#476964}
[modify] https://crrev.com/119740b795871b7c4fc374c3e13d15ebf20ced4f/components/crash/android/javatests/src/org/chromium/components/crash/browser/CrashDumpManagerTest.java

According to the flakiness dashboard these tests did fail a couple of times when run using swarming:

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=chrome_public_test_apk&tests=testUploadMinidump
just after Ilya's CL landed. AFAICT they failed consistently (2/2 times) until pkasting@ landed the revert.

Sign in to add a comment