It looks like many of the tests are UNKNOWN because before they run/finish, DatabaseIdentifierTest crashes with "[FATAL:url_canon_icu.cc(104)] Check failed: false. failed to open UTS46 data with error: 4"

The check fail is here: https://code.google.com/p/chromium/codesearch#chromium/src/url/url_canon_icu.cc&sq=package:chromium&type=cs&l=104

Error 4 is U_FILE_ACCESS_ERROR ("The requested file cannot be found")

jbroman suggests that the blink_platform_unittests.isolate file needs some more info, like webkit_unit_tests.isolate. Created https://codereview.chromium.org/1748203002 to try that.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0d486bcc84474c58e5060fe8bc1349f52e690717

commit 0d486bcc84474c58e5060fe8bc1349f52e690717
Author: suzyh <suzyh@chromium.org>
Date: Wed Mar 02 02:08:15 2016

Add data to blink_platform_unittests.isolate

The blink_platform_unittests target is failing on the
"Android Tests (trial)(dbg)" bot and some of the failures suggest
that it is due to files not being available on the device.
This patch adds an include and a data path to the
blink_platform_unittests.isolate file, as used in
webkit_unit_tests.isolate, to address this.

BUG= 590888 

Review URL: https://codereview.chromium.org/1748203002

Cr-Commit-Position: refs/heads/master@{#378655}

[modify] https://crrev.com/0d486bcc84474c58e5060fe8bc1349f52e690717/third_party/WebKit/Source/platform/blink_platform_unittests.isolate

Well, that didn't help much.

The "failed to open UTS46 data with error: 4" check fail is still there, and the FontCacheAvailabilityTests are no longer failing, but only because they're not there any more, not because I actually fixed something.

https://build.chromium.org/p/chromium.fyi/builders/Android%20Tests%20%28trial%29%28dbg%29/builds/9202/steps/test_report/logs/stdio
"""
gtest results for Android Tests (trial)(dbg) build 9202:
...
blink_platform_unittests      ALL: 661       PASS: 641      SKIP: 0        FAIL: 1        CRASH: 0       TIMEOUT: 0     UNKNOWN: 19
"""

https://build.chromium.org/p/chromium.fyi/builders/Android%20Tests%20%28trial%29%28dbg%29/builds/9202/steps/blink_platform_unittests/logs/stdio
"""
C   27.089s Main  [==========] 661 tests ran.
C   27.089s Main  [  PASSED  ] 641 tests.
C   27.090s Main  [  FAILED  ] 20 tests, listed below:
C   27.090s Main  [  FAILED  ] LocaleICUTest.reversible
C   27.090s Main  [  FAILED  ] DatabaseIdentifierTest.CreateIdentifierAllHostChars (UNKNOWN)
C   27.090s Main  [  FAILED  ] DatabaseIdentifierTest.CreateIdentifierFromSecurityOrigin (UNKNOWN)
C   27.090s Main  [  FAILED  ] FontCacheAndroid.fallbackFontForCharacter (UNKNOWN)
C   27.090s Main  [  FAILED  ] KURLTest.Offsets (UNKNOWN)
C   27.090s Main  [  FAILED  ] KURLTest.PathAfterLastSlash (UNKNOWN)
C   27.090s Main  [  FAILED  ] KURLTest.UserPass (UNKNOWN)
C   27.090s Main  [  FAILED  ] KURLTest.Valid_HTTP_FTP_URLsHaveHosts (UNKNOWN)
C   27.090s Main  [  FAILED  ] OriginAccessEntryTest.AllowSubdomainsTest (UNKNOWN)
C   27.090s Main  [  FAILED  ] OriginAccessEntryTest.DisallowSubdomainsTest (UNKNOWN)
C   27.090s Main  [  FAILED  ] PaintControllerTest.SmallPaintControllerHasOnePaintChunk (UNKNOWN)
C   27.090s Main  [  FAILED  ] SchemeRegistryTest.PartialCSPBypass (UNKNOWN)
C   27.090s Main  [  FAILED  ] ScrollAnimatorTest.MainThreadStates (UNKNOWN)
C   27.090s Main  [  FAILED  ] SecurityOriginTest.LocalAccess (UNKNOWN)
C   27.090s Main  [  FAILED  ] SecurityOriginTest.Suborigins (UNKNOWN)
C   27.090s Main  [  FAILED  ] SecurityPolicyTest.TrustworthyWhiteList (UNKNOWN)
C   27.090s Main  [  FAILED  ] testsWithPlatformSetUp/PurgeableVectorTestWithPlatformSupport.clear/1 (UNKNOWN)
C   27.091s Main  [  FAILED  ] testsWithPlatformSetUp/PurgeableVectorTestWithPlatformSupport.clearDoesNotResetLockCounter/1 (UNKNOWN)
C   27.091s Main  [  FAILED  ] testsWithPlatformSetUp/PurgeableVectorTestWithPlatformSupport.reserveCapacityDoesNotChangeSize/0 (UNKNOWN)
C   27.091s Main  [  FAILED  ] testsWithPlatformSetUp/PurgeableVectorTestWithPlatformSupport.reserveCapacityDoesNotChangeSize/1 (UNKNOWN)
C   27.091s Main  
C   27.091s Main  20 FAILED TESTS
"""

On the other hand, there is a warning in the blink_platform_unittests stdio:
"W    0.020s Main  No isolate file provided. No data deps will be pushed."

So perhaps I'm missing a step in getting the isolate file actually onto the bot.

I think you're on the right track here. There's also this line raised during test setup (when ICU is initialized):

I   17.849s run_tests_on_device(03848f07f0e94615)  [ERROR:icu_util.cc(157)] Invalid file descriptor to ICU data received.

And indeed, blink_heap_unittests does have --isolate_file_path specifed, while blink_platform_unittests does not:
https://code.google.com/p/chromium/codesearch#chromium/src/testing/buildbot/chromium.fyi.json&l=104

I was just looking at this. But that's a BlinkHeapUnitTests.isolate, which is different from blink_heap_unittests.isolate, and I don't know what its function is. Looking for blink_heap_unittests.isolate and blink_platform_unittests.isolate in codesearch I don't see any difference.

On a different tack, ericwilligers has noticed that blink_platform in Source/platform/blink_platform.gyp contains
"""
      '<(DEPTH)/third_party/icu/icu.gyp:icui18n',
      '<(DEPTH)/third_party/icu/icu.gyp:icuuc',
"""
whereas blink_platform_unittests in Source/platform/blink_platform_tests.gyp does not, which could be relevant.

My two cents: I'd expect the linkage with icu.gyp to be a non-issue, because blink_platform_unittests already depends on blink_platform, and because the first signs of trouble are encountered before any code in platform is run.

I suspect it's a bug that there are two .isolate files for blink_heap_unittests, but I don't know. What does seem reasonable is that neither chromium.fyi.json nor gtest_test_instance.py associates the unit tests step with the isolate. My money's still on the isolate as the issue.

Yeah, I think you're right. There's something weird with blink_heap_unittests and webkit_unit_tests which both have this problem of two .isolate files. I'll try pointing chromium.fyi.json and chromium.android.json(*) at the blink_platform_unittests.isolate file and see what happens.

(*) since https://codereview.chromium.org/1677433002 does both.

Patch I wrote for comment 12: https://codereview.chromium.org/1757773002
Commit is failing; the win_chromium_x64_rel_ng bot is consistently failing at the "test results" stage, but I don't understand why.

jbudorick: Do you know the magic incantations needed to get the isolate file on the bots correctly?

Looks like the commit failure was a flake, or was fixed by me rebasing. Let's see how this goes on the waterfall bots.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5be96e51bdc0bd354d3b9cf8ca77b7fc92e40bac

commit 5be96e51bdc0bd354d3b9cf8ca77b7fc92e40bac
Author: suzyh <suzyh@chromium.org>
Date: Fri Mar 04 04:34:44 2016

Add blink_platform_unittests.isolate file to bot json files.

The "Android Tests (trial)(dbg)" bot is running blink_platform_unittests
but does not appear to be picking up the .isolate file with necessary
configuration. This patch modifies both chromium.fyi.json and
chromium.android.json (following https://crrev.com/1677433002) to refer
to the blink_platform_unittests.isolate file.

BUG= 590888 

Review URL: https://codereview.chromium.org/1757773002

Cr-Commit-Position: refs/heads/master@{#379212}

[modify] https://crrev.com/5be96e51bdc0bd354d3b9cf8ca77b7fc92e40bac/testing/buildbot/chromium.android.json
[modify] https://crrev.com/5be96e51bdc0bd354d3b9cf8ca77b7fc92e40bac/testing/buildbot/chromium.fyi.json

OK, that seems to have been reasonably successful.

https://build.chromium.org/p/chromium.fyi/builders/Android%20Tests%20%28trial%29%28dbg%29/builds/9251/steps/blink_platform_unittests/logs/stdio
"""
C   27.748s Main  [==========] 662 tests ran.
C   27.748s Main  [  PASSED  ] 656 tests.
C   27.748s Main  [  FAILED  ] 6 tests, listed below:
C   27.748s Main  [  FAILED  ] ContiguousContainerTest.AllocateLots (UNKNOWN)
C   27.748s Main  [  FAILED  ] FontCacheAndroid.fallbackFontForCharacter (UNKNOWN)
C   27.748s Main  [  FAILED  ] OriginAccessEntryTest.IPAddressTest (UNKNOWN)
C   27.748s Main  [  FAILED  ] RunSegmenterTest.ArabicHangul (UNKNOWN)
C   27.748s Main  [  FAILED  ] SecurityOriginTest.Suborigins (UNKNOWN)
C   27.748s Main  [  FAILED  ] ThreadSafeDataTransportTest.setData (UNKNOWN)
C   27.748s Main  
C   27.748s Main  6 FAILED TESTS
"""

In FontCacheAndroid.fallbackFontForCharacter there's two warnings about font files
"[SkFontMgr Android Parser] '/system/etc/fonts.xml' could not be opened"
"[SkFontMgr Android Parser] '/vendor/etc/fallback_fonts.xml' could not be opened"
then an assertion failure "ASSERTION FAILED: m_purgePreventCount"

In SecurityOriginTest.Suborigins there's a warning
"[WARNING] ../../testing/gtest/src/gtest-death-test.cc:834:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 12 threads."
and then it crashed.

I'll look into this more next week.

FontCacheAndroid.fallbackFontForCharacter:
If I understand correctly, the m_purgePreventCount assertion failure comes because the test calls fontDataFromFontPlatformData with shouldRetain == DoNotRetain but a FontCachePurgePreventer object has not been created.
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/platform/fonts/FontCache.h&sq=package:chromium&l=213&dr=C&rcl=1457541024

I do not know if the "'...xml' could not be opened" warnings are important here, and I haven't established whether the 'testing/data' clause I added to the isolate file in https://codereview.chromium.org/1748203002 is making a difference.

drott: You have been working on this code and the FontCacheTest recently. Would you be able to take a look at this failure and help me fix it? You might also be interested in the FontCacheAvailabilityTest failures that were present when I first opened this bug.

Suzy, thanks for working on bringing blink_platform_unittests to android/arm.

The original FontCacheAvailability tests have been reverted, but I am indeed interested in getting these blink platform tests to work.

The .xml warnings are critical: Without having access to fonts.xml, Blink on Android cannot access any information about which fonts are available on the system and thus will not be able to instantiate any fonts, which I suspect to cause to assertion failure ultimately.

Is there anything peculiar about the setup that these tests run in? Is there a fonts.xml file and actual font files in the environment that this runs in?

I don't have a good understanding of how the bots are set up. jbudorick and dpranke may be able to help more. One thing we might need to do is add the .xml files to the 'variables' 'files' list of the isolate file, as in https://codereview.chromium.org/1748203002, but I don't know whether the path would be set so that the file is where it's expected to be. Do you know of another build target that is successfully running on an Android bot that similarly needs the .xml files? If so, there might be some setup we can copy from there.

Yes, for running font and text related tests this .xml file and the actual font files from /system/fonts (or so) would be needed. Unfortunately I am not familiar with running tests on  Android, but the closest would be the webkit_unittests, I believe.

jbudorick, dpranke, jbroman: I've been looking around but don't understand how basic environment setup like fonts is done on the bots. Can any of you help?


(Aside: the UNKNOWN test failures are now no longer bothering us thanks to https://codereview.chromium.org/1703863003/ Yay!)

I'd be happy to help starting Wednesday :)

Fixing component name after Monorail migration

They fail consistently on
https://uberchromegw.corp.google.com/i/chromium.android/builders/Marshmallow%2064%20bit%20Tester/ too.

If you can't get these test pass, disable them please. 

Leaving them on. Will adjust tools appropriately for sheriffs.

I haven't had any time to look into this further. jbudorick: have you had any luck?

Not yet.

(though mostly for lack of time to try)

SecurityOriginTest.Suborigins: fails because a RELEASE_ASSERT in SecurityOrigin::addSuborigin is (intentionally) triggered in the last line of the test, an EXPECT_DEATH call. Somewhat surprisingly (to me), the EXPECT_DEATH call passes if I switch the RELEASE_ASSERT to CHECK, as suggested here: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/wtf/Assertions.h&l=277

Given the apparent security implications, I have no idea if this is ok to do or not. Regardless, I'm going to look into the implementation of EXPECT_DEATH more, as I didn't expect it to succeed on Android because I thought it was dependent on fork() and had been planning to simply #ifdef this test out of Android builds.

FontCacheAndroid.fallbackFontForCharacter fails on debug builds builds due to an ASSERT in FontCache::fontDataFromFontPlatformData on m_purgePreventCount being nonzero. AFAICT the failure is limited to Android only because no other platform is testing its fallbackFontForCharacter implementation (i.e., I think Linux would similarly fail if it were running the same kind of test).

The easy way to "fix" this would be to add a FontCachePurgePreventer somewhere (e.g., in the test), but I have no idea if that's an appropriate fix. I'll try to revisit this one later.

Blink>ToolsTest renamed to Blink>Infra

Ping? Just checking in on this during bug triage. Is there any more progress on this? Should it be made status: Available, and/or split up into multiple bugs for individual tests?

re #29 SecurityOriginTest.Suborigins:

That RELEASE_ASSERT is under EXPECT_DEATH, so it's ok. The test script is misinterpreting the result of the test. It actually passed, but because the test prints out both "CRASHED" and "PASSED", the script just took it as crashed instead.

From logcat:
06-02 22:59:09.270 10533 10533 I cr_NativeTest: >>ScopedMainEntryLogger
06-02 22:59:09.272 10533 10533 I cr_NativeTest: Note: Google Test filter = SecurityOriginTest.Suborigins
06-02 22:59:09.274 10533 10533 I cr_NativeTest: [==========] Running 1 test from 1 test case.
06-02 22:59:09.276 10533 10533 I cr_NativeTest: [----------] Global test environment set-up.
06-02 22:59:09.277 10533 10533 I cr_NativeTest: [----------] 1 test from SecurityOriginTest
06-02 22:59:09.280 10533 10533 I cr_NativeTest: [ RUN      ] SecurityOriginTest.Suborigins
06-02 22:59:09.283 10533 10533 I cr_NativeTest: [WARNING] ../../testing/gtest/src/gtest-death-test.cc:834:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test detected 12 threads.
06-02 22:59:09.286 10533 10533 I cr_NativeTest: [ CRASHED      ]
06-02 22:59:09.287 10533 10533 I cr_NativeTest: À[       OK ] SecurityOriginTest.Suborigins (284 ms)
06-02 22:59:09.289 10533 10533 I cr_NativeTest: [----------] 1 test from SecurityOriginTest (285 ms total)
06-02 22:59:09.292 10533 10533 I cr_NativeTest: [----------] Global test environment tear-down
06-02 22:59:09.293 10533 10533 I cr_NativeTest: [==========] 1 test from 1 test case ran. (287 ms total)
06-02 22:59:09.294 10533 10533 I cr_NativeTest: [  PASSED  ] 1 test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d5db1ef9a94f0d7e58118c9ff4822df7b6f56317

commit d5db1ef9a94f0d7e58118c9ff4822df7b6f56317
Author: thakis <thakis@chromium.org>
Date: Wed Nov 02 18:00:28 2016

Don't run blink_platform_unittests, browser_tests on x64 Android bot.

They don't pass on Android yet.

BUG=661623,611756, 584508 , 590888 

Review-Url: https://codereview.chromium.org/2473463003
Cr-Commit-Position: refs/heads/master@{#429337}

[modify] https://crrev.com/d5db1ef9a94f0d7e58118c9ff4822df7b6f56317/testing/buildbot/chromium.fyi.json

It looks like the builder mentioned no longer exists:
https://build.chromium.org/p/chromium.fyi/builders/Android%20Tests%20%28trial%29%28dbg%29

I assume it's been replaced by Android Builder (dbg)? Which doesn't run blink_platform_unittests.
https://build.chromium.org/p/chromium.fyi/builders/Android%20Builder%20%28dbg%29

What are the next steps for this bug?

The trial bot got moved to chromium.android.fyi: https://build.chromium.org/p/chromium.android.fyi/builders/Android%20Tests%20%28trial%29%28dbg%29

... where it seems they're not being built, so that's probably the next step. After that, it'd be figuring out why the remaining tests are failing.

I haven't had time to look at this in the last year, so I'm releasing it.

Is this still applicable?

Strictly speaking, no, as the trial bot has been turned down. The suite passing on Android is still applicable AFAIK, and it'd still block  issue 584508 .