I don't think my CL has anything with loading resources on Android. Maybe the other Android-specific change?

https://chromium-review.googlesource.com/c/1340002

amalova@, any ideas?

- this is not Android specific, many builders fail.
- your CL touched fonts and the exception relates to resources.

Koji attempts a reland in https://chromium-review.googlesource.com/c/chromium/src/+/1343583 - I am not sure that's the best strategy. Koji, would you have a chance to reproduce the Android failure, with and without your CL locally? Let's hope the reland does not cause this failure again.

> would you have a chance to reproduce the Android failure, with and without your CL locally?

Me neither, but I found the android_cfi_rel_ng builder in the FYI bots and it passes with or without the CL.

Still Android CFI bot is flakily failing.
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20CFI?limit=200

Could this be a problem with build dependencies on Android? Some things I've dug up:

25400 (IDR_BROKENCANVAS) == the base id for third_party/blink/public/blink_image_resources.grd and has been since r604352 (Oct 31, 2018).

//third_party/blink/public:image_resources emits blink_image_resources_100_percent.pak
//third_party/blink/public:scaled_resources_100_percent repacks that into blink_scaled_resources_100_percent.pak
//content/shell:pak repacks that into content_shell.pak

Could it be that two of these pak files are being loaded during these failing tests? Would loading the same pak file twice cause this DCHECK to be hit?

Could it be that one of the pak files isn't being rebuilt, such that an old blink_resources.pak (which used to have 25400) is being used with a new blink_image_resources_100_percent.pak?

I don't think this is related to RendererView* tests. I think there's either something wrong with the devices, or else how we load resources on to the devices.

Example of this same issue occuring for Autofill components_browsertests:
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-marshmallow-arm64-rel/140467

+ jbudorick, martiniss, nednguyen

Actually, I take that back. Given that the same resources is failing to load in all these test suites, this is unlikely to be a device issue. Likely something with how we are packing/loading resources.

Also a lot of FormAutofillUtilsTest and others are also flakily failing due to the same check failure:

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-marshmallow-arm64-rel/141342

https://findit-for-me.appspot.com/flake/occurrences?key=ag9zfmZpbmRpdC1mb3ItbWVyTAsSBUZsYWtlIkFjaHJvbWl1bUBjb21wb25lbnRzX2Jyb3dzZXJ0ZXN0c0BGb3JtQXV0b2ZpbGxVdGlsc1Rlc3QuSXNSZWFkb25seQw

FindIt starts filing crbug issues just today, but this is at least since around Nov 23, so I'll merge the recent issues to this.

 Issue 910619  has been merged into this issue.

 Issue 910624  has been merged into this issue.

 Issue 910623  has been merged into this issue.

 Issue 910622  has been merged into this issue.

 Issue 910621  has been merged into this issue.

 Issue 910620  has been merged into this issue.

 Issue 910618  has been merged into this issue.

 Issue 910617  has been merged into this issue.

 Issue 910625  has been merged into this issue.

 Issue 910616  has been merged into this issue.

 Issue 910615  has been merged into this issue.

 Issue 910614  has been merged into this issue.

 Issue 910613  has been merged into this issue.

 Issue 910612  has been merged into this issue.

 Issue 910611  has been merged into this issue.

 Issue 910610  has been merged into this issue.

 Issue 910609  has been merged into this issue.

 Issue 910608  has been merged into this issue.

 Issue 910607  has been merged into this issue.

 Issue 910606  has been merged into this issue.

 Issue 910605  has been merged into this issue.

 Issue 910604  has been merged into this issue.

 Issue 910626  has been merged into this issue.

The components_browsertests_apk only has 2 .pak files:

"""
erikchen@erikchen:~/projects/chromium-android/src$ ls out/gn/components_browsertests_apk/assets/
components_tests_resources.pak  content_shell.pak  icudtl.dat  natives_blob.bin  snapshot_blob_64.bin
"""

Adding logging to the binary, we see that there are only 3 places that attempt to load pak files.
"""
  655   0000000003fd0beb  ui::ResourceBundle::AddDataPackFromFileRegion(base::File, base::MemoryMappedFile::Region const&, ui::ScaleFactor)
  656   0000000003f5bf6f  content::ShellMainDelegate::InitializeResourceBundle()
  657   0000000003f5bd4f  content::ShellMainDelegate::PreSandboxStartup()
  658   0000000003dc62cb  content::ContentMainRunnerImpl::Initialize(content::ContentMainParams const&)
  659   00000000051e4377  service_manager::Main(service_manager::MainParams const&)
  660   0000000003dc5983  Java_org_chromium_content_app_ContentMain_nativeStart
  661   0000000000046b3b  <UNKNOWN>
"""

"""
  668   00000000042087cf  ui::DataPack::CheckForDuplicateResources(std::__ndk1::vector<std::__ndk1::unique_ptr<ui::ResourceHandle, std::__ndk1::default_delete<ui::ResourceHandle> >, std::__ndk1::alloc      ator<std::__ndk1::unique_ptr<ui::ResourceHandle, std::__ndk1::default_delete<ui::ResourceHandle> > > > const&)
  669   0000000003fd0ca3  ui::ResourceBundle::AddDataPack(std::__ndk1::unique_ptr<ui::DataPack, std::__ndk1::default_delete<ui::DataPack> >)\
  670   0000000003fd0beb  ui::ResourceBundle::AddDataPackFromFileRegion(base::File, base::MemoryMappedFile::Region const&, ui::ScaleFactor)
  671   0000000003f5bf6f  content::ShellMainDelegate::InitializeResourceBundle()
  672   0000000003f5bd4f  content::ShellMainDelegate::PreSandboxStartup()
  673   0000000003f8a12b  content::ContentBrowserTest::SetUp()
  674   000000000205b82f  testing::Test::Run()
  675   000000000205bd77  testing::TestInfo::Run()
  676   000000000205c0f7  testing::TestCase::Run()
  677   0000000002060c17  testing::internal::UnitTestImpl::RunAllTests()
  678   00000000020609bb  testing::UnitTest::Run()
  679   0000000003f3464b  base::TestSuite::Run()
  680   0000000003f8a4b3  content::ContentTestLauncherDelegate::RunTestSuite(int, char**)
  681   0000000003f937df  content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**)
  682   0000000003f8a477  main
  683   000000000204ccbb  Java_org_chromium_native_1test_NativeTest_nativeRunTests
  684   00000000000613ab  <UNKNOWN>
"""

"""
  879   00000000042087cf  ui::DataPack::CheckForDuplicateResources(std::__ndk1::vector<std::__ndk1::unique_ptr<ui::ResourceHandle, std::__ndk1::default_delete<ui::ResourceHandle> >, std::__ndk1::alloc      ator<std::__ndk1::unique_ptr<ui::ResourceHandle, std::__ndk1::default_delete<ui::ResourceHandle> > > > const&)
  880   0000000003fd0ca3  ui::ResourceBundle::AddDataPack(std::__ndk1::unique_ptr<ui::DataPack, std::__ndk1::default_delete<ui::DataPack> >)
  881   0000000003fd0af3  ui::ResourceBundle::AddDataPackFromPathInternal(base::FilePath const&, ui::ScaleFactor, bool)
  882   0000000002027c9b  dom_distiller::DistillerPageWebContentsTest::AddComponentsResources()
  883   0000000002026f33  dom_distiller::DistillerPageWebContentsTest::SetUpOnMainThread()
  884   0000000003f8c13b  content::BrowserTestBase::ProxyRunTestOnMainThreadLoop()
  885   000000000202b71f  void base::internal::Invoker<base::internal::BindState<void (offline_pages::PageRenovatorBrowserTest::*)(), base::internal::UnretainedWrapper<offline_pages::PageRenovatorBrow      serTest_CorrectRenovationsRun_Test> >, void ()>::RunImpl<void (offline_pages::PageRenovatorBrowserTest::*)(), std::__ndk1::tuple<base::internal::UnretainedWrapper<offline_pages::PageRenovatorBro      wserTest_CorrectRenovationsRun_Test> >, 0ul>(void (offline_pages::PageRenovatorBrowserTest::*&&)(), std::__ndk1::tuple<base::internal::UnretainedWrapper<offline_pages::PageRenovatorBrowserTest_C      orrectRenovationsRun_Test> >&&, std::__ndk1::integer_sequence<unsigned long, 0ul>)
  886   0000000003f7cb6b  content::ShellBrowserMainParts::PreMainMessageLoopRun()
  887   000000000293fde3  content::BrowserMainLoop::PreMainMessageLoopRun()
....
2679   0000000003f3464b  base::TestSuite::Run()                                                         
 2680   0000000003f8a577  content::ContentTestLauncherDelegate::RunTestSuite(int, char**)
 2681   0000000003f938a3  content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**)
 2682   0000000003f8a53b  main
 2683   000000000204ccbb  Java_org_chromium_native_1test_NativeTest_nativeRunTests
 2684   00000000000613ab  <UNKNOWN>
"""

The first two look very similar. I suspect those are loading the same assets into two different processes? The latter one is loading dom_distiller assets from components_tests_resources.pak.

Suspiciously, when we look at the failure in components_browsertests:
https://chromium-swarm.appspot.com/task?id=4179247c38737710&refresh=10&show_raw=1#

The last test to pass is DomDistillerJsTest.RunJsTests. The test that runs after that fails with Check failed: !handle->HasResource(resource_id). Duplicate resource 25400 with scale 1. This makes me suspect that the dom_distiller resources [components_tests_resources.pak] are potentially duplicating resources in content_shell.pak.

As an aside, the resource bundle initialization code seems to regularly hit errors:
https://cs.chromium.org/chromium/src/content/shell/app/shell_main_delegate.cc?type=cs&q=shell_main_delegate.cc&sq=package:chromium&g=0&l=386

"""
 2835 11-30 13:32:40.742 10837 10837 E ApkAssets: Error while loading asset assets/content_shell.pak: java.io.FileNotFoundException: This file can not be opened as a file descriptor; it is probably compressed
 2836 11-30 13:32:40.742 10837 10837 E chromium: [ERROR:shell_main_delegate.cc(407)] ShellMainDelegate::InitializeResourceBundle3 /storage/emulated/0/chromium_tests_root/paks/content_shell.pak
"""

Specifically, this line seems to always fail for components_browsertests.
"""
    pak_fd =
        base::android::OpenApkAsset("assets/content_shell.pak", &pak_region);
"""

Not sure where to go next with this. nednguyen, jbudorick: thoughts?

I downloaded the apk from one of the failign swarming runs for content_browsertests:
python ~/projects/chromium/src/tools/swarming_client/isolateserver.py download -I isolateserver.appspot.com --namespace default-gzip -f 57a05deebb198d7caad383564783a0fadc90c201 57a05deebb198d7caad383564783a0fadc90c201

I confirmed that there is only 1 .pak file, and it only has resource w/ id = 25400:
"""
~/projects/chromium/src/tools/grit/pak_util.py print ./assets/content_shell.pak | grep 25400
Entry(id=25400, canonical_id=25400, size=289, sha1=a0869fe164): <data>
"""

+gbiv, pcc: who did the CFI work, can either of you help with triaging this bug?

Another theory. When I run one of the typically failing content_browsertests locally, I see the error message:
"""
[ERROR:resource_bundle_android.cc(52)] Failed to open pak file: assets/chrome_100_percent.pak
"""
followed by a successful attempt to open a .pak file. 

Looking at a successful run of content_browsertests on Android CFI, we see the same error message for the test RenderViewImplTest.BrowserNavigationStart.
https://chromium-swarm.appspot.com/task?id=417df3b8b91e9810&refresh=10&show_raw=1

However, when we look at a failing run of content_browsertests on Android CFI, we don't see that error message.
https://chromium-swarm.appspot.com/task?id=417e5e70eb664810&refresh=10

This suggests that the passing runs are the ones that are failing to open this assets/chrome_100_percent.pak file, and the failing ones are the ones that are successfully opening that pak file. That being said, I don't see this .pak file in the isolate input for the failing test. I wonder if this file is somehow being carried over from other APKs or previous versions of the APK?

And for reference, the two callstacks that try to open a pak file are:
"""
128   RELADDR   FUNCTION                                                                                                                                                                                                                       FILE:LINE
129   00000000050f4577  ui::(anonymous namespace)::LoadFromApkOrFile(char const*, base::FilePath const*, int*, base::MemoryMappedFile::Region*)                                                                                                        ??:0:0
130   00000000050f43c3  ui::ResourceBundle::LoadCommonResources()                                                                                                                                                                                      ??:0:0
131   00000000050f2b47  ui::ResourceBundle::InitSharedInstanceWithLocale(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, ui::ResourceBundle::Delegate*, ui    ::ResourceBundle::LoadResources)  ??:0:0
132   00000000044668f3  content::TestContentClient::TestContentClient()                                                                                                                                                                                ??:0:0
133   000000000445463f  content::RenderViewTest::CreateContentClient()                                                                                                                                                                                 ??:0:0
134   0000000004453dc7  content::RenderViewTest::SetUp()                                                                                                                                                                                               ??:0:0
135   0000000002c7ebe7  testing::Test::Run()                                                                                                                                                                                                           ??:0:0
136   0000000002c7f247  testing::TestInfo::Run()                                                                                                                                                                                                       ??:0:0
137   0000000002c7f59f  testing::TestCase::Run()                                                                                                                                                                                                       ??:0:0
138   0000000002c84023  testing::internal::UnitTestImpl::RunAllTests()                                                                                                                                                                                 ??:0:0
139   0000000002c83dc7  testing::UnitTest::Run()                                                                                                                                                                                                       ??:0:0
140   000000000446e6af  base::TestSuite::Run()                                                                                                                                                                                                         ??:0:0
141   000000000443dc33  content::ContentTestLauncherDelegate::RunTestSuite(int, char**)                                                                                                                                                                ??:0:0
142   000000000445ad7b  content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**)                                                                                                                                               ??:0:0
143   000000000443dbf7  main                 
"""

"""
276   00000000050f8467  ui::DataPack::CheckForDuplicateResources(std::__ndk1::vector<std::__ndk1::unique_ptr<ui::ResourceHandle, std::__ndk1::default_delete<ui::ResourceHandle> >, std::__ndk1::allocator<std::_    _ndk1::unique_ptr<ui::ResourceHandle, std::__ndk1::default_delete<ui::ResourceHandle> > > > const&)  ??:0:0
277   00000000050f2f37  ui::ResourceBundle::AddDataPack(std::__ndk1::unique_ptr<ui::DataPack, std::__ndk1::default_delete<ui::DataPack> >)                                                                                                                                                                                ??:0:0
278   00000000050f2e7f  ui::ResourceBundle::AddDataPackFromFileRegion(base::File, base::MemoryMappedFile::Region const&, ui::ScaleFactor)                                                                                                                                                                                 ??:0:0
279   00000000050f4cb7  ui::LoadMainAndroidPackFile(char const*, base::FilePath const&)                                                                                                                                                                                                                                   ??:0:0
280   0000000004466913  content::TestContentClient::TestContentClient()                                                                                                                                                                                                                                                   ??:0:0
281   000000000445463f  content::RenderViewTest::CreateContentClient()                                                                                                                                                                                                                                                    ??:0:0
282   0000000004453dc7  content::RenderViewTest::SetUp()                                                                                                                                                                                                                                                                  ??:0:0
283   0000000002c7ebe7  testing::Test::Run()                                                                                                                                                                                                                                                                              ??:0:0                                                                                                  
284   0000000002c7f247  testing::TestInfo::Run()                                                                                                                                                                                                                                                                          ??:0:0
285   0000000002c7f59f  testing::TestCase::Run()                                                                                                                                                                                                                                                                          ??:0:0
286   0000000002c84023  testing::internal::UnitTestImpl::RunAllTests()                                                                                               DEBUG:root:Finished resolving symbols. Elaps    ed time: 0.7577s
287                                                                                                                                                  ??:0:0
288   0000000002c83dc7  testing::UnitTest::Run()                                                                                                                                                                                                                                                                          ??:0:0
289   000000000446e6af  base::TestSuite::Run()                                                                                                                                                                                                                                                                            ??:0:0
290   000000000443dc33  content::ContentTestLauncherDelegate::RunTestSuite(int, char**)                                                                                                                                                                                                                                   ??:0:0
291   000000000445ad7b  content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**)                                                                                                                                                                                                                  ??:0:0
292   000000000443dbf7  main                                                                                                                                                                                                                                                                                              ??:0:0
293   000000000269ba53  Java_org_chromium_native_1test_NativeTest_nativeRunTests                                                                                                                                                                                                                                          ??:0:0
294   00000000000613ab  <UNKNOWN>                                                                                 
"""

The former is the one that typically fails to load. The latter is the one that always successfully loads.

jbudorick@ mentioned that we don't use incremental deploys yet.

I wonder if this is related to monochrome/trichrome. We probably attempts to share resources across the various APK/deployment targets. 

+ torne, could you look at c#41?

Issue 910563 has been merged into this issue.

Cancelling the previous operation, sorry for the noise

I spoke with torne@. He said that there's no resource sharing on Android, and that APKs are never unzipped in the directory structure, so there should be no way to mix resources between APKs.

That being said, the code to load the usually non-existent chrome_100_percent.pak file also attempts to directly load the file from disk:
https://cs.chromium.org/chromium/src/ui/base/resource/resource_bundle_android.cc?type=cs&q=ResourceBundle::LoadCommonResources&g=0&l=86

And early in the log files, we see that there's a differential updater that pushes resources to a common [test suite independent] location on disk:
"""
I   72.134s TimeoutThread-1-for-individual_device_set_up(01e176d5a2b5c289)  Pushing 3293 files via .zip of size 81451886
...
unzip /data/local/tmp/temp_file-e737d8381b452.zip&&chmod -R 777 /sdcard/chromium_tests_root/paks/content_shell.pak /sdcard/chromium_tests_root/content/test/data 
...
"""

Code location:
https://cs.chromium.org/chromium/src/third_party/catapult/devil/devil/android/device_utils.py?type=cs&q=%22via+.zip+of+size%22&sq=package:chromium&g=0&l=1768

The name of the function that emits this log statement: "_PushChangedFilesZipped" suggests that this is a differential update. And the location of the file: "/sdcard/chromium_tests_root/content/test/data" suggests that different test suites will mix resources, assuming the directory is not nuked between tests.

On my local device, content_browsertests attempts to load the resource from:

"""
/storage/emulated/0/chromium_tests_root/paks/chrome_100_percent.pak
"""

Digging around in that directly with the adb shell, we see that there are other pak files as well from other test suites:

"""
walleye:/storage/emulated/0/chromium_tests_root # find . -name *pak
./paks/components_tests_resources.pak
./paks/content_shell.pak
"""

I can repro this issue locally:

1) Run "out/gn/bin/run_unit_tests"
No need to wait for tests to finish, the APK & resources just need to be loaded to the device.
2) Run "./out/gn/bin/run_content_browsertests -f RenderViewImplTest.BrowserNavigationStart"

I've leased a swarming bot and confirmed that the following command does not delete unused pak files in "/sdcard/chromium_tests_root/paks"
"""
/b/swarming/w/ir/build/android/test_runner.py gtest --suite content_browsertests --output-directory /b/swarming/w/ir/out/Release --runtime-deps-path /b/swarming/w/ir/out/Release/gen.runtime/content/test/content_browsertests__test_runner_script.runtime_deps -v --store-tombstones --gs-results-bucket=chromium-result-details --recover-devices --test-launcher-summary-output=/b/swarming/w/ioOuHdxG/output.json
"""

There are two problems that led to this issue occurring:

1) content_browsertests and components_browsertests rely on the resource file chrome_100_percent.pak NOT being loaded in order to function correctly. Filed  Issue 911764  to track this.

2) We apply differential updates to resource updates, using a common directory .../chromium_tests_root/ which allows resources to leak between tests. I don't know when this directory is supposed to be cleared, but it does not appear to be cleared in several cases that I checked in c#52.

#54.2: this is intentional, for speed. Pushing things down to the device & doing filesystem manipulations is sloooow.

John, is the tool intended to *remove* files that the current test doesn't need, and it's just not working, or is it intending to leave them behind and it just happens that they aren't there for some other reason in the passing cases? :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/75d116ca56e6212e4ec6afc752957cd8916977ab

commit 75d116ca56e6212e4ec6afc752957cd8916977ab
Author: erikchen <erikchen@chromium.org>
Date: Wed Dec 05 18:14:02 2018

TestContentClient should not try to load common resources.

All resources used by content shell are in content_shell.pak. Attempting to load
common resources will either fail to find the .pak file, or succeed and hit a
DCHECK as there are duplicated resources between content_shell.pak and common
resources.

I've confirmed that content_browsertests fails without this CL, passes with this
CL with the following repro steps:
https://bugs.chromium.org/p/chromium/issues/detail?id=906587#c52

Ditto for compontents_browsertests.

Change-Id: Ica7234898c6dc2c220340e942f827b331dfd14b0
Bug:  911764 ,  906587 
Reviewed-on: https://chromium-review.googlesource.com/c/1363441
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Avi Drissman <avi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614026}
[modify] https://crrev.com/75d116ca56e6212e4ec6afc752957cd8916977ab/content/test/test_content_client.cc

#56: the latter.

Oh. So we mustn't have any tests that depend on the absence of a file, then :)

CL to fix all other test suites as well in review:
https://chromium-review.googlesource.com/c/chromium/src/+/1363447

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a706d62ed54b601a3527b61a7f89b17564c262e0

commit a706d62ed54b601a3527b61a7f89b17564c262e0
Author: erikchen <erikchen@chromium.org>
Date: Thu Dec 06 13:26:55 2018

Require that .pak files are successfully loaded.

Several test suites relied on the common resources .pak being unsuccessfully
loaded in order to function correctly. On Android, .pak files can leak between
successive runs of different test suites, which would cause flaky test failures.
This CL makes it so that failing to load the common resources .pak file will hit
a DCHECK. This CL fixes several instances where test suites were incorrectly
trying to load [and failing to load] the common resources .pak file.

Change-Id: I5294f921e42524b699c7019b7a56c0a30106c89c
Bug:  906587 
Reviewed-on: https://chromium-review.googlesource.com/c/1363447
Reviewed-by: Avi Drissman <avi@chromium.org>
Reviewed-by: Marc Treib <treib@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614331}
[modify] https://crrev.com/a706d62ed54b601a3527b61a7f89b17564c262e0/components/ntp_tiles/icon_cacher_impl_unittest.cc
[modify] https://crrev.com/a706d62ed54b601a3527b61a7f89b17564c262e0/ui/base/resource/resource_bundle_android.cc
[modify] https://crrev.com/a706d62ed54b601a3527b61a7f89b17564c262e0/ui/base/test/run_all_unittests.cc