CC thakis@ and erikchen@, do you happen to know what happened?

Build links would be convenient in the future. Note that the error is quite explicit - icudtl.dat not found in bundle. This means that either icudtl.dat not found in bundle isn't being built, or isn't being brought into the bundle correctly, [or less likely, is being deleted by something]

link to bot?

https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release

https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release?numbuilds=200 is green?

I can't exactly find the link. (I'd like to learn where the link is). But here's some more info:

Here's the bucket: chrome-test-builds_media_mac-release (aarya@ mentioned that it was from media bucket. I'm not sure if this is the exact bucket name.)
Here's the revision: asan-mac-release-451230

I hope it helps. Thank you.

The problem is all these lkgr bots don't run basic tests, otherwise, we would catch this on the build bots itself rather than on clusterfuzz. Dirk, you were planning to enable tests on these and detach from lkgr, any progress on that.

Anyway, we have to find who changed the build rules on mac to cause this. causing startup crashes all over :(

How can you crash on startup if your build is failing?

This bug is super confusing. What's the actual bug here? Something's broken at runtime, not build time?

This was misfiled, fixing subject

Build is happening fine, but build is crashing at runtime instantly with this failure.

rsesek, have there been changes to the gn app bundling recently that could cause icudtl.dat to not be found in the Chromium bundle?

actually +rsesek (forgot to + him back in after colliding with comment 10. rsesek, have there been changes to the gn app bundling recently that could cause icudtl.dat to not be found in the Chromium bundle?

Yes, yesterday we started versioning the framework ( issue 662466 ).

Robert, can you please see how to fix this icudtl.dat bundling error.

The icudtl.dat file is there: asan-mac-release-451282/Chromium.app/Contents/Versions/58.0.3016.0/Chromium Framework.framework/Versions/A/Resources/icudtl.dat

Previously it used to exist at Chromium.app\Contents\Versions\58.0.3015.0\Chromium Framework.framework\Resources\, so maybe it cannot find the file in this new path. see https://www.googleapis.com/download/storage/v1/b/chromium-browser-asan/o/mac-release%2Fasan-mac-release-451144.zip?generation=1487303026726223&alt=media

and then 500 mb size regression as well on https://www.googleapis.com/download/storage/v1/b/chromium-browser-asan/o/mac-release%2Fasan-mac-release-451201.zip?generation=1487315839615891&alt=media

Chromium.app/Contents/Versions/58.0.3015.0/Chromium Framework.framework/Resources/ is now a symlink to Chromium.app/Contents/Versions/58.0.3015.0/Chromium Framework.framework/Versions/Current/Resources, which itself is a symlink to Chromium.app/Contents/Versions/58.0.3015.0/Chromium Framework.framework/Versions/A/Resources.

Maybe the zip isn't preserving symlinks properly?

Not a build issue.

% cat out/asan-rel/args.gn 
enable_nacl = false
is_asan = true
is_component_build = false
is_debug = false
use_goma = true
% ninja -C out/asan-rel -j250 chrome
ninja: Entering directory `out/asan-rel'
[26766/26766] STAMP obj/chrome/chrome.stamp
 % ./out/asan-rel/Chromium.app/Contents/MacOS/Chromium 
2017-02-17 14:10:16.476 Chromium[72200:3933806] NSWindow warning: adding an unknown subview: <FullSizeContentView: 0x60e0001018a0>. Break on NSLog to debug.
2017-02-17 14:10:16.477 Chromium[72200:3933806] Call stack:
(
    "+callStackSymbols disabled for performance reasons"
)
% echo $?
0

Is the icu file included in `gn desc out/asan-rel //chrome:chrome runtime_deps`? (Not sure if that bot uses swarming)

There is no resources symlink in that directory, so looks like a https://cs.chromium.org/chromium/build/scripts/slave/zip_build.py issue. Pawel - thoughts or can help with owner ?

(ENV) aarya@aarya-linux2:~/Downloads/asan-mac-release-451282/Chromium.app/Contents/Versions/58.0.3016.0/Chromium Framework.framework$ ls -l
total 585740
-rwxr-xr-x 1 aarya eng 599785900 Feb 17 06:21 Chromium Framework
drwxr-xr-x 3 aarya eng      4096 Feb 17 06:24 Versions

Re #19: No it's not a runtime_deps because we consider it build-time dependency by virtue of being bundled.

I just verified that the buildbot is producing the right thing, so it does seem like a packaging issue:

chrome-bot@build111-b1:(Mac 10.12.2):/b/c/b/Mac_ASAN_Release/src/out/Release$ ls -l Chromium.app/Contents/Versions/58.0.3016.0/Chromium\ Framework.framework/
total 24
lrwxr-xr-x  1 chrome-bot  staff   35 Feb 17 11:00 Chromium Framework -> Versions/Current/Chromium Framework
lrwxr-xr-x  1 chrome-bot  staff   24 Feb 17 11:00 Helpers -> Versions/Current/Helpers
lrwxr-xr-x  1 chrome-bot  staff   26 Feb 17 11:00 Resources -> Versions/Current/Resources
drwxr-xr-x  4 chrome-bot  staff  136 Feb 17 10:21 Versions

chrome-bot@build111-b1:(Mac 10.12.2):/b/c/b/Mac_ASAN_Release/src/out/Release$ ls -l Chromium.app/Contents/Versions/58.0.3016.0/Chromium\ Framework.framework/Resources/icudtl.dat 
-rw-r--r--  6 chrome-bot  staff  10130464 Feb 17 10:20 Chromium.app/Contents/Versions/58.0.3016.0/Chromium Framework.framework/Resources/icudtl.dat

We do pass -y here: https://cs.chromium.org/chromium/build/scripts/common/chromium_utils.py?type=cs&q=MakeZip&l=729

And yet those symlinks don't show up in the output of zip: https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release/builds/4651/steps/zipping/logs/stdio. I wonder if this has to do with this step called "filter_build_dir" that runs before? https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release/builds/4651/steps/filter%20build_dir/logs/stdio

"filter_build_dir" has been changed 5 weeks ago (https://chromium.googlesource.com/chromium/tools/build/+log/30afdb56aa7eb469c44ae42541c7cdb91876b8ac/scripts/slave/recipe_modules/archive/resources/filter_build_files.py), how long do we experience this issue?

"zipping" stage (which happens right after the "filter_build_dir") does put that file into an archive: https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release/builds/4662/steps/zipping/logs/stdio and ctrl+F "icudtl.dat":
<...>
  adding: asan-mac-release-451456/Chromium Framework.framework/Versions/A/Resources/icudtl.dat (deflated 54%)
<...>
  adding: asan-mac-release-451456/Chromium.app/Contents/Versions/58.0.3017.0/Chromium Framework.framework/Versions/A/Resources/icudtl.dat (deflated 54%)
<...>
  adding: asan-mac-release-451456/Content Shell Framework.framework/Resources/icudtl.dat (deflated 54%)
<...>
  adding: asan-mac-release-451456/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources/icudtl.dat (deflated 54%)
<...>
  adding: asan-mac-release-451456/icudtl.dat (deflated 54%)
<...>


I've just downloaded (https://storage.cloud.google.com/chromium-browser-asan/mac-release/asan-mac-release-451456.zip) and I see:

mmoroz@mmoroz1:~/Downloads/asan-mac-release-451456$ find . -name icudtl.dat
./Chromium Framework.framework/Versions/A/Resources/icudtl.dat
./Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources/icudtl.dat
./icudtl.dat
./Chromium.app/Contents/Versions/58.0.3017.0/Chromium Framework.framework/Versions/A/Resources/icudtl.dat
./Content Shell Framework.framework/Resources/icudtl.dat


Looks fine for me. 

Can someone with Mac just add a few debugging prints or set a breakpoint at https://cs.chromium.org/chromium/src/base/mac/foundation_util.mm?type=cs&q=PathForFrameworkBundleResource&l=94 and inside function called from it to see where does it actually look for that file? I cannot find any recent commits that could change that logic, but something has definitely changed...

Max, please read comment #13-#17, that is what changed. Is your filter files code handling symlinks properly ?

Hm, maybe it's an issue of os.walk (https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/archive/resources/filter_build_files.py?l=126).

Though I don't see any difference locally:

>>> for root, dirnames, filenames in os.walk(dir_path):
...     for f in filenames:
...             if f == 'icudtl.dat':
...                     print root, f
... 
/home/mmoroz/Downloads/asan-mac-release-451456 icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Chromium Framework.framework/Versions/A/Resources icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Chromium.app/Contents/Versions/58.0.3017.0/Chromium Framework.framework/Versions/A/Resources icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Content Shell Framework.framework/Resources icudtl.dat
>>> 
>>> 
>>> for root, dirnames, filenames in os.walk(dir_path, followlinks=True):
...     for f in filenames:
...             if f == 'icudtl.dat':
...                     print root, f
... 
/home/mmoroz/Downloads/asan-mac-release-451456 icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Chromium Framework.framework/Versions/A/Resources icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Chromium.app/Contents/Versions/58.0.3017.0/Chromium Framework.framework/Versions/A/Resources icudtl.dat
/home/mmoroz/Downloads/asan-mac-release-451456/Content Shell Framework.framework/Resources icudtl.dat


Changing that argument in filter_build_files.py also doesn't change anything, I'm still getting 5 'icudtl.dat' paths.

Comparing an old and a new archives:

####### OLD ########
asan-mac-release-451144$ find . -name icudtl.dat
./Chromium Framework.framework/Resources/icudtl.dat
./Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources/icudtl.dat
./icudtl.dat
./Chromium.app/Contents/Versions/58.0.3015.0/Chromium Framework.framework/Resources/icudtl.dat
./Content Shell Framework.framework/Resources/icudtl.dat

####### NEW ########
asan-mac-release-451456$ find . -name icudtl.dat
./Chromium Framework.framework/Versions/A/Resources/icudtl.dat
./Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources/icudtl.dat
./icudtl.dat
./Chromium.app/Contents/Versions/58.0.3017.0/Chromium Framework.framework/Versions/A/Resources/icudtl.dat
./Content Shell Framework.framework/Resources/icudtl.dat


The comment at https://cs.chromium.org/chromium/src/base/i18n/icu_util.cc?type=cs&q=icu_util.cc&l=126 says:

"// Assume it is in the framework bundle's Resources directory."

Looks like now this assumption may be wrong, as it is not in Bundle's Resources directly, it's inside "Versions/A/" subdirectories of Bundle.

Maybe os.walk(top, topdown=True, onerror=None, followlinks=False)

try setting followlinks=True.

'followlinks=True' is what I tried in c#29, nothing changed, but I've tried it in the wrong directory (i.e. in already filtered build).

Current build archive differs from the structure described in c#17, as we are missing Chromium Framework.framework/Resources/ and Chromium Framework.framework/Versions/Current/.

That's a known python issue: http://stackoverflow.com/questions/14483147/is-os-walk-missing-symlinks-to-directories

Something like https://chromium-review.googlesource.com/444229 should help.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/93d377c213b7775ee7993ff67a841121fe716d9e

commit 93d377c213b7775ee7993ff67a841121fe716d9e
Author: Max Moroz <mmoroz@chromium.org>
Date: Mon Feb 20 16:11:54 2017

Fix filter_build_files.py to archive symlinks pointing to directories.

BUG= 693624 

Change-Id: Ic2e5ad31e876ff6405a7251233129297f442c651
Reviewed-on: https://chromium-review.googlesource.com/444229
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>

[modify] https://crrev.com/93d377c213b7775ee7993ff67a841121fe716d9e/scripts/slave/recipe_modules/archive/resources/filter_build_files.py

The most recent build (https://storage.googleapis.com/chromium-browser-asan/mac-release/asan-mac-release-451616.zip) from https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release/builds/4681 looks better:

asan-mac-release-451616$ find . -name icudtl.dat
./Chromium.app/Contents/Versions/58.0.3019.0/Chromium Framework.framework/Resources/icudtl.dat
./Chromium.app/Contents/Versions/58.0.3019.0/Chromium Framework.framework/Versions/Current/Resources/icudtl.dat
./Chromium.app/Contents/Versions/58.0.3019.0/Chromium Framework.framework/Versions/A/Resources/icudtl.dat
./Content Shell Framework.framework/Resources/icudtl.dat
./Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources/icudtl.dat
./icudtl.dat
./Chromium Framework.framework/Resources/icudtl.dat
./Chromium Framework.framework/Versions/Current/Resources/icudtl.dat
./Chromium Framework.framework/Versions/A/Resources/icudtl.dat


But the archive became even more bigger :(

For some reason 'Chromium Framework.framework/Resources/' and './Chromium Framework.framework/Versions/Current/' are not symlinks. Investigating.

Another issue comes from here: https://cs.chromium.org/chromium/build/scripts/common/chromium_utils.py?q=MakeZip&l=671

...
          shutil.copytree(src_path, os.path.join(archive_dir, needed_file),
                          symlinks=True)
...

is `src_path` is a symlink and not an actual directory, `shutil.copytree` will follow the link and copy all tree to the destination path. Instead of that we should do something like:

if os.path.islink(src_path):
  os.symlink(os.readlink(src_path), %destination%)

to create a proper symlink instead of another copy of directory tree.

I'll upload another CL for that.

Looks like scottmg@ only fixed it for windows :) - https://chromium.googlesource.com/chromium/tools/build/+/30286421b926f4517212b38ac24eab5fc27188ff.

Thanks Max for fixing this.

https://chromium-review.googlesource.com/c/445216/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/b85a15616fa964bed66a4a7d0bbc53f420682e12

commit b85a15616fa964bed66a4a7d0bbc53f420682e12
Author: Max Moroz <mmoroz@chromium.org>
Date: Tue Feb 21 10:52:37 2017

Fix chromium_utils.MakeZip to copy symlinks pointing to directories.

BUG= 693624 ,654550

Change-Id: I71c07d4331885fd040cafc8c1933bfa72e0ef84b
Reviewed-on: https://chromium-review.googlesource.com/445216
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>

[modify] https://crrev.com/b85a15616fa964bed66a4a7d0bbc53f420682e12/scripts/common/chromium_utils.py
[modify] https://crrev.com/b85a15616fa964bed66a4a7d0bbc53f420682e12/scripts/slave/recipe_modules/archive/resources/filter_build_files.py

Got an exception: https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release/builds/4692/steps/zipping/logs/stdio

Traceback (most recent call last):
  File "/b/rr/tmppZBXo5/rw/checkout/scripts/slave/recipe_modules/archive/resources/zip_archive.py", line 34, in <module>
    sys.exit(main(sys.argv))
  File "/b/rr/tmppZBXo5/rw/checkout/scripts/slave/recipe_modules/archive/resources/zip_archive.py", line 24, in main
    raise_error=True)
  File "/b/rr/tmppZBXo5/rw/checkout/scripts/common/chromium_utils.py", line 675, in MakeZip
    os.symlink(os.readlink(src_path), dst_path)
OSError: [Errno 2] No such file or directory
step returned non-zero exit code: 1


The only possible reason why it could happen is |src_path| pointing to a file/dir that doesn't exist. That sounds strange, as |os.path.islink| can't be True for non-existing file.

|os.symlink| doesn't care what has been passed as the first argument, so it shouldn't throw such an error.

I believe it makes sense to wait for another build...

Ok, I got it. If |dst_path| is something like "A/B/file" and either A or B directory doesn't exist, that exception happens. I'm working on fix.

https://chromium-review.googlesource.com/c/445239/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/2ce6852980b606d89021802beb1595081a41da69

commit 2ce6852980b606d89021802beb1595081a41da69
Author: Max Moroz <mmoroz@chromium.org>
Date: Tue Feb 21 13:53:49 2017

MakeZip: always create intermediate directories + small refactoring.

BUG= 693624 

Change-Id: I448bd5b7640847f01ea0edbe85e15d50cd5b8550
Reviewed-on: https://chromium-review.googlesource.com/445239
Commit-Queue: Sergiy Byelozyorov <sergiyb@chromium.org>
Reviewed-by: Sergiy Byelozyorov <sergiyb@chromium.org>

[modify] https://crrev.com/2ce6852980b606d89021802beb1595081a41da69/scripts/common/chromium_utils.py
[modify] https://crrev.com/2ce6852980b606d89021802beb1595081a41da69/scripts/slave/recipe_modules/archive/resources/filter_build_files.py

Build is successful: https://build.chromium.org/p/chromium.lkgr/builders/Mac%20ASAN%20Release/builds/4694

Archive size is returned back to what we had on Feb 17-20th. Archive structure looks good, symlinks to directories are there:

asan-mac-release-451713$ find . -name icudtl.dat
./Chromium.app/Contents/Versions/58.0.3020.0/Chromium Framework.framework/Versions/A/Resources/icudtl.dat
./Content Shell Framework.framework/Resources/icudtl.dat
./Content Shell.app/Contents/Frameworks/Content Shell Framework.framework/Resources/icudtl.dat
./icudtl.dat
./Chromium Framework.framework/Versions/A/Resources/icudtl.dat

asan-mac-release-451713$ cd Chromium\ Framework.framework/
asan-mac-release-451713/Chromium Framework.framework$ ls -la
total 584440
drwxr-x---  3 mmoroz eng      4096 Feb 21 15:39 .
drwxr-x--- 48 mmoroz eng      4096 Feb 21 15:39 ..
-rwxr-x---  1 mmoroz eng 598447720 Feb 21 15:06 Chromium Framework
lrwxrwxrwx  1 mmoroz eng        24 Feb 21 15:39 Helpers -> Versions/Current/Helpers
lrwxrwxrwx  1 mmoroz eng        26 Feb 21 15:39 Resources -> Versions/Current/Resources
drwxr-x---  3 mmoroz eng      4096 Feb 21 15:39 Versions
mmoroz@mmoroz0:~/Downloads/asan-mac-release-451713/Chromium Framework.framework$ ls -la Resources/icudtl.dat 
-rw-r----- 1 mmoroz eng 10130464 Feb 21 14:55 Resources/icudtl.dat

Thank you Max! Our dashboard reports that the build is not failing anymore (https://viceroy.corp.google.com/clusterfuzz/#_VG_-JOM1Va0).

Thanks for fixing, Max.

Just out of curiosity: have you considered using `gn desc out/dir target runtime_deps` or `gn gen out/dir --runtime-deps-list-file` to compute the list of required runtime files for a binary? This is how isolate/swarming works and may be less error prone.

https://cs.chromium.org/chromium/src/tools/mb/mb.py?q=isolate.py&sq=package:chromium&dr=C&l=813

That's an interesting trick, Robert! I haven't tried it. I assume that we can try this way, but I guess it requires to specify all dependencies properly in GN files? If there any other way how some file (which is required) gets into build directory, runtime_deps wouldn't catch it?

> but I guess it requires to specify all dependencies properly in GN files?

Yes, but it's a bug if that's already not the case today. Any test suite on the waterfall would already fail to run today if this were not the case, due to swarming. So all runtime dependencies should be specified as data_deps if they are not.

> If there any other way how some file (which is required) gets into build directory, runtime_deps wouldn't catch it?

No, runtime_deps have to be explicitly added. You can catch things by only archiving things that are explicitly listed as runtime_deps, and things should fail otherwise.

That sounds super cool, we should use it. Thanks Robert for the suggestion! Filed  issue 695365 .