We do the 2nd way (have update.py build it) for Android; probably good to do the same for fuchsia.

as for 

> some additional configuration to find the fuchsia sdk, and of course also require the person doing to acquire and have the right sdk in place so that the sdk/sysroot/.a's all match

update.py relies on downloading tarballs for the correct version of gcc, cmake, etc. to build the toolchain. It could download the SDK for this in the same way.

OK, also didn't want that to be too big of a hassle for update.py'rs. I'll take a look at doing it that way.

Looks like ~2M to be included in the archive fwiw.

[(e4cc35c...)] sgraham@river:/work/cr/src$ ls -l third_party/fuchsia-sdk/toolchain_libs/clang/5.0.0/lib/fuchsia/
total 1764
-rw-r----- 1 sgraham sgraham 1014122 Apr 28 12:50 libclang_rt.builtins-aarch64.a
-rw-r----- 1 sgraham sgraham  786598 Apr 28 12:50 libclang_rt.builtins-x86_64.a

> update.py relies on downloading tarballs for the correct version of gcc, cmake, etc. to build the toolchain. It could download the SDK for this in the same way.

Hmm, OK. The sysroot has to match I assume, so I guess there's a bit of a chicken and egg problem there. We'd have to change to update.py to point at a new fuchsia sdk, then run update.py, then make sure to update chrome's DEPS to also point at that new SDK when that clang was rolled. Seems a bit fiddly?

(Any suggestions on working on update.py more quickly? I set CLANG_REVISION to something fixed, but every time I run tools/clang/scripts/update.py --force-local-build it rebuilds ~everything I guess because the cmake cache gets nuked.)

Comment out anything that looks destructive: calls to RmCmakeCache(), calls to RmTree(), ...

> Hmm, OK. The sysroot has to match I assume, so I guess there's a bit of a chicken and egg problem there. We'd have to change to update.py to point at a new fuchsia sdk, then run update.py, then make sure to update chrome's DEPS to also point at that new SDK when that clang was rolled. Seems a bit fiddly?

Sorry, I'm probably misunderstanding what you're trying to do. I'd imagine that you could just assume that a fuchsia sdk exists if some --with-fuchsia switch is passed in (like we do on android; and the bots would pass that), and then you'd use that to build the runtime libs. And then you just put the runtime libs int he clang package -- it sounds like you want to also put them back in the SDK and put a new SDK at the same time as new clang binaries? (That'd be not so good for us.)

I'm only trying to do what you said. :) I was just thinking it'll be a bit involved to update the Fuchsia SDK that way, as first clang will have to be rebuilt with the new SDK, and then the clang roll + update to Chrome DEPS have to happen together.

Why are clang rolls and sdk rolls linked to each other? That's bad, it must be possible to do both independent of each other. Are there any ABI dependencies between compiler and sdk? If so, can we break them?

Oh, I don't know, I just assumed they would be, maybe they're not. Wouldn't it be possible that the SDK changed a header in the sysroot so then the implementation of the builtins might need to change at the same time?

(Quite possibly I'm lost here! It's likely a pretty uncommon situation anyway I suppose.)

Usually platform providers keep their ABI stable so they don't break all existing binaries on the platforms. I'd expect that fuchsia would do that too -- but the Fuchsia folks can probably more authoritatively speak to this :-)

OK, ignore me! Now to actually get these dratted .a's to build...

CMake has convinced me I've made a dire mistake in career choice, and perhaps should try being a barista. Or perhaps something in the arts.

I tried followed Fuchsia's toolchain build instructions https://fuchsia.googlesource.com/docs/+/HEAD/toolchain.md but it seems that we put compiler-rt in a different place, so the relative paths in runtimes/ don't work.

I also tried a great number of variations like https://codereview.chromium.org/2893973003 but in most cases configuring doesn't work as expected, I think because lld is trying to link against the thing I'm trying to eventually build.

...
/work/cr/src/third_party/llvm-build/Release+Asserts/bin/lld: error: cannot open /work/cr/src/third_party/llvm-build/Release+Asserts/lib/clang/5.0.0/lib/fuchsia/libclang_rt.builtins-x86_64.a: No such file or directory

If I put that file in place, then it does sort of seem to work (obviously not useful) to build the .a, but even then it's still building to lib/linux (though it's got the right --target and --sysroot).

So I'm clearly very lost here. Halp?

I can give this a try tomorrow. What exactly is it what you want here to happen (bundling those two ma files I assume), and why?

Yup, I just want lib/clang/5.0.0/lib/fuchsia/libclang_rt.builtins-x86_64.a to get built and included in the clang package. The "why" is because when targeting Fuchsia, the linker seems to want to link against it, e.g.

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FFuchsia%2F1320%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

FAILED: libtest_shared_library.so libtest_shared_library.so.TOC 
python "/b/c/builder/linux/src/build/toolchain/gcc_solink_wrapper.py" --readelf="readelf" --nm="nm" --sofile="./libtest_shared_library.so" --tocfile="./libtest_shared_library.so.TOC" --output="./libtest_shared_library.so"  -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -Wl,--no-as-needed -lpthread -Wl,--as-needed -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin --target=x86_64-fuchsia -m64 -Werror -Wl,-O1 -Wl,--gc-sections -Wl,--gdb-index --sysroot=../../third_party/fuchsia-sdk/sysroot/x86_64-fuchsia  -o "./libtest_shared_library.so" -Wl,-soname="libtest_shared_library.so" @"./libtest_shared_library.so.rsp"
../../third_party/binutils/Linux_x64/Release/bin/ld.gold: error: cannot open /b/c/builder/linux/src/third_party/llvm-build/Release+Asserts/lib/clang/5.0.0/lib/fuchsia/libclang_rt.builtins-x86_64.a: No such file or directory
clang: error: ld.lld command failed with exit code 1 (use -v to see invocation)

(There's no aarch64 sysroot in the SDK yet, so only the x86_64 one for the time being.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7d38657a69469a0c14213e9be65bfe1d682abef2

commit 7d38657a69469a0c14213e9be65bfe1d682abef2
Author: Nico Weber <thakis@chromium.org>
Date: Fri May 19 19:55:32 2017

fuchsia: Let linker know where to find libclang_rt.builtin.a

BUG= 724204 
R=scottmg@chromium.org

Review-Url: https://codereview.chromium.org/2895583004 .
Cr-Commit-Position: refs/heads/master@{#473292}

[modify] https://crrev.com/7d38657a69469a0c14213e9be65bfe1d682abef2/build/config/fuchsia/BUILD.gn

Filling back from email a bit:

- As of today, fuchsia has one patch on top of upstream compiler_rt (namely, https://fuchsia.googlesource.com/third_party/compiler-rt/+/6c9f8c6ba77090158373490ac010437603c8ff50). they're actively working on fixing that. until then, putting the fuchsia libclang_rt.builtin with our clang package isn't really an option.

- https://fuchsia.googlesource.com/docs/+/HEAD/toolchain.md#Developing-Clang describes how to build compiler_rt once it's upstreamed. (an incomplete, untested attempt at doing that is here: https://codereview.chromium.org/2897463003/)

- in the meantime, we're passing -resource-dir to clang when linking to let it know where to look for libclang_rt.builtin in the fuchsia sdk (see comment 16)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c688af81fcfa4c072dcca79e698641839e8a42f

commit 1c688af81fcfa4c072dcca79e698641839e8a42f
Author: Nico Weber <thakis@chromium.org>
Date: Mon Jul 24 11:25:28 2017

Roll clang 307486:308728.

Ran `tools/clang/scripts/upload_revision.py 308728`.

This changes the clang version from 5.0.0 to 6.0.0, so simplify
some things that dealt with both numbers.

This changes clang to no longer look for libc++ headers in the
Fuchsia SDK.  To keep things building, use "our" libc++, like
on linux (use_custom_libcxx).  This in turn means we pass
-nodefaultlibs to the linker, which sadly also disables the
automatic linking of clang_rt.builtins, so do that manually now.
(In exchange, we no longer need to pass in -resource-dir.)

This makes -Wformat fire in Google Toolbox for Mac in iOS builds,
so disable -Wformat in that config for now.

Bug:  746303 , 746505 , 747638 , 747643 , 724204 

Change-Id: I6196a2a173a1b4871f22d0ce92436d0197fd7845
Reviewed-on: https://chromium-review.googlesource.com/582074
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#488946}
[modify] https://crrev.com/1c688af81fcfa4c072dcca79e698641839e8a42f/build/config/c++/BUILD.gn
[modify] https://crrev.com/1c688af81fcfa4c072dcca79e698641839e8a42f/build/config/c++/c++.gni
[modify] https://crrev.com/1c688af81fcfa4c072dcca79e698641839e8a42f/build/config/fuchsia/BUILD.gn
[modify] https://crrev.com/1c688af81fcfa4c072dcca79e698641839e8a42f/build/toolchain/toolchain.gni
[modify] https://crrev.com/1c688af81fcfa4c072dcca79e698641839e8a42f/third_party/google_toolbox_for_mac/BUILD.gn
[modify] https://crrev.com/1c688af81fcfa4c072dcca79e698641839e8a42f/tools/clang/scripts/update.py

I'm told the Fuchsia libbuiltin bits are now fully upstreamed, I should continue https://codereview.chromium.org/2897463003/ eventually.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff69d3018c42eb994533de337f81d417a0209913

commit ff69d3018c42eb994533de337f81d417a0209913
Author: Nico Weber <thakis@chromium.org>
Date: Wed Aug 09 18:12:04 2017

Use clang's spiffy new -nostdlib++.

Reverts some Fuchsia bits of https://chromium-review.googlesource.com/c/582074

Also only default use_custom_libcxx to true on Linux if is_clang
(which on our bots is always the case), so that we can assume that
this flag is available.  (If gcc ever adds a -nostdlib++ flag, we
can undo this part.)

Bug:  750434 , 724204 
Change-Id: Ib548b403703f6dc77e5ec09c7ee14e360fb09b72
Reviewed-on: https://chromium-review.googlesource.com/606790
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#493055}
[modify] https://crrev.com/ff69d3018c42eb994533de337f81d417a0209913/build/config/c++/c++.gni
[modify] https://crrev.com/ff69d3018c42eb994533de337f81d417a0209913/build/config/fuchsia/BUILD.gn
[modify] https://crrev.com/ff69d3018c42eb994533de337f81d417a0209913/build/config/posix/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/695966bcd6b2698201c517c0e5af623f7028e388

commit 695966bcd6b2698201c517c0e5af623f7028e388
Author: Nico Weber <thakis@chromium.org>
Date: Wed Aug 09 19:07:59 2017

linux: Lower libc expectations after libstdlib++ change.

https://chromium-review.googlesource.com/c/606790 made it so we no longer
explicitly pass -lc.  Apparently manually adding -lc picks up the C library
from a slighly different place somehow (?)

Bug:  750434 , 724204 
Change-Id: I6a60a07c909842a88dcc4e670976b4b848866216
Reviewed-on: https://chromium-review.googlesource.com/609182
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Thomas Anderson <thomasanderson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#493083}
[modify] https://crrev.com/695966bcd6b2698201c517c0e5af623f7028e388/chrome/installer/linux/debian/expected_deps_x64
[modify] https://crrev.com/695966bcd6b2698201c517c0e5af623f7028e388/chrome/installer/linux/rpm/expected_deps_x86_64

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3d0476a897f85ba83f7fa280f625160518c64487

commit 3d0476a897f85ba83f7fa280f625160518c64487
Author: Scott Graham <scottmg@chromium.org>
Date: Fri Jan 26 18:08:16 2018

Roll Fuchsia SDK to 893404234442cf13abbc4e22ed996839ff5b1338

Updates clang toolchain_libs path to 7.0.0.

Includes https://fuchsia-review.googlesource.com/c/zircon/+/115121
which should fix component builds crashes.

Also renames calls from zx_time_get() to zx_clock_get() per recent
Fuchsia syscall change.

TBR=kmackay@chromium.org

Bug:  724204 , 707030,  803110 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: If2cb1d320c27b60fb32b0337ce1c6d2886b45b33
Reviewed-on: https://chromium-review.googlesource.com/884822
Reviewed-by: Kenneth MacKay <kmackay@chromium.org>
Reviewed-by: Wez <wez@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532014}
[modify] https://crrev.com/3d0476a897f85ba83f7fa280f625160518c64487/DEPS
[modify] https://crrev.com/3d0476a897f85ba83f7fa280f625160518c64487/base/time/time_fuchsia.cc
[modify] https://crrev.com/3d0476a897f85ba83f7fa280f625160518c64487/build/config/fuchsia/BUILD.gn
[modify] https://crrev.com/3d0476a897f85ba83f7fa280f625160518c64487/chromecast/media/cma/backend/audio_decoder_for_mixer.cc
[modify] https://crrev.com/3d0476a897f85ba83f7fa280f625160518c64487/chromecast/media/cma/backend/fuchsia/mixer_output_stream_fuchsia.cc
[modify] https://crrev.com/3d0476a897f85ba83f7fa280f625160518c64487/media/audio/fuchsia/audio_output_stream_fuchsia.cc

Any chance we could move Chromium to download the tool chain lib outside of the Fuchsia SDK? 

FYI, the toolchain libs that are currently distributed as part of the Fuchsia SDK  are actually downloaded from CIPD:
  https://fuchsia.googlesource.com/buildtools/+/master/fuchsia.ensure#33

The current makesdk.go script simply copies that content into the tarball:
  https://fuchsia.googlesource.com/scripts/+/master/makesdk.go#128

One way to prevent that unnecessary copy/paste would be for the SDK to provide some sort of manifest (or other type of files) that indicates which version of the buildtools should be used if needed. Using that same file, Chromium can run a hook to download the necessary tools/libraries into //third_party instead of getting them directly from the Fuchsia SDK.

What do you think?

... reading back through all this ...

This seems somewhat related to moving from tarball to cipd, but I think the correct fix for this specific bug is to pick up Nico's change from above https://bugs.chromium.org/p/chromium/issues/detail?id=724204#c20 and get that working.

Then Chrome can just use the upstream version with our Clang build. That way we don't need to pull them from elsewhere (Fuchsia SDK or otherwise).

(Out of interest, what's wrong with the current approach? Seems to have worked fine so far.)

It's awkward for us (Fuchsia) to distribute a small part of the clang toolchain build's output, but not a full toolchain and not a sysroot that's completely independent of the toolchain.  Since Chrome is already building and distributing the rest of the clang toolchain, it seems reasonable to also build these libraries as part of that process.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/9ecc581da8004dec4d019a69c4e13d24a8baf41e

commit 9ecc581da8004dec4d019a69c4e13d24a8baf41e
Author: Scott Graham <scottmg@chromium.org>
Date: Mon Sep 10 19:39:49 2018

linux_upload_clang: Include fuchsia in .gclient to pull Fuchsia SDK

This is necessary so that the Fuchsia SDK will be downloaded so that the
Clang toolchain will be able to use the sysroot to build the builtins.a
in tools/clang/scripts/update.py.

Bug:  724204 
Change-Id: Ia40d37e1bd0dbf2a626a569fc9b39b8e05832d72
Reviewed-on: https://chromium-review.googlesource.com/1216789
Commit-Queue: Scott Graham <scottmg@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/9ecc581da8004dec4d019a69c4e13d24a8baf41e/scripts/slave/recipes/chromium_upload_clang.py
[modify] https://crrev.com/9ecc581da8004dec4d019a69c4e13d24a8baf41e/scripts/slave/recipes/chromium_upload_clang.expected/full_tryserver_chromium_linux_linux_upload_clang.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c3dc46d8c259a9d7127d999c0432152f936a9ff4

commit c3dc46d8c259a9d7127d999c0432152f936a9ff4
Author: Scott Graham <scottmg@chromium.org>
Date: Tue Sep 11 15:51:52 2018

Quietly exit if there's no Fuchsia SDK for the platform

More suitable for DEPS runhooks running.

Bug:  724204 
Change-Id: I105a57269b4c4f7b4b05bab80df7a4ef112bdf1f
Reviewed-on: https://chromium-review.googlesource.com/1217850
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590332}
[modify] https://crrev.com/c3dc46d8c259a9d7127d999c0432152f936a9ff4/build/fuchsia/update_sdk.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/a9113e8e59271b3d95d1354389dec5bab3725fbb

commit a9113e8e59271b3d95d1354389dec5bab3725fbb
Author: Scott Graham <scottmg@chromium.org>
Date: Tue Sep 11 16:00:59 2018

Run build/fuchsia/update_sdk.py to pull Fuchsia SDK

Required so that the Fuchsia sysroot is available so that
libclang_rt.builtins.a can be built and included.

Bug:  724204 
Change-Id: I0da974a45d6cc8d3ee8ab1b7b8d8083e80fc5273
Reviewed-on: https://chromium-review.googlesource.com/1217464
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>

[modify] https://crrev.com/a9113e8e59271b3d95d1354389dec5bab3725fbb/scripts/slave/recipes/chromium_upload_clang.expected/full_tryserver_chromium_win_win_upload_clang.json
[modify] https://crrev.com/a9113e8e59271b3d95d1354389dec5bab3725fbb/scripts/slave/recipes/chromium_upload_clang.py
[modify] https://crrev.com/a9113e8e59271b3d95d1354389dec5bab3725fbb/scripts/slave/recipes/chromium_upload_clang.expected/full_tryserver_chromium_linux_linux_upload_clang.json
[modify] https://crrev.com/a9113e8e59271b3d95d1354389dec5bab3725fbb/scripts/slave/recipes/chromium_upload_clang.expected/full_tryserver_chromium_mac_mac_upload_clang.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9fc3a5c27d64ef9eedb31d33ad09da9e6a98f6d7

commit 9fc3a5c27d64ef9eedb31d33ad09da9e6a98f6d7
Author: Scott Graham <scottmg@chromium.org>
Date: Tue Sep 11 23:08:09 2018

Include libclang_rt.builtins.a for Fuchsia

Based on https://codereview.chromium.org/2897463003/.

Test: Run update.py --force-local-build, remove https://cs.chromium.org/chromium/src/build/config/fuchsia/BUILD.gn?l=41 and referenced files, and rebuild some Fuchsia binaries
Test: Run package.py and verify the .a's are included.
Bug:  724204 
Change-Id: If1492138829846a0f970d034f28a84d517554401
Reviewed-on: https://chromium-review.googlesource.com/1211879
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Scott Graham <scottmg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590521}
[modify] https://crrev.com/9fc3a5c27d64ef9eedb31d33ad09da9e6a98f6d7/build/config/fuchsia/BUILD.gn
[modify] https://crrev.com/9fc3a5c27d64ef9eedb31d33ad09da9e6a98f6d7/tools/clang/scripts/package.py
[modify] https://crrev.com/9fc3a5c27d64ef9eedb31d33ad09da9e6a98f6d7/tools/clang/scripts/update.py

_In theory_, the SDK can remove those now.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/cc72db6e1dee608f4ecd59dd5c13d08c524de834

commit cc72db6e1dee608f4ecd59dd5c13d08c524de834
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Thu Sep 13 19:33:40 2018

publish_tarball: Pass --without-fuchsia to clang's update.py.

https://chromium-review.googlesource.com/1211879 added this new option to
update.py, and it defaults to false just like --without-android.

This caused the publish_tarball bot to fail, as we do not check out the
Fuchsia SDK before calling update.py:
https://ci.chromium.org/buildbot/chromium.infra.cron/publish_tarball/12898

Add a version check and pass --without-fuchsia if we're packaging a version
that includes it.

Bug:  724204 
Change-Id: I6818e2efef0d5501d73775ca786fd42548a8f605
Reviewed-on: https://chromium-review.googlesource.com/1224250
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Raphael Kubo da Costa (CET) <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#17564}
[add] https://crrev.com/cc72db6e1dee608f4ecd59dd5c13d08c524de834/recipes/recipes/publish_tarball.expected/clang-no-fuchsia.json
[modify] https://crrev.com/cc72db6e1dee608f4ecd59dd5c13d08c524de834/recipes/recipes/publish_tarball.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/368b97765344091296daa06bac5d3148285cd378

commit 368b97765344091296daa06bac5d3148285cd378
Author: Raphael Kubo da Costa <raphael.kubo.da.costa@intel.com>
Date: Mon Sep 17 15:43:24 2018

build_from_tarball: Pass --without-fuchsia to clang's update.py.

This recipe needs the same fix as publish_tarball, which was updated in
https://chromium-review.googlesource.com/1224250.

https://chromium-review.googlesource.com/1211879 added --without-fuchsia new
option to update.py, and it defaults to false just like --without-android.

This caused the "Build From Tarball" bot to start failing in the "Build
clang" step:
https://ci.chromium.org/buildbot/chromium.infra.cron/Build%20From%20Tarball/490

Add a version check and pass --without-fuchsia if we're building a version
that includes it.

Bug:  724204 
Change-Id: I5052083675cd59003aaa3227565a9e38c0d08268
Reviewed-on: https://chromium-review.googlesource.com/1227994
Reviewed-by: Michael Moss <mmoss@chromium.org>
Commit-Queue: Raphael Kubo da Costa (CET) <raphael.kubo.da.costa@intel.com>
Cr-Commit-Position: refs/heads/master@{#17635}
[add] https://crrev.com/368b97765344091296daa06bac5d3148285cd378/recipes/recipes/build_from_tarball.expected/clang-no-fuchsia.json
[modify] https://crrev.com/368b97765344091296daa06bac5d3148285cd378/recipes/recipes/build_from_tarball.py