just confirmed running into the same issue when building on a fresh checkout of the public-only overlays

We added a workaround to some font cache problems that runs fc-cache as part of the package.

https://bugs.gentoo.org/666418 indicates that upstream gentoo saw a similar bug and reverted part of its upstream fontconfig sources to deal with it. We probably need to do the same.

Tim has been hacking on this lately.  hopefully he should be able to roll it.

I assume the break is caused by my recent CL to upgrade font-config to 2.13.0 with crrev.com/c/1313968.
I'm going to work it on Monday 11/26 after the Thanksgiving holiday. In the meantime, I have several questions.
* How do I launch a tryjob to reproduce it? So far I've only run tryjobs from Gerrit for my own CLs.
* How do I run the build on my workstation for public overlay? Do I just change the "internal" USE flag?
* pgeorgi@, what workarounds have been made to fong cache problems? You mentioned this in commnet 2. Can you give some links to the previous work?
* Just for my curiosity, how is the public overlay builds used? Who's using them? And what board is kukui?

#4:

* How do I launch a tryjob to reproduce it? So far I've only run tryjobs from Gerrit for my own CLs.

cros tryjob kukui-full-tryjob, that just makes a kukui-full (public build) on ToT.

* How do I run the build on my workstation for public overlay? Do I just change the "internal" USE flag?

We usually use a completely separate Chromium OS checkout (just follow public Chromium OS instructions).

People in the team here who _just_ checked out the tree could reproduce the issue (I wasn't, at first). For me, I first had to recreate my chroot:
 - cros_sdk -r
 - cros_sdk
 - ./setup_board --board=kukui 
 - ./build_packages --board=kukui --accept_licenses=@CHROMEOS
 => error

* pgeorgi@, what workarounds have been made to fong cache problems? You mentioned this in commnet 2. Can you give some links to the previous work?

https://bugs.gentoo.org/666418#c13
https://bugs.gentoo.org/666418#c22

Sounds like we should "just" re-sync font-config to upstream Gentoo?

* Just for my curiosity, how is the public overlay builds used? Who's using them?

Any member of public can use it (Chromium OS is an open-source project!). We also have partners using public overlays (e.g. contractors, Wifi chip vendors, etc...).

* And what board is kukui?

It's a reference design, feel free to ping me for more details.

#4,

 My steps (see https://chromium.googlesource.com/chromiumos/docs/+/master/developer_guide.md#Building-Chromium-OS). are:
1. mkdir a chromiumOS folder
2. 
(outside)
cd <path_to_chromiumOS_folder>
repo init -u https://chromium.googlesource.com/chromiumos/manifest.git --repo-url https://chromium.googlesource.com/external/repo.git;
repo sync -j4;

3. 
export BOARD=kukui; 
./setup_board --board=${BOARD}; 
./build_packages --board=${BOARD} --accept_licenses=@CHROMEOS;

Observed error while running "./build_packages --board=${BOARD} --accept_licenses=@CHROMEOS;"

Hope above help!!

This CL fixes it for me: https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1348493 .

Tryjob: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929111642686688224 => chromeos-bmpblk builds fine.

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1340839 does look related, I haven't tested that one locally.

Tryjob: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929104503010872496 (just started)

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1340839 also fixes this problem. But an upstream patch that we can drop after we uprev font-config to 2.13.1 sounds preferable.

Reassign to drinkcat@.

I upstreamed my fix to fc-cache not respecting --sysroot here: https://gitlab.freedesktop.org/fontconfig/fontconfig/merge_requests/16
2.13.1 was tagged two months ago so it won't include my change, but the fix would be in 2.13.2 whenever that gets tagged.

nice work!

Since the patch in #8/#10 also fixes the issue, I don't think there is much point to merge my CL that backports an upstream patch

Maybe we should just uprev fc-cache once 2.13.2 is released, so that we get both patches?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/f3a6472bc88c70f50156d81d30d949712562aa6c

commit f3a6472bc88c70f50156d81d30d949712562aa6c
Author: Nicolas Boichat <drinkcat@chromium.org>
Date: Sat Dec 01 09:08:45 2018

media-libs/fontconfig: Add upstream patch to fix uuid generation

This upstream patch takes into account sysroot, and fixes the build
issue seen in  crbug.com/907781 .

BUG= chromium:907781 
BUG=chromium:908588
TEST=cros tryjob kukui-full-tryjob

Change-Id: I5b4081c4ffe48c4300753e9e0d975b481454d733
Reviewed-on: https://chromium-review.googlesource.com/1348493
Commit-Ready: Nicolas Boichat <drinkcat@chromium.org>
Tested-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[rename] https://crrev.com/f3a6472bc88c70f50156d81d30d949712562aa6c/media-libs/fontconfig/fontconfig-2.13.0-r7.ebuild
[add] https://crrev.com/f3a6472bc88c70f50156d81d30d949712562aa6c/media-libs/fontconfig/files/fontconfig-2.13.0-fc-cache-doesn-t-use-y-option.patch
[delete] https://crrev.com/e481567febdd6226b111ab842dd9fdb2d60c0678/media-libs/fontconfig/fontconfig-2.13.0-r6.ebuild

Manoj said the patch helps with chromium:908588, so we merged the fix anyway.

This patch doesn't fix the problem. Still getting similar failures on soraka and sarien, at least on Intel side. 

please verify what was recommended in the thread:
$ ./update_chroot
$ sudo fc-cache --really-force

This is happening even when you clone a fresh new tree.
I'm going to try sudo fc-cache --really-force right after I enter chroot for the first time and see if that helps. 

BTW QC is reporting the same thing https://bugs.chromium.org/p/chromium/issues/detail?id=911696

 Issue 911696  has been merged into this issue.

Running "sudo fc-cache --really-force" allowed build_packages to succeed after the initial failure. I'm trying to see if running the command first time after creating the chroot will prevent the build failure from reoccurring.