If I understand correctly, you need a builder on the chrome pfq that produces chrome prebuilts with use flags that are specific to rialto?

Infra can assist but it will be self service to create that config and get it up on the waterfall. You should rope in the gardeners because adding builders to the pfq is going to affect them.

 Issue 594571  has been merged into this issue.

Issue 597866 has been merged into this issue.

https://code.google.com/p/chrome-os-partner/issues/detail?id=50644 is now in the commit queue to switch rialto from app_shell to chrome, if that sticks OK then it will make this bug obsolete.

Unfortunately even without the app_shell USE flag, the rialto paladin is still failing to find a chrome binary. 

https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-paladin/builds/481/steps/BuildPackages/logs/stdio

15:14:28: ERROR: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_rialto

I expect it's because the USE flags are still a bit different to a standard config:

[ebuild  N    ] chromeos-base/chromeos-chrome-51.0.2701.0_rc-r1 to /build/veyron_rialto/ USE="accessibility autotest build_tests buildcheck chrome_debug chrome_internal chrome_remoting evdev_gestures gold hardfp highdpi nacl neon opengles ozone runhooks v4l2_codec v4lplugin -X -afdo_use -app_shell -asan -chrome_debug_tests -chrome_media -clang -component_build -deep_memory_profiler -envoy -fonts -internal_gles_conform -internal_khronos_glcts -mojo -opengl -vaapi -verbose -vtable_verify -xkbcommon" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -cast -egltest {-test}" 


... "-xkbcommon" and "-fonts" for example are set in the rialto make.conf so probably not used in other boards (at least, in that exact combination)

Copying in some tips from related email thread:

It looks like a PFQ builder is defined here (in config_dump.json), but not enabled:

 "veyron_rialto-chrome-pfq": {
        "_template": "chrome-pfq",
        "boards": [
            "veyron_rialto"
        ],
        "important": false,
        "manifest": "official.xml",
        "vm_tests": []
    },

It doesn't actually run, because there isn't a waterfall attached to it. There is some logic somewhere in chromeos_config.py that does this, but I'm not sure where.

....
There's a section in chromeos_config.py for listing experimental configs for various waterfalls.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0d449ab8e91f06d42bd041e77d6f8de347081b6d

commit 0d449ab8e91f06d42bd041e77d6f8de347081b6d
Author: Jonathan Dixon <joth@google.com>
Date: Mon Apr 11 23:16:16 2016

chromite: Add waterfall for veyron_rialto-chrome-pfq

BUG= chromium:590784 
TEST=TBC - veyron_rialto-chrome-pfq builder should appear in https://uberchromegw.corp.google.com/i/chromeos/waterfall or Chrome PFQ page of the same?

Change-Id: I68f7a030c7e0b7ecb3ec8ddefffca833e742e18c
Reviewed-on: https://chromium-review.googlesource.com/338201
Commit-Ready: Jonathan Dixon <joth@chromium.org>
Tested-by: Jonathan Dixon <joth@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/0d449ab8e91f06d42bd041e77d6f8de347081b6d/cbuildbot/config_dump.json
[modify] https://crrev.com/0d449ab8e91f06d42bd041e77d6f8de347081b6d/cbuildbot/waterfall_layout_dump.txt
[modify] https://crrev.com/0d449ab8e91f06d42bd041e77d6f8de347081b6d/cbuildbot/chromeos_config.py

akeshet, I don't yet see veyron_rialto-chrome-pfq  on the waterfall. What's the next step to get it there? Does it need a restart?

This is one of the reasons for the failure here, I think:

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/10785

 Issue 610087  has been merged into this issue.

From 610087, davidjames says:

veyron_rialto has sync_chrome=False. This was set because veyron_rialto didn't use Chrome at all when the flag was set. It looks like veyron_rialto does use Chrome now, so suggest just removing the sync_chrome flag (and ask the rialto folks whether their inclusion of Chrome is accidental or not).

Prior to the release builder ungrouping, the sync_chrome=False setting for veyron_rialto had no effect because it was pulling Chrome from the group prebuilts, so it didn't need to sync Chrome to pull prebuilts. So that's why you only are noticing this now.

So.... scanning through everything I'm a little confused.

Was a Chrome installation intentionally added to veyron_rialto? Or was that an accident?

Chrome installation was intentional, see http://go/rialto-appshell-deprecation
and https://code.google.com/p/chrome-os-partner/issues/detail?id=50644

But even before that change, it was using app_shell which in essence was a chrome build with a particular USE flag set (app_shell) so it's always needed Chrome source code of some flavor for 12+ months.

I'll make the change for the release builder, but leave the PFQ changes to Aviv.

So hold on, if veyron_rialto no longer needs app_shell, do we even need a PFQ for veyron_rialto anymore? Can't it just reuse one of the existing chrome binaries?

Is it using non-standard USE flags for Chrome?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/d9938c088f7604081bbd0ae91e20f3e0dffc5aa9

commit d9938c088f7604081bbd0ae91e20f3e0dffc5aa9
Author: Don Garrett <dgarrett@google.com>
Date: Tue May 10 00:18:07 2016

chromeos_config: Build Chrome on veyron_rialto-release

Apparently, we DO need to build Chrome for veyron_rialto.

BUG= chromium:590784 
TEST=run_tests

Change-Id: I51b9656cae4e9d5409a0a9bf0e8cd401062a0e63
Reviewed-on: https://chromium-review.googlesource.com/343623
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/d9938c088f7604081bbd0ae91e20f3e0dffc5aa9/cbuildbot/config_dump.json
[modify] https://crrev.com/d9938c088f7604081bbd0ae91e20f3e0dffc5aa9/cbuildbot/chromeos_config.py

I'm expecting the release builder to start working. This doesn't help with the PFQ, CQ or anything else though.

Thanks for the update Don.

So.... what will help the PFQ? I still see this error
https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-paladin/builds/860/steps/BuildPackages/logs/stdio

07:25:26: ERROR: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_rialto


What do we need to do so it can find the prebuilts?

Nope. It fixes the release builder, but has no effect on any other build type.

Actually, the release build is still failing, but it's for different reasons later in the build.

What is the action on me here?

The pfq builder already exists, and is seemingly perma-failing in SimpleChromeWorkflow stage. Also, it is still marked as important=False.

https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-chrome-pfq

The SimpleChromeWorkflow error is this:  issue 611784 

The fix for it has landed so the next build should pick it up.

Ok. Before this pfq builder can help the cq, it needs to be marked as important=True. That's a config change to chromeos_config. Our team treats those as self-service. You can put me on review. See go/chromeos_config for a doc.

Actually, the veyron_rialto failure is not (or not just)  issue 611784 , it is:

FAILED: obj/remoting/host/linux/remoting_host.unicode_to_keysym.o 
armv7a-cros-linux-gnueabi-g++ -B/tmp/cbuildbot-tmpGqLxge/chrome-sdk-cacheptwiW8/cache/chrome-sdk/tarballs/veyron_rialto+__b__cbuild__internal_master__buildbot_archive__veyron_rialto-chrome-pfq__R52-8332.0.0-rc1+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -MMD -MF obj/remoting/host/linux/remoting_host.unicode_to_keysym.o.d -DBINARY_CORE=1 -DBINARY_DESKTOP=2 -DBINARY_HOST_ME2ME=3 -DBINARY_NATIVE_MESSAGING_HOST=4 -DBINARY_REMOTE_ASSISTANCE_HOST=5 -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DCHROMIUM_BUILD -DCR_CLANG_REVISION=268813-1 -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_ASH=1 -DUSE_PANGO=1 -DUSE_CAIRO=1 -DUSE_CRAS=1 -DUSE_OZONE=1 -DUSE_DEFAULT_RENDER_THEME=1 -DUSE_LIBJPEG_TURBO=1 -DIMAGE_LOADER_EXTENSION=1 -DENABLE_WEBRTC=1 -DENABLE_MEDIA_ROUTER=1 -DENABLE_PEPPER_CDMS -DENABLE_NOTIFICATIONS -DENABLE_TOPCHROME_MD=1 -DENABLE_WAYLAND_SERVER=1 -DUSE_UDEV -DFIELDTRIAL_TESTING_ENABLED -DENABLE_TASK_MANAGER=1 -DENABLE_EXTENSIONS=1 -DENABLE_PDF=1 -DENABLE_PLUGINS=1 -DENABLE_SESSION_SERVICE=1 -DENABLE_THEMES=1 -DENABLE_PRINTING=1 -DENABLE_PRINT_PREVIEW=1 -DENABLE_SPELLCHECK=1 -DENABLE_CAPTIVE_PORTAL_DETECTION=1 -DENABLE_APP_LIST=1 -DENABLE_SUPERVISED_USERS=1 -DENABLE_MDNS=1 -DENABLE_SERVICE_DISCOVERY=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -D__ARMEL__ -D__arm__ -DNVALGRIND '-DHOST_BUNDLE_NAME="ChromotingHost.bundle"' '-DPREFPANE_BUNDLE_NAME="Chromoting.prefPane"' -DVERSION=52.0.2738.0 -DMOJO_USE_SYSTEM_IMPL -DTOOLKIT_VIEWS=1 -DPROTOBUF_USE_DLLS -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DFEATURE_ENABLE_SSL -DFEATURE_ENABLE_VOICEMAIL -DEXPAT_RELATIVE_PATH -DGTEST_RELATIVE_PATH -DNO_MAIN_THREAD_WRAPPING -DNO_SOUND_SYSTEM -DLINUX -DWEBRTC_LINUX -DWEBRTC_POSIX -DCHROMEOS -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_NOEXCEPT= -DU_STATIC_IMPLEMENTATION -DSK_SUPPORT_GPU=1 -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DWEBRTC_CHROMIUM_BUILD -DUSE_LIBPCI=1 -DUSE_NSS_CERTS=1 -DOS_CHROMEOS=1 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DNDEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -I../../third_party/skia/include/utils -I../../third_party/khronos -I../../gpu -I../../skia/config -Igen/angle -I../../third_party/WebKit/Source -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/webrtc_overrides -I../../third_party/libjingle/overrides -I../../third_party/libjingle/source -I../../testing/gtest/include -I../../third_party -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -Igen/policy -I../../third_party/skia/include/core -I../../third_party/skia/include/effects -I../../third_party/skia/include/pdf -I../../third_party/skia/include/gpu -I../../third_party/skia/include/lazy -I../../third_party/skia/include/pathops -I../../third_party/skia/include/pipe -I../../third_party/skia/include/ports -I../../skia/ext -Igen/ui/views/resources -Werror -pthread -fno-strict-aliasing -Wall -Wno-extra -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -I/tmp/cbuildbot-tmpGqLxge/chrome-sdk-cacheptwiW8/cache/chrome-sdk/tarballs/veyron_rialto+__b__cbuild__internal_master__buildbot_archive__veyron_rialto-chrome-pfq__R52-8332.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/nss -I/tmp/cbuildbot-tmpGqLxge/chrome-sdk-cacheptwiW8/cache/chrome-sdk/tarballs/veyron_rialto+__b__cbuild__internal_master__buildbot_archive__veyron_rialto-chrome-pfq__R52-8332.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/include/nspr -Wno-maybe-uninitialized -march=armv7-a -mtune=generic-armv7-a -mfpu=neon -mfloat-abi=hard -mthumb --sysroot=/tmp/cbuildbot-tmpGqLxge/chrome-sdk-cacheptwiW8/cache/chrome-sdk/tarballs/veyron_rialto+__b__cbuild__internal_master__buildbot_archive__veyron_rialto-chrome-pfq__R52-8332.0.0-rc1+sysroot_chromeos-base_chromeos-chrome.tar.xz -O2 -fno-ident -fdata-sections -ffunction-sections -funwind-tables -gsplit-dwarf -pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -D__google_stl_debug_vector=1 -Wno-unknown-warning-option -gsplit-dwarf -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-abi -std=gnu++11 -Wno-narrowing -Wno-literal-suffix  -c ../../remoting/host/linux/unicode_to_keysym.cc -o obj/remoting/host/linux/remoting_host.unicode_to_keysym.o
../../remoting/host/linux/unicode_to_keysym.cc:27:27: fatal error: X11/keysymdef.h: No such file or directory
 #include <X11/keysymdef.h>
                           ^
compilation terminated.

Someone needs to fix this before we can mark it important.

@stevenjb where did you get the error message in #28 from?
There's some links in https://bugs.chromium.org/p/chromium/issues/detail?id=608832 but they're not working for me.

https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-chrome-pfq/builds/112/steps/SimpleChromeWorkflow/logs/stdio

As of build 119 veyron_rialto-chrome-pfq is now completing successfully:
https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-chrome-pfq/builds/119

AIUI the next step is to mark the pfq build as important=True, I'll send a CL for this now.

Still happening (5/24/16) with build 940

08:41:23: ERROR: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_rialto
08:41:23: INFO: Running cidb query on pid 14491, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x542e210>
08:41:28: INFO: cidb query succeeded after 1 retries
08:41:28: INFO: Running cidb query on pid 14491, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x5afc890>

Thanks, yes it's still waiting on the important=True step:  https://chromium-review.googlesource.com/346214

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/65cf8cbcc59f7118a913d2dda2fac4cf75699e05

commit 65cf8cbcc59f7118a913d2dda2fac4cf75699e05
Author: Jonathan Dixon <joth@google.com>
Date: Fri May 20 20:31:42 2016

Mark veyron_rialto-chrome-pfq as important

BUG= chromium:590784 
TEST=veyron_rialto-chrome-pfq is now important (per config_dump.json)

Change-Id: I5d0e47942454b235ea28a2c90b9b6c69d7be910e
Reviewed-on: https://chromium-review.googlesource.com/346214
Commit-Ready: Jonathan Dixon <joth@chromium.org>
Tested-by: Jonathan Dixon <joth@chromium.org>
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/65cf8cbcc59f7118a913d2dda2fac4cf75699e05/cbuildbot/config_dump.json
[modify] https://crrev.com/65cf8cbcc59f7118a913d2dda2fac4cf75699e05/cbuildbot/chromeos_config.py

Still happening 5/25/16 as of build #949:

02:22:28: ERROR: Cannot find prebuilts for chromeos-base/chromeos-chrome on veyron_rialto

That build should have included CL:346214, which landed in 8369.0.0.

Is there another step needed here?

Build 949 failed :(

But build 950+ are green :-)

https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-paladin/builds/950
https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_rialto-paladin/


AIUI the next step is monitor the build for a day or two, and if all looks good mark it as important. 

...actually this bug is just for the chrome builder, which I think is all done now. remaining work tracked in 583465