Help! This is a chrome builder and I am still pretty unfamiliar with the python in that repo.

This is the builder:
https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/amd64-generic%20Trusty%20%28Informational%29

I can not figure out where it is configured, but looking at a sample log file:
https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/amd64-generic%20Trusty%20%28Informational%29/builds/42168/steps/steps/logs/stdio

I do not see any evidence that run_gn (in api.py) is getting called.

Frustratingly, I can not find any reference to chrome-tot-chromeos-amd64-generic-trusty in the Chrome, cros, or build repos.

Yup, you are correct that it doesn't look like this builder is running GN. 

This is the first I've heard of this builder (or waterfall) doing Chrome builds. I have no idea who is supposed to be maintaining this, or what it's being used for, but someone needs to update the recipe and builder configs to be running MB and building with some set of GN args, but I do not know what args should be being used?

+davidjames@, +dnj@

I believe the builder exists to test Simple Chrome (it is part of the chromeos.chrome internal waterfall: https://uberchromegw.corp.google.com/i/chromeos.chrome/waterfall)

Unfortunately that is not clear from either the descriptive name (amd64-generic Trusty (Informational)) or the actual builder name (chrome-tot-chromeos-amd64-generic-trusty), neither of which appear to be searchable.

While we are at it, I would love for us to rename it (and make the builder and descriptive names match as we have done elsewhere), to something like chrome-tot-simple-chrome-amd64-generic-informational). Might be nice to add a searchable comment somewhere as well.

+akeshet@, +sosa@

Can somebody please take a moment to look into this? I can not find any reference to chrome-tot-chromeos in chromite, I have no idea where / how this builder is getting configured.

So, I found the configuration in chrome_info_cfg.py in the infra_internal/build_internal repo (which doesn't seem to be convered by any code search), "B('amd64-generic Trusty (Informational)'". 

However, I can not figure out how that maps to a recipe. It does not appear to be using SimpleChromeWorkflowStage in chrome_stages.py (which always runs 'gn gen' in _BuildChrome) but I don't se anywhere else where we run 'cros chrome-sdk'.

This builder is maintained entirely by the Chrome infra team (dnj@ is the original author but there are many others on the team who can help you). Suggest pinging someone on chrome-infrastructure-team@ and they can fill in for dnj@ while he's out of the office. 

I'm actually not the original author of this suite of builders, but I have worked on them a while back. They *should not* be maintained by the Chrome Infra team; someone who cares about them should represent them and own the configuration and underlying build scripts. I don't know who cares about them. If nobody does, we should delete them. These are, indeed, not "cbuildbot"-based builders, so it doesn't have anything to do with CrOS infra.

This particular builder is configured to run the "chrome" build recipe, whose steps are simulated here:

https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave.git/+/master/recipes/chrome.expected/full_chromeos_chrome_amd64_generic_Trusty__Informational_.json

The high-level build parameters are defined here:
https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave.git/+/master/recipe_modules/chrome/chromeos_chrome.py

The main driver is the "chromium" recipe:
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave/recipe_modules/chromium/api.py

I don't have any idea why this stopped working. It looks like the functional error is:
ninja: Entering directory `/mnt/data/b/build/slave/chrome-tot-chromeos-amd64-generic-trusty/build/src/out_amd64-generic/Release'
ninja: fatal: chdir to '/mnt/data/b/build/slave/chrome-tot-chromeos-amd64-generic-trusty/build/src/out_amd64-generic/Release' - No such file or directory
[Errno 2] No such file or directory: '/mnt/data/b/build/slave/chrome-tot-chromeos-amd64-generic-trusty/build/src/out_amd64-generic/Release/.ninja_log'

It looks like this directory doesn't exist:
/mnt/data/b/build/slave/chrome-tot-chromeos-amd64-generic-trusty/build/src/out_amd64-generic/Release

Logging onto the bot, the parent directory does exist, but it is empty:
/mnt/data/b/build/slave/chrome-tot-chromeos-amd64-generic-trusty/build/src/out_amd64-generic/

I know that:
a) This used to work.
b) I didn't break it.
c) A *lot* has changed since I looked at this last.
d) We don't have much history to go off of, since the bot only goes back ~2 days.

Looking at the history (https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave.git/+log/master/recipes/chrome.expected/full_chromeos_chrome_amd64_generic_Trusty__Informational_.json) for the recipe expectations, and assuming this breakage happened recently, two things changed:
1) (4 weeks ago) "--build-tool ninja" was removed from the command-line: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave.git/+/d87ed8e620bda91d3d1ef41c109b90621ebdcef0%5E%21/#F21
2) (3 weeks ago) The "cwd" was set for runhooks: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave.git/+/a2d431029203b2a37d65ddb3ffbea796892ac72e%5E%21/#F1

Questions:
1) Was this always building with Ninja? If not, maybe switching to Ninja almost worked but forgot to generate the Release directory.
2) Does anyone actually care about these builders? If so, why wasn't this sheriffed earlier when it actually started breaking so we had some history available?
3) Pawel, you have been way more integrated into the Chromium build than I have been. The extra CrOS parameters look like they're still doing what they always did. Any idea why this might have stopped working?

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/0cebc71cb39bc08119a29c8687891ff505684753

commit 0cebc71cb39bc08119a29c8687891ff505684753
Author: phajdan <phajdan@google.com>
Date: Mon Aug 08 13:24:43 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/179a2c254da922b1768e31d698cabb28358c638a

commit 179a2c254da922b1768e31d698cabb28358c638a
Author: Paweł Hajdan, Jr <phajdan.jr@chromium.org>
Date: Mon Aug 08 13:36:30 2016

mb_config.pyl: add entry for "amd64-generic Trusty (Informational)" on chromeos.chrome

TBR=dpranke

BUG= 633350 

Review URL: https://codereview.chromium.org/2224993002 .

Cr-Commit-Position: refs/heads/master@{#410340}

[modify] https://crrev.com/179a2c254da922b1768e31d698cabb28358c638a/tools/mb/mb_config.pyl

It's now at least building: https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/amd64-generic%20Trusty%20%28Informational%29/builds/43151

Please verify it's doing what you expect though.

Cool! For posterity's sake, can you explain what went wrong and why that fixed it?

So that did fix the problem, thank you.

Unfortunately I didn't follow most of comment #7. I thought I finally had a handle on all of the 

 but if this is something that "*should not* be maintained by the Chrome Infra team", but is "not "cbuildbot"-based builders, so it doesn't have anything to do with CrOS infra", that strikes me as untenable.

I will open a separate issue to brainstorm a maintainable solution for a simple chrome builder.

What I meant by that is that Chrome Infrastructure generally doesn't know:
1) What build configurations are needed by various teams.
2) The specifics of what those build configurations are supposed to do, or what makes them special.
3) Whether they are sufficient for their goals.

If you say, "here's a script, I'd like to run it every time this event happens", we can take it from there, make that happen, and make sure all of the systems involved are working as intended. When something goes wrong with the script itself, it's generally not something infrastructure is best suited to handle or own, although we're happy to help where possible.

Each builder is generally owned by some group of people with expertise on what that specific builder is supposed to be doing. These builders seem to not be, which means that any debugging has to start from scratch and just sort of defaults to infra, which is not ideal.