Shall we teach the recipe how to build 32-bit packages on that builder or do you want to disable it?

The suffix comes from self.m.cipd.platform_suffix(): https://cs.chromium.org/chromium/infra/recipes/recipe_modules/third_party_packages/api.py?q=third_party_packages/api.py&sq=package:chromium&dr&l=61

Which is derived from platform.bits: https://cs.chromium.org/chromium/tools/depot_tools/recipes/recipe_modules/cipd/api.py?q=cipd/api.py&sq=package:chromium&dr&l=207

Which is derived from python's 'platform.machines()' here:
https://cs.chromium.org/chromium/infra/recipes-py/recipe_modules/platform/api.py?dr&q=platform/api.py&sq=package:chromium&l=28

---

The bot is vm185-m1. On this bot:
$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=f26e73b24e72f5a6d787a8ccbb30be6ca2251cc1, stripped

$ file /bin/bash
/bin/bash: ELF 32-bit LSB  executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=4ead65aeca4e9f1eabf3a0d63eb1f96c225b25fd, stripped

$ dpkg --print-architecture
i386

So user land is definitely 32 bit.

But 

$ python -c "import platform; print platform.machine()"
x86_64

Which makes recipes think it is 64 bit system.

This is violation of what platform.bits() recipe API is supposed to return. From its doc:

"""Returns the bitness of the userland for the current system (either 32 or 64 bit)."""

Userland is 32 bit, but the function returns 64.

---

The question is what to do about that? Elsewhere we are shelling out to 'dpkg' to unambiguously grab userland bitness:

https://chromium.googlesource.com/infra/infra/+/master/build/build.py#587

Should we perhaps do it in 'platform' recipe module too?

Also, looks like the builder is really building i386 binaries and then packaging them as amd64 :-

Ew .. so yeah using "sys.platform" to determine target bitness is not going to work when the native platform is 64-bit. We should be telling this explicitly that it"s building 32-bit packages.

We can fix third_party_packages recipe relatively easily, but I still think we need to fix 'platform' recipe module, since it currently does wrong thing:

  @property
  def bits(self):
    """Returns the bitness of the userland for the current system (either 32 or
    64 bit).
    ...

LIES! it returns kernel bitness (at least on subset of machines)

We should change the third_party_package so it explicitly sets -m32 CFLAGS/LDFLAGS when building 32-bit packages. The bitness of the host is then irrelevant. The question is what's the right way to do that? We could check the buildername, or add an additional recipe property?

RE #5, I believe what's happening is:
1) We're using a CIPD-deployed Python bundle, which is specified as .../${platform}.
2) ${platform} is expanded by CIPD based on the kernel bitness, so 64-bit here.
3) This means we're deploying 64-bit Python bundle to all 64-bit kernels.
4) So "platform.bits" on this bundle is properly returning "64".

I see two options, either (a) make it so we use 32-bit Python here, so we have a 32-bit Python userspace; or (b) explicitly tell our build scripts which bitness they are building for and let them figure out the rest.

(a) is less appealing to me, both because it really shouldn't matter what the bitness of *Python* is and because that would require us to plumb a way for CIPD to know that, on some systems, "${platform}" should be 32-bit even if the kernel is 64-bit. I like (b) b/c it is explicit and doesn't rely on any external hacks.

RE hacks, systems can have multiple userlands, so it's not reasonable (IMO) to merely say "use what's first in PATH, that's your bitness". PATH can get messed up and shuffled around quite easily.

> 1) We're using a CIPD-deployed Python bundle, which is specified as .../${platform}.

Nope. Buildbot (and recipes) are using system python.

> 2) ${platform} is expanded by CIPD based on the kernel bitness, so 64-bit here.

Nope. Puppet uses userland bitness for ${platform}. I've confirmed that /opt/infra-system is also 32 bit. (But it is irrelevant, since it's not used to run recipes).

> Nope. Buildbot (and recipes) are using system python.

Yep, we're using the bundled python for these infra builders.

> Nope. Puppet uses userland bitness for ${platform}. I've confirmed that /opt/infra-system is also 32 bit. (But it is irrelevant, since it's not used to run recipes).

So cipd uses go's bitness; if puppet uses ruby bitness, and it's 32bit ruby, it might install 32bit cipd. cipd_bootstrap_v2 uses the buildbot-python's bitness (maybe 64) to pick a cipd, which will then expand ${arch} to amd64 within the recipe.

:-/ 

I completely forgot that we have a third python installed by cipd_bootstrap_v2. I'll reevalute my investigation.

Ok, so I think I found part of the cause:

/mnt/data/b/cipd_path_tools contains all 64bit tools (including python) because /mnt/data/b/cipd_client/cipd is 64bit.

I'm going to blow it all away and try a fresh build to see what it comes up with.

Yep, it bootstraps the wrong version

INFO:slave.cipd_bootstrap_v2:Bootstrapping fresh /mnt/data/b/cipd_client/cipd for version [git_revision:cf5b0de14e94b06274d15e84b19449f580576e80] on platform [linux-amd64]


https://uberchromegw.corp.google.com/i/chromium.infra.cron/builders/third-party-packages-linux-386/builds/734/steps/steps/logs/stdio

Yep, ok it's this code here:

https://cs.chromium.org/chromium/build/scripts/slave/infra_platform.py?q=infra_platform&sq=package:chromium&l=38

Basically infra_platform.get() is returning a triple of `('linux', 'amd64', 32)`.

er, ('linux', 'x86_64', 32)... but still, it's clearly wrong.

I think the solution is to change cipd_arch to return the lesser of machine/bits. It's only used by cipd_bootstrap_v2, afaict

https://chromium-review.googlesource.com/646239

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/6c5c7e9c7d95ea876cd1275fc0ff0e70e61323ed

commit 6c5c7e9c7d95ea876cd1275fc0ff0e70e61323ed
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Sep 01 17:56:44 2017

Fix cipd bootstrap bitness on 64bit linux with 32bit userland.

R=dnj@chromium.org, vadimsh@chromium.org

Bug:  760841 
Change-Id: I6c91baf8a7ea0234e6533664f5bbbf678b183787
Reviewed-on: https://chromium-review.googlesource.com/646239
Reviewed-by: Daniel Jacques <dnj@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/6c5c7e9c7d95ea876cd1275fc0ff0e70e61323ed/scripts/slave/cipd_bootstrap_v2.py
[modify] https://crrev.com/6c5c7e9c7d95ea876cd1275fc0ff0e70e61323ed/scripts/slave/infra_platform.py