+@sosa, do you know what is going on here?

This looks like a bad bot configuration issue (bad gitcookies). Ping the chrome infra trooper.

+ chrome trooper

Looks like this morning's build succeeded where the previous build failed. 

Green:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R49-7834.B/builds/55

Red:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R49-7834.B/builds/54

I'll have a look at possible authentication issues and report back here.

Turned out the machine the build is running on is not under Chrome-Infra's control, but ChromeOS Infra's.

sosa: could you find a ChromeOS infra member who can take care of that?

Which machine? what kind of action is needed?

It looks like we need to authorize git access https://chrome-internal.googlesource.com
on cros130-c2 as mentioned on comment #3. I don't know which account to use either.

The machine name can be found on this page: https://uberchromegw.corp.google.com/i/chromeos_release/builders/master-release%20release-R49-7834.B/

+@fdeng, @akeshet. do you know how to do this?

Assigning to akeshet@ in the hope to get some traction here.

shuqianz@ you can try deleting and re-creating that instance, using chromite/compute/cros_compute . Some instructions available here: go/cros-gce-bots

That might fix it, if it is some weird local problem with the cookies on that bot.

+phobbs who is also touching the compute/cros_compute stuff to fix up our image and add some other credentials to it

when I recreate the instance, I need to specify the builder config option. What should I put here?
./cros_compute instances create --instance=cros130-c2 --config <???>

The configs are defined in one of those files in cros_compute. I think it's called configs or something.

The one called "master" or similar should be appropriate, since this is builder is apparently for a master config (ie it doesn't need to do much building work of its own, so it can be fairly small gce instance)

$ ./cros_compute instances create --instance=cros130-c2 --config=cros-master

14:35:48: INFO: RunCommand: gcloud compute instances create cros130-c2 --image chromeos-bot-v5 --metadata 'cros-image="chromeos-bot-v5"' --machine-type n1-standard-8 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview --project chromeos-bot --zone us-east1-a
ERROR: (gcloud.compute.instances.create) Some requests did not succeed:
 - The global natIP resource pool is exhausted
cros_compute: Unhandled exception:
Traceback (most recent call last):
  File "./cros_compute", line 164, in <module>
    commandline.ScriptWrapperMain(FindTarget)
  File "/usr/local/google/home/shuqianz/chromiumos/chromite/lib/commandline.py", line 834, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/usr/local/google/home/shuqianz/chromiumos/chromite/compute/cros_compute.py", line 256, in main
    gcctx.CreateInstance(opts.instance, address=opts.address, **config)
  File "/usr/local/google/home/shuqianz/chromiumos/chromite/compute/gcloud.py", line 269, in CreateInstance
    ret = self.DoZoneSpecificCommand(cmd, **kwargs)
  File "/usr/local/google/home/shuqianz/chromiumos/chromite/compute/gcloud.py", line 107, in DoZoneSpecificCommand
    return self.DoCommand(cmd, **kwargs)
  File "/usr/local/google/home/shuqianz/chromiumos/chromite/compute/gcloud.py", line 92, in DoCommand
    raise GCCommandError(e.msg, e.result, e.exception)
chromite.compute.gcloud.GCCommandError: return code: 1; command: gcloud compute instances create cros130-c2 --image chromeos-bot-v5 --metadata 'cros-image="chromeos-bot-v5"' --machine-type n1-standard-8 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview --project chromeos-bot --zone us-east1-a
cwd=None


Failed

Gack. Are we out of IP addresses in that project now?

dgarrett@ have you been adding a bunch of instances lately?

That command looks wrong, shouldn't it also specify the IP address?

gcloud compute instances create cros130-c2 --image chromeos-bot-v5 --metadata 'cros-image="chromeos-bot-v5"' --machine-type n1-standard-8 --scopes https://www.googleapis.com/auth/devstorage.full_control,https://www.googleapis.com/auth/gerritcodereview --project chromeos-bot --zone us-east1-a

Also, the external IP address block we use is not fully supported by gcloud and can lead to weird/random errors.

how can I check the IP address of the bot?

https://pantheon.corp.google.com/compute/instances?project=chromeos-bot&graph=GCE_CPU&duration=P4D

I've run './cros_compute instances delete --instance=cros130-c2' to delete this instance. Now I cannot find this bot using 'cros130-c2' on that page. Did I do something wrong?

The quota page shows that we are using 100% of our static IP addresses.

Static IP addresses us-east1	100% 256 of 256


However, if you check the networking page, it shows a bunch of addresses unused.

https://pantheon.corp.google.com/networking/addresses/list?project=chromeos-bot

I see it now.

The instance has been recreated.

Anything more to do here? Is this resolved?

I will assume it has been fixed, since no reply on this for weeks. Feel free to reopen it if I am wrong.

Bulk verified