Transient (lasting a couple hours?)

pprabhu@pprabhu:~$ curl https://chromium.googlesource.com/chromium/src.git/+refs/tags?format=JSON | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0)]}'
{
  "10.0.601.0": {
    "value": "9041f79a9b38d724a97537b43c4f07bc91c9e37b"
  },
  "10.0.602.0": {
    "value": "f8070eb68e4537f87d341f0e9d6fcf19fedba7ea"
  },
  "10.0.603.0": {
    "value": "42d654daa74834b8dd0ed5de50bf9b232436a25e"
100  7620    0  7620    0     0  17723      0 --:--:-- --:--:-- --:--:-- 17720


I have no issues pinging that repo.
And this is the external repo. You don't even need any credentials to get to it, iirc.

This is the error text I see on the pfq-chrome-informational builders, e.g:

https://uberchromegw.corp.google.com/i/chromeos.chrome/builders/cyan-tot-chrome-pfq-informational/builds/3054

(These builders are failing consistently)

GET /a//chromium/src.git/+/25472e0c801c621e7bd907bbbc57b317efa56fa0/chrome/VERSION?format=text HTTP/1.1
HTTP/1.1 301 Moved Permanently
Response body: '<a href="/a/chromium/src.git/+/25472e0c801c621e7bd907bbbc57b317efa56fa0/chrome/VERSION?format=text">Moved Permanently</a>.\n\n'
12:21:13: WARNING: conn.sock.getpeername(): ('74.125.21.82', 443)

Note that there is a small difference between the OP and #3.

I can repro the failure from #3:

pprabhu@pprabhu:chromite$ curl https://chromium.googlesource.com//a//chromium/src.git/+/f11d62080a80c7de948ec4c4d7e9c1673becb257/chrome/VERSION?format=text
<a href="/a/chromium/src.git/+/f11d62080a80c7de948ec4c4d7e9c1673becb257/chrome/VERSION?format=text">Moved Permanently</a>.



That's trying to get to a particular href that doesn't exist.

FYI: There's some other P0 bugs so I'm not expecting to get to this in a hurry.

This is blocking the PFQ completely, so it's kind of a P0 also (but I won't escalate.... yet....)

Is there anyone else that might be able to help with this one?

+chrome infra troopers in case they are aware of a change to https://chromium.googlesource.com that might somehow be responsible?

#2 

https://chromium.googlesource.com/chromium/src.git/+refs/tags?format=JSON 
is not the same as
https://chromium.googlesource.com/a/chromium/src.git/+refs/tags?format=JSON
is not the same as
https://chromium.googlesource.com/a//chromium/src.git/+refs/tags?format=JSON

The first one works, the second requires a login, the third returns 301.

HTTP 301 is a redirect, it doesn't mean the URL is wrong (that's 4xx).

The redirect actually points back to the second URL, which requires a login.

So a couple of things:

1. Some library somewhere is not handling the 301 transparently.
2. Even if the redirect is handled, is said library set up with HTTP basic auth?

Could this be related to the LATEST changes?

We haven't committed any yet...?

Re #11 Ah... I thought part of them were in.

Re #8 Good detective work.

Did the URL used change recently? Perhaps incorrectly?

It seems like using the first one you listed is the right thing to do.

So we hand-rolled our own http request code in our gob library.  We of course treat anything that's not a 200 as an error, even though 3xx are very clearly not errors and should be handled transparently.

That answers 1.  I'll keep digging for 2, and perhaps #12

Spoke too soon.  We have an if check against 302, 303, and 307, but not 301.  Apparently that's supposed to mean authorization error (HTTP 401, 403).

I'll go ahead and fix the double slash, and we'll see if that magically fixes everything.

I suspect the GoB fixed their end to redirect /a//* to /a/* which is in my opinion the correct behavior, but our decidedly not correct code was insufficiently prepared for its lot in life.

The URL comes from here:
https://cs.corp.google.com/chromeos_public/chromite/lib/constants.py?rcl=b3992a74d789cf8aaa75c02e370d36def33537cd&l=475

And is modified/used here:
https://cs.corp.google.com/chromeos_public/chromite/scripts/cros_mark_chrome_as_stable.py?rcl=b3992a74d789cf8aaa75c02e370d36def33537cd&l=142


If I read that code correctly, it should have generated the URL:

https://chromium.googlesource.com/chromium/src.git/+refs/tags?format=JSON 

Which begs the question.... where did the /a/ values come from?

Neither constants.py or cros_mark_chrome_as_stable.py have changed within the last month.

I believe there is not a bad source code change in this path. There has either been a chance in builder configuration, or a GoB change of some kind.

We can add additional logging to cros_mark_chrome_as_stable to confirm what I think I've read. If that confirms, we should invole the GoB team.

I'll add the additional logging.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7580576717627b3e28e62da3ede6435abff6d086

commit 7580576717627b3e28e62da3ede6435abff6d086
Author: Allen Li <ayatane@chromium.org>
Date: Sat Jun 24 05:56:46 2017

gob: Fix double slash in GoB URLs

BUG= chromium:736427 
TEST=None

Change-Id: Iee31fd33cb5c78cb3d25da04e9ad16d0cd897d2c
Reviewed-on: https://chromium-review.googlesource.com/547002
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/7580576717627b3e28e62da3ede6435abff6d086/lib/gob_util.py

PFQs are still failing, but not because of this.

Re-opening as I see same error in M60 branch which started around same date:
https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-chrome-pre-flight-branch%20release-R60-9592.B/builds/102/steps/SyncChrome/logs/stdio

latest: 
https://uberchromegw.corp.google.com/i/chromeos_release/builders/samus-chrome-pre-flight-branch%20release-R60-9592.B/builds/121/steps/SyncChrome/logs/stdio

 Issue 737608  has been merged into this issue.

It sounds like the fix needs to be backported to M60, M59, etc.  The respective owners are welcome to go ahead and do that since I am not familiar with how that process works.

CLs:
M60: https://chromium-review.googlesource.com/c/552860
M59: https://chromium-review.googlesource.com/c/552861 

Land once the merge request is approved here for each milestone.

This bug requires manual review: M60 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I'm guessing josafat@ meant Merge-Approved-60 there as well?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5787bda402ccd4bb0efc9de6efad2f4c80f480a7

commit 5787bda402ccd4bb0efc9de6efad2f4c80f480a7
Author: Allen Li <ayatane@chromium.org>
Date: Wed Jun 28 23:35:57 2017

gob: Fix double slash in GoB URLs

BUG= chromium:736427 
TEST=None

Change-Id: Iee31fd33cb5c78cb3d25da04e9ad16d0cd897d2c
Reviewed-on: https://chromium-review.googlesource.com/547002
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
(cherry picked from commit 7580576717627b3e28e62da3ede6435abff6d086)
Reviewed-on: https://chromium-review.googlesource.com/552861
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/5787bda402ccd4bb0efc9de6efad2f4c80f480a7/lib/gob_util.py

This bug requires manual review: M60 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), josafat@(ChromeOS), bustamante@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/991f9a1e35c92a355f85abbbce61663d101fbc7b

commit 991f9a1e35c92a355f85abbbce61663d101fbc7b
Author: Allen Li <ayatane@chromium.org>
Date: Wed Jun 28 23:37:52 2017

gob: Fix double slash in GoB URLs

BUG= chromium:736427 
TEST=None

Change-Id: Iee31fd33cb5c78cb3d25da04e9ad16d0cd897d2c
Reviewed-on: https://chromium-review.googlesource.com/547002
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
(cherry picked from commit 7580576717627b3e28e62da3ede6435abff6d086)
Reviewed-on: https://chromium-review.googlesource.com/552860
Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/991f9a1e35c92a355f85abbbce61663d101fbc7b/lib/gob_util.py