Today, I no longer see this with the same devices and the same or basically the same versions:
Chrome Version: 62.0.3199.3, 62.0.3199.4
Device: Nexus 5X and Pixel XL
Host: Linux 60.0.3112.113 and Mac 63.0.3206.0

However, klausw is seeing this.

Desktop version: Google Chrome	60.0.3112.113 (Official Build) (64-bit)
Revision	95c454326a7a3153e984e50a4719924968490717-refs/branch-heads/3112@{#744}
OS	Linux

Device: Pixel XL
Version: ToT build as of 3c65aab7d36bd856733eb0c73c03d1076b4022e0 ("gclient sync" may be a few days behind)

If I inspect the "404 Not Found The resource could not be found.", its document.location.href is: 

chrome-devtools://devtools/remote/serve_rev/@7331e76175e473136ef673768ed82c8c0be43b37/inspector.html?remoteFrontend=true&dockSide=undocked

Inspect works for an official Chrome Canary build.

To clarify, I was able to open the "chrome://inspect/?tracing#devices" page on my desktop browser successfully. The 404 appears when I click on the "inspect" link for one of the tabs open on the remote ToT build.

I think I figured out what's happening, though not sure why.

The core problem in my case seems to be that build/util/LASTCHANGE somehow ends up containing the hash of a commit that only exists in my local repository, so the appengine devtools doesn't know anything about it and I get a 404.

LASTCHANGE is apparently the output of build/util/lastchange.py according to https://cs.chromium.org/chromium/src/DEPS?q=LASTCHANGE&sq=package:chromium&dr=C&l=871 , though I'm unsure under which circumstances that gets run.

Next problem is that build/util/lastchange.py appears to assume that the most recent log entry is an official commit that contains a Cr-Commit-Position:

https://cs.chromium.org/chromium/src/build/util/lastchange.py?&sq=package:chromium&dr=C&l=59

However, somehow this ends up being run on a tree state where the most recent commit is one of my local work branches, so it's not valid.

As a quick hack, I tried modifying lastchange.py to look at a list of recent commits and take the first one with a Cr-Commit-Position, and then manually running:

$ build/util/lastchange.py -o build/util/LASTCHANGE

Example output:

$ cat build/util/LASTCHANGE
LASTCHANGE=cb4da43870053c2857661082f2ef29c938a69d6a-refs/heads/master@{#540119}

For comparison, the bad output looked something like:

$ build/util/lastchange.py 
LASTCHANGE=3460296d1ed94cc88ea09965d0b1c03a1199c72b-

I'm assuming this isn't the right way to fix it if the assumption is that LASTCHANGE gets auto-updated as part of syncing, and the real bug would be that this ends up in a bad state. 

In the meantime, would it be possible to add a sanity check that LASTCHANGE should have a refs/heads/master in it? Looks like the next step is that it gets written to gen/build/util/webkit_version.h as part of being baked into the APK.

Also, this doesn't just seem to be an isolated problem, several team members have this issue where they can't use the inspector with locally created builds.

There may of course be other reasons for these 404s, but in the cases I've specifically checked (including my current breakage and the one reported in comment #3, plus one from lincolnfrog@), devtools was using hashes that only exist locally.

I've put my change in https://chromium-review.googlesource.com/c/chromium/src/+/949312 , please let me know what you think.

The issue with no front-end available for your build is that it has not yet been pushed to the backend yet. The Cr-Commit-Position is working properly, but points to something that is too fresh.

pfeldman@, please see comments 3 and 5 - LASTCHANGE contained the hash of a local commit and did not have a Cr-Commit-Positon. For example, the LASTCHANGE=3460296d1ed94cc88ea09965d0b1c03a1199c72b- link to
 https://chromium.googlesource.com/chromium/src/+/3460296d1ed94cc88ea09965d0b1c03a1199c72b is invalid, and "git show" for that hash retrieves a local commit in my repository.

--- copied from the review thread

I'm aware of the not-yet-pushed issue, but this is something different. I've verified on three separate occasions (one of them on a teammate's machine) that the autogenerated LASTCHANGE file definitely contained a hash referring to a local-only commit not in origin/master, and that this hash was also used as part of the chrome-devtools:// URL. This local change is never going to be pushed to a backend as-is - even if the change ends up being submitted at a later point, it'll get a different hash after being merged. 

I agree that the core issue may be elsewhere since the lastchange hook which generates the file is apparently being unexpectedly executed at a time when the current workspace has local changes in it, but I don't know enough about the hook execution why this is happening and how to prevent it. The proposed change is a workaround.

Got it. gclient sync is going to rewrite the build/util/LASTCHANGE... 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b0f0d1a7402a84cda225408e7396f6454148d4d4

commit b0f0d1a7402a84cda225408e7396f6454148d4d4
Author: Klaus Weidner <klausw@chromium.org>
Date: Tue Mar 06 17:39:20 2018

Ignore recent local commits when computing the LASTCHANGE position.

For unclear reasons, the build/util/LASTCHANGE file frequently contained
the hash of a local-only commit from a working branch, and the result
was a 404 error when trying to inspect a page in devtools.

To avoid this, add a --filter regex that is passed to "git log --grep",
defaulting to '^Change-Id:'. This header is present for the main
source tree and third_party/skia/ official commits.

BUG= 760304 

Change-Id: I9464ba37a1197872a7ffd31b8c2f1a2a85856276
Reviewed-on: https://chromium-review.googlesource.com/949312
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Pavel Feldman <pfeldman@chromium.org>
Commit-Queue: Klaus Weidner <klausw@chromium.org>
Cr-Commit-Position: refs/heads/master@{#541137}
[modify] https://crrev.com/b0f0d1a7402a84cda225408e7396f6454148d4d4/build/util/lastchange.py

FYI, I haven't seen this issue recur since the patch from comment #11 was merged. There's still the known issue that brand-new ToT builds may not work due to delays in the push pipeline, but at least that seems to have solved the problem with custom builds having permanently broken inspector links.

 Issue 710701  has been merged into this issue.

Added a note https://chromium-review.googlesource.com/c/chromium/src/+/949312#message-f09b11bf4f756e45fd70b3fcbcd61a8d386ad0a7 suggesting that we don't rely solely on change-id, but this generally looks like the right approach.