Bisected this to the SDK roll in https://chromium-review.googlesource.com/1121686, so this looks like a regression/incompatible change in the Fuchsia SDK itself.

+jbudorick, dpranke FYI

Suspecting that this has the same root-cause as  issue 858692 ; if we were failing to (re)create the content_unittests.far, then the meta.far for it would refer to blobs that the stale archive didn't contain.

 Issue 858692  failed on June 28th, so too early to be related.

'pm archive' command brokenness is tracked by Fuchsia bug PKG-148, and SDK roll-back is in CQ.

This bug will track investigation of why our GN/ninja configuration didn't spot that the package archives were stale.

Issue appears to be with our rule that invokes 'pm archive', which ignores whether the archive file is stale with respect to its inputs.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/79cbcdf6fbd6bd8d204e7229e21ae4235a70dc9d

commit 79cbcdf6fbd6bd8d204e7229e21ae4235a70dc9d
Author: Wez <wez@chromium.org>
Date: Tue Jul 03 22:39:40 2018

Roll-back Fuchsia SDK to the last-known-good version.

Recent SDK builds have a broken 'pm archive' command, so builds have not
actually been rebuilding the package archives, and things have only been
"working" because our GN rules are broken in such a way that ninja
doesn't notice the brokenness.

TBR: thakis
Bug:  859955 
Change-Id: Ic71ccee289d4cee532f539e19f4d9baa63408e0a
Reviewed-on: https://chromium-review.googlesource.com/1125214
Reviewed-by: Wez <wez@chromium.org>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Commit-Queue: Wez <wez@chromium.org>
Cr-Commit-Position: refs/heads/master@{#572378}
[modify] https://crrev.com/79cbcdf6fbd6bd8d204e7229e21ae4235a70dc9d/base/test/launcher/test_launcher.cc
[modify] https://crrev.com/79cbcdf6fbd6bd8d204e7229e21ae4235a70dc9d/build/fuchsia/sdk.sha1

Adding thakis@, since scottmg@ is OOO, for GN/ninja input.  Right now if I:

1. Manually remove the base_unittests-0.far archive.
2. Touch base/run_loop.cc.
3. Run ninja on base_unittests.

then (with the broken SDK) the archive generation step "succeeds" but does not produce an output file, so ninja short-cuts the remaining dependent steps.  I'm surprised that a step that fails to produce the expected output(s) doesn't trigger a failure.

De-prioritizing and assigning to thakis@ for input on whether there's anything we can do in GN to mitigate this, versus just making sure to clobber once in a while...

What exactly did pm do? Not write anything and exit with a non-0 return code? Or with a 0 return code?

I can see how the latter might happen since ninja uses a timestamp of 0 for "file not found" internally, and if the output isn't found the 2nd time but the tool succeeds, then the output timestamp doesn't change and the restat optimization kicks in.

After https://github.com/ninja-build/ninja/pull/1292 (merged and in 1.8.2) we should probably not do the restat optimization if the timestamp is 0. But that's nothing that will change soon.

All the other platforms have a clobber bot on the main waterfall; Fuchsia should probably get one too regardless, since that catches all kinds of other issues too.

Re #13: Yes, exactly - 'pm archive' did nothing, but returned a zero exit-code.

Sounds like this issue is an ExternalDependency - or is there a Chromium bug already tracking the restat optimization interaction with not-found?

No, before Fuchsia's pm no tool ever managed to do nothing, successfully.

OK, closing this out as expected behaviour given the current ninja impl.