Issue 776469  has been merged into this issue.

I think I know the cause of this. Looking into it. 

So, this seems to be a little less straightforward to fix than I thought. I found an example of this bug in production and staging with the group "bot affinity 'build105-m1' is broken on chromium.perf/Win 7 ATI GPU Perf, affecting 30 tests" but I wasn't able to repliate the bug locally with the same group. 

Sean/Martiniss: Do you know of any possible differences in routing between the local and prod environments? 

The 404 response is "404 page not found" rather than any of the custom 404 responses in the code paths, so I think this is probably a top level routing problem. 

 Issue 777815  has been merged into this issue.

app.yaml may be involved, though it could also be an issue with paths getting truncated (if they're too long; I don't know what the limit is though) by GFE.

 Issue 780522  has been merged into this issue.

example URL that 404s when the client tries to POST: https://sheriff-o-matic-staging.appspot.com/api/v1/annotations/chromium.perf.Android%20One%20Perf.smoothness.tough_texture_upload_cases.smoothness.tough_texture_upload_cases%2Fextra_large_texture_uploads.html/add

The request log for that 404 on staging:

{
 httpRequest: {
  status:  404   
 }
 insertId:  "59fa108a000060594bdf0c08"  
 labels: {
  clone_id:  "00c61b117caadb315ac8dfe8af4d98adcd01fb940267aab474de053aa806216e4850dc0510f8fd95"   
 }
 logName:  "projects/sheriff-o-matic-staging/logs/appengine.googleapis.com%2Frequest_log"  
 operation: {
  first:  true   
  id:  "59fa108a00ff00ff512d0c61aa390001737e736865726966662d6f2d6d617469632d73746167696e67000131323536382d66343765303866000100"   
  last:  true   
  producer:  "appengine.googleapis.com/request_id"   
 }
 protoPayload: {
  @type:  "type.googleapis.com/google.appengine.logging.v1.RequestLog"   
  appEngineRelease:  "1.9.54"   
  appId:  "s~sheriff-o-matic-staging"   
  cost:  1.2627999999999999e-8   
  endTime:  "2017-11-01T18:20:58.024390Z"   
  finished:  true   
  first:  true   
  host:  "sheriff-o-matic-staging.appspot.com"   
  httpVersion:  "HTTP/1.1"   
  instanceId:  "00c61b117caadb315ac8dfe8af4d98adcd01fb940267aab474de053aa806216e4850dc0510f8fd95"   
  instanceIndex:  -1   
  ip:  "2620:0:1002:1006:cd5a:cec3:c6c5:f0f0"   
  latency:  "0.003609s"   
  megaCycles:  "26"   
  method:  "POST"   
  requestId:  "59fa108a00ff00ff512d0c61aa390001737e736865726966662d6f2d6d617469632d73746167696e67000131323536382d66343765303866000100"   
  resource:  "/api/v1/annotations/chromium.perf.Android%20One%20Perf.smoothness.tough_texture_upload_cases.smoothness.tough_texture_upload_cases%2Fextra_large_texture_uploads.html/add"   
  responseSize:  "113"   
  startTime:  "2017-11-01T18:20:58.020781Z"   
  status:  404   
  traceId:  "975e7a29d5e8e0affe917b22415d9cc7"   
  urlMapEntry:  "_go_app"   
  userAgent:  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36"   
  versionId:  "12568-f47e08f"   
 }
 receiveTimestamp:  "2017-11-01T18:20:58.029553084Z"  
 resource: {
  labels: {
   module_id:  "default"    
   project_id:  "sheriff-o-matic-staging"    
   version_id:  "12568-f47e08f"    
   zone:  "us12"    
  }
  type:  "gae_app"   
 }
 timestamp:  "2017-11-01T18:20:58.020781Z"  
}

unable to repro locally. This exact request works on devserver but not in prod.

Devserver gets the same request as staging (as logged on the client) while staging server sees a slightly different request than devserver as logged on the server.

devserver succeeds and logs a request for:
"/api/v1/annotations/chromium.perf.Android%20One%20Perf.smoothness.tough_texture_upload_cases.smoothness.tough_texture_upload_cases%252Fextra_large_texture_uploads.html/add" 

while staging fails and logs a request for:
"/api/v1/annotations/chromium.perf.Android%20One%20Perf.smoothness.tough_texture_upload_cases.smoothness.tough_texture_upload_cases%2Fextra_large_texture_uploads.html/add"

The difference is the first 2 chars of "%2Fextra_large_texture_uploads" (staging, 404) vs "%252Fextra_large_texture_uploads" (devserver, 200)

Perhaps the GFE is doing some (un)escaping before handing the request to our code?

"smoothness.tough_texture_upload_cases/extra_large_texture_uploads.html" is the string in question and it contains a "/" at the position where we see the difference. 

I'm going to just put the key and action parameters into the JSON body of the POST request. 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9

commit f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Nov 01 21:19:07 2017

[som] Move annotation key into POST body.

Bug:  775266 
Change-Id: Iccc9eb49e42372bed2f107ab1e492860ff507f7e
Reviewed-on: https://chromium-review.googlesource.com/749688
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9/go/src/infra/appengine/sheriff-o-matic/frontend/test/som-annotations-test.html
[modify] https://crrev.com/f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9/go/src/infra/appengine/sheriff-o-matic/som/handler/main_test.go
[modify] https://crrev.com/f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9/go/src/infra/appengine/sheriff-o-matic/frontend/main.go
[modify] https://crrev.com/f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9/go/src/infra/appengine/sheriff-o-matic/frontend/elements/som-annotations/som-annotations.js
[modify] https://crrev.com/f8cd51e0edb618970c8d1cb7be3d53e9bd4fb8a9/go/src/infra/appengine/sheriff-o-matic/som/handler/annotations.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e1336ba2dc768a397b8dd4aa527e42ecd5531330

commit e1336ba2dc768a397b8dd4aa527e42ecd5531330
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Nov 01 23:17:46 2017

[som] Update RELNOTES.md for an off-schedule push

Bug:  775266 
Change-Id: I2f2efb0afdf69968933999bef7c40424194681e4
Reviewed-on: https://chromium-review.googlesource.com/749677
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/e1336ba2dc768a397b8dd4aa527e42ecd5531330/go/src/infra/appengine/sheriff-o-matic/RELNOTES.md

Thanks so much for the quick fix!!

 Issue 674624  has been merged into this issue.