Woah, it looks like the search results are appended at the bottom of the page somehow.

I reverted the version on the Flakiness Dashboard, but that doesn't seem to help. 

I believe this is probably related to the stale builder data though, since I was able to find references in the logs starting Dec 4 that there were errors in uploading Builder data. ie: 

{
 insertId:  "5a25a165000c3cd0b79344ea"  
 labels: {
  clone_id:  ""   
 }
 logName:  "projects/test-results-hrd/logs/appengine.googleapis.com%2Frequest_log"  
 operation: {
  first:  true   
  id:  "5a25a15b00ff0a9cbb30805e4c0001737e746573742d726573756c74732d687264000131333032342d36366463386131000100"   
  producer:  "appengine.googleapis.com/request_id"   
 }
 protoPayload: {
  @type:  "type.googleapis.com/google.appengine.logging.v1.RequestLog"   
  appId:  "s~test-results-hrd"   
  endTime:  "2017-12-04T19:26:26.939365Z"   
  first:  true   
  host:  "test-results.appspot.com"   
  httpVersion:  "HTTP/1.1"   
  instanceIndex:  -1   
  ip:  "74.125.248.84"   
  latency:  "7.243882s"   
  line: [
   0: {
    logMessage:  "Processing full_results.json for master chromium.memory, builder Linux TSan Tests, build 15050, test type net_unittests"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:20.524908Z"     
   }
   1: {
    logMessage:  "Added data entry. :: {"id":5904504584929280, "index":0, "size":1046528}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:20.773007Z"     
   }
   2: {
    logMessage:  "Added data entry. :: {"id":6224952900190208, "index":1, "size":1046528}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:20.897836Z"     
   }
   3: {
    logMessage:  "Added data entry. :: {"id":4644369501782016, "index":2, "size":1046528}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.023248Z"     
   }
   4: {
    logMessage:  "Added data entry. :: {"id":5579023641739264, "index":3, "size":1046528}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.132736Z"     
   }
   5: {
    logMessage:  "Added data entry. :: {"id":6356410843267072, "index":4, "size":1046528}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.277910Z"     
   }
   6: {
    logMessage:  "Added data entry. :: {"id":5805975350804480, "index":5, "size":1046528}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.414299Z"     
   }
   7: {
    logMessage:  "Added data entry. :: {"id":6338533041897472, "index":6, "size":804078}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.490447Z"     
   }
   8: {
    logMessage:  "Read loading DataEntry ID: 5614079030591488"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.598091Z"     
   }
   9: {
    logMessage:  "Read loading DataEntry ID: 6262917458296832"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.598994Z"     
   }
   10: {
    logMessage:  "Added data entry. :: {"id":4723044578492416, "index":0, "size":236864}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.885513Z"     
   }
   11: {
    logMessage:  "deleteKeys: enqueing :: {"keys":[]string{"ahJzfnRlc3QtcmVzdWx0cy1ocmRyFgsSCURhdGFFbnRyeRiAgPCSmr_8CQw"}}"     
    severity:  "INFO"     
    time:  "2017-12-04T19:26:21.888750Z"     
   }
   12: {
    logMessage:  "Added data entry. :: {"id":4758122247487488, "index":0, "size":267437}"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:21.889190Z"     
   }
   13: {
    logMessage:  "deleteKeys: enqueing :: {"keys":[]string{"ahJzfnRlc3QtcmVzdWx0cy1ocmRyFgsSCURhdGFFbnRyeRiAgPDS8YKQCww"}}"     
    severity:  "INFO"     
    time:  "2017-12-04T19:26:21.892160Z"     
   }
   14: {
    logMessage:  "adding taskqueue task for [/internal/monitoring/upload]"     
    severity:  "DEBUG"     
    time:  "2017-12-04T19:26:22.056276Z"     
   }
   15: {
    logMessage:  "eventupload: Uploader.Put failed :: {"error":"1 row insertion failed"}"     
    severity:  "ERROR"     
    time:  "2017-12-04T19:26:24.550364Z"     
   }
  ]
  method:  "POST"   
  requestId:  "5a25a15b00ff0a9cbb30805e4c0001737e746573742d726573756c74732d687264000131333032342d36366463386131000100"   
  resource:  "/testfile/upload"   
  startTime:  "2017-12-04T19:26:19.695483Z"   
  urlMapEntry:  "_go_app"   
  userAgent:  "Python-urllib/2.7"   
  versionId:  "13024-66dc8a1"   
 }
 receiveTimestamp:  "2017-12-04T19:26:29.804003152Z"  
 resource: {
  labels: {
   module_id:  "default"    
   project_id:  "test-results-hrd"    
   version_id:  "13024-66dc8a1"    
   zone:  "us12"    
  }
  type:  "gae_app"   
 }
 severity:  "ERROR"  
 timestamp:  "2017-12-04T19:26:19.695483Z"  
}

So it seems there are problems uploading to Bigquery due to size. Not completely sure how to fix this but looking into it. 

+ katthomas since I think she might know more about this from the eventupload side. 
+ Troopers since, unfortunately, my test-results experience is a bit too limited for me to figure this out quickly, so maybe an extra pair of eyes might help. 

Trooper checking in -- is this actually a P0...?

Also, #0: clicking on that link shows that test, and that test alone, on four builders:

  linux-chromeos-dbg
  linux-chromeos-rel
  Linux Chromium OS ASan LSan Tests (1)
  Linux ChromiumOS MSan Tests

Given that the test in question is a chromeos test, this seems roughly accurate.

I know I've seen the flakiness dashboard just give up on updating when new inputs are specified, but the URL updates, so refreshing typically gives the expected result. I have no idea how long that's been the case.

Hmm, flakiness dashboard should query test-results data in Dremel. That data is reported by event_mon, not event_upload. I don't think that eventupload operation failing should cause the event_mon operation to fail. So I think that error is unrelated. 

I believe this can probably be lowered to P1. 

Sean is OOO at the moment, but I believe he's been working on changing the Flakiness Dashboard schema, so that we don't run into row size problems anymore. 

#7: lowering priority (and removing troopers) SGTM given that this does appear to work at the moment with some finagling.

Ah, yeah, if you reload the page every time you change the search it seems to work. I swear I had tried that before filing the bug. In either case,that's non-obvious enough that it's basically completely broken.

Given that, it seems very likely someone broke the JS on this page and that it's not related to anything on the server-side.

There are two pretty serious issues here - 1) stale data and 2) loading results at the bottom of the very long page.

Tiffany, can you break this down into sub-bugs?

@jparent: Sure, https://bugs.chromium.org/p/chromium/issues/detail?id=792175

It seems odd to me that this would be a problem in the frontend, though, since I reverted to a version from Nov 10 and that didn't fix it. So if it is a frontend issue, it'd have to have been happening for a long time. 

The stale data notices seem to be gone, so I wonder if some updates might have gone through later and caused this to work again. 

@zhangtiff: In #11, by, "this", are you referring to the stale data or the appending results?

If the stale data, I wouldn't expect that to be a frontend problem.  That later could certainly have been happening for longer than Nov 10, as its incredibly not obvious what is going on, and people very well could have either thought it was totally broken or that you just weren't supposed to be able to click through to see results.

It can wait for Sean to get back, but, I'd like a follow up on the stale data.  This was a problem we saw a *long* time ago that I thought was resolved, but if it isn't, it needs to be tracked down (and monitoring needs to be added - stale data should be firing alerts).

By "this", I meant appending the results. This particular bug doesn't seem to be happening for me when I test it, so I wonder if it could be that the stale data caused the frontend to not clear the old results, but it could also just be flaky. 

I'll switch the bugs so that this one is about the UX issue. 

That's ... interesting, because its happening for me.  Did you see it happening yesterday?  I wonder if this is a device/chrome version problem?

Huh, it's happening for me now. I thought it stopped happening at some point, but I might be wrong. anyways, I think the frontend issue seems like it should be fixable, and it's causing a lot of user pain, so I'll work on it. 

Wait, this is test-results not flakiness dashboard (vi/chrome_infra/Services/flakiness_pipeline), correct? 

[I emphatically did *not* come up with these names, but to clarify:]

"flakiness dashboard" == https://test-results.appspot.com/dashboards/flakiness_dashboard.html 

So to answer your question, this is referring to *both* test-results and "flakiness dashboard" (because one is hosted on the other). But not the viceroy page.

I don't know if anyone in the chromium dev world calls that flakiness_pipeline viceroy page the "flakiness dashboard". They are almost always talking about the flakiness_dashboard.html page hosted on test-results.appspot.com when they use that term.

What Sean said.  https://test-results.appspot.com/dashboards/flakiness_dashboard.html is "The Flakiness Dashboard".  Pretty sure no one in chromium dev knows about viceroy consoles.

Thank you!

 Issue 800856  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/04cd58a49e860bab1553a9796e7f3531726426e8

commit 04cd58a49e860bab1553a9796e7f3531726426e8
Author: Sean McCullough <seanmccullough@chromium.org>
Date: Wed Jan 10 20:19:11 2018

[test-results] Replace instead of append divs that already exist.

Bug:  791651 
Change-Id: I761f985bf06f7e7faa2e3826bb1c759b7f43ca09
Reviewed-on: https://chromium-review.googlesource.com/860867
Reviewed-by: Tiffany Zhang <zhangtiff@chromium.org>
Commit-Queue: Sean McCullough <seanmccullough@chromium.org>

[modify] https://crrev.com/04cd58a49e860bab1553a9796e7f3531726426e8/go/src/infra/appengine/test-results/frontend/static/dashboards/js/flakiness_dashboard.js

Pushed this fix to prod.