The problem isn't specific to your CL or the new test suite. The iOS bots don't upload results properly, because they test multiple configurations on the same bot and so the test-results server thinks they are duplicates.

There's a bug on file for this somewhere, I think, but I'm not finding it at the moment.

@sergeyberezin - can I assign this to you to de-dupe as appropriate?

Thanks dpranke@! I'll go ahead and submit that CL.

#1 shouldn't be the case because we are not duplicating data sent to the dashboard. We adjust the --builder-name to ensure data isn't duplicated. And the first ios_web_view_inttests upload shouldn't fail regardless because the first one for a build couldn't possible be a dupe, and yet it failed.

Here's the specific error:
test_results_uploader.PermanentError: Received HTTP status 400 loading "https://test-results.appspot.com/testfile/upload": model: nil FixableCounts

I believe we would be getting a different error if it was a dupe.

okay, sergiyb@, can you take a look?

The error message is not very descriptive. I'll fix that. Essentially the uploaded JSON is missing required "num_failures_by_type" field. Please add that field and error should be gone. The reason for "FixableCounts" error is that we still accept deprecated field "fixable", which is automatically converted to "num_failures_by_type", but I wouldn't rely on this logic since it's deprecated and can be removed at any point.

FYI: JSON format is documented here: https://www.chromium.org/developers/the-json-test-results-format.

CL to improve error message: https://chromium-review.googlesource.com/c/561139/

The full_results.json for that run is at https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=2a2f8222c4748f6ee8756d8f847066b8b8e5f532&as=full_results.json

It looks like it has the necessary field already?

"num_failures_by_type": {"FAIL": 0, "PASS": 5},

Interestingly enough, the upload failures did not happen on the main waterfall, even though the json file was identical (except for the timestamp): https://build.chromium.org/p/chromium.mac/builders/ios-simulator/builds/19306


I see similar failures on the main waterfall for ios_chrome_integration_egtests, which also uploads data that has the required field:

https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=ac8d0ba8ace8cc18bf38bc69a98b2fa2459c14d9&as=full_results.json

Hm, according to [1], Test Result Server expects this field to be a dict mapping strings to lists of integers. This is not consistent with documented field type in https://www.chromium.org/developers/the-json-test-results-format. Even the comment next to field seems to suggest that it should be map[string]int. I'll fix that.

[1]: https://cs.chromium.org/chromium/infra/go/src/infra/appengine/test-results/model/aggregate_result.go?l=82&rcl=987c685563e5f9976233d696aae7c71e4daf9442

Scratch that. The [1] from previous comment was linking a model for AggregatedResult, not the FullResult type that is used to parse uploaded full_results.json files.

After looking more through the code, I think the reason this fails is that results fail to be aggregated with older results, which are already in the app and are missing the required field.

Found it: https://test-results.appspot.com/testfile?builder=ios-simulator-iPhone%206s-10.0&name=results.json&master=tryserver.chromium.mac&testtype=ios_web_view_inttests. To fix this, you'll need to either manually go through the datastore and delete corrupted files (not sure how they got there) or write code in the app to fix them automatically. I guess it depends on the number of different builders/steps that are failing with this error...

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c8205894a77af1b8da38e7954dbd3d75bd10052e

commit c8205894a77af1b8da38e7954dbd3d75bd10052e
Author: Sergiy Byelozyorov <sergiyb@chromium.org>
Date: Thu Jul 06 21:35:53 2017

Improve error message to make it actionable by clients

R=smut@chromium.org

Bug: 739387
Change-Id: I827c8c774d4c24b992b563d92da5c8f8ed3a3f34
Reviewed-on: https://chromium-review.googlesource.com/561139
Commit-Queue: smut <smut@google.com>
Reviewed-by: smut <smut@google.com>

[modify] https://crrev.com/c8205894a77af1b8da38e7954dbd3d75bd10052e/go/src/infra/appengine/test-results/model/aggregate_result.go