More notes:

To describe what "update-flaky-expectations" does in prosey-pseudocode:

For each line in TestExpectations:
    If the line is a flaky expectation (e.g. [ Failure Pass ]),
        Check the latest results from the flakiness dashboard.
        If the test is actually not flaky,
            Remove the line.

In order to remove unexpected passes, we want to check tests with non-flaky fail expectations -- lines that say the test should fail or crash or timeout -- and if the test actually consistently passes, then that line should be removed.

This could be another script based on (and sharing code with) update_flaky_expectations.py, or if it's simpler if could be a different mode of update-flaky-expectations.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/83b05b0fc9bd8e5bd18de4219749e17165e6c557

commit 83b05b0fc9bd8e5bd18de4219749e17165e6c557
Author: Bugs Nash <bugsnash@chromium.org>
Date: Thu Sep 07 05:06:36 2017

Updated suggested CL description to match new gerrit syntax.

Bug:  730704 
Change-Id: I5855242cce027f44f20ec181b313cb187e897c35
Reviewed-on: https://chromium-review.googlesource.com/647953
Commit-Queue: Bugs Nash <bugsnash@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#500219}
[modify] https://crrev.com/83b05b0fc9bd8e5bd18de4219749e17165e6c557/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/update_flaky_expectations.py
[modify] https://crrev.com/83b05b0fc9bd8e5bd18de4219749e17165e6c557/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/update_flaky_expectations_unittest.py

bugsnash@, are you still working on this?

I was planning on picking this up again this week actually! I have dropped the ball on this cause I'm in the process of moving teams. Will let you know if I can't get it done :)

bugsnash@, this came up in triage of P2 issues older than 60 days. I see you're on leave, but rather than looking for someone else I think I'll call this a P3, since we can get along well enough without this and nobody is blocked on it, although it would be really nice to have.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5ad2dabdc5c202853fab082a356526412cc1988d

commit 5ad2dabdc5c202853fab082a356526412cc1988d
Author: Bugs Nash <bugsnash@chromium.org>
Date: Wed May 23 00:30:22 2018

Fix presubmit errors in update_flaky_expectations.py by styling const.

Fixes pylint presubmit errors in the update_flaky_expectations.py
script by adhering to constant styling.

Bug:  730704 
Change-Id: I365f2f642b1e64c6b1ee1837be99819a95b1e160
Reviewed-on: https://chromium-review.googlesource.com/1067521
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Commit-Queue: Bugs Nash <bugsnash@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560853}
[modify] https://crrev.com/5ad2dabdc5c202853fab082a356526412cc1988d/third_party/blink/tools/update_flaky_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b2bbdf25ee205f8e48cf4f19e5f1ebea00c2f2f7

commit b2bbdf25ee205f8e48cf4f19e5f1ebea00c2f2f7
Author: Bugs Nash <bugsnash@chromium.org>
Date: Wed May 23 02:28:28 2018

Make update_flaky_expectations.py include lines with no passing result.

Make update_flaky_expectations.py script delete lines that do not
fail in the specified way when they don't include any passing results.

Currently expectations that do not fail in the specified way are
removed only when they include a passing result, e.g. the expectation
  bug(test) fast/test.html [ Crash Pass ]
will be removed if the results on the builders show only Passes and
Timeouts because there is no Crash result.

This patch changes the script to also delete the line if the builders
show only Timeouts (and no Passes). This is in the spirit of the test's
description "Scans the TestExpectations file and uses results from
actual builder bots runs to remove tests that are marked as flaky but
don't fail in the specified way."

Bug:  730704 
Change-Id: Ib70b5eb43fb58471ef28c346c9a0f7c334283a13
Reviewed-on: https://chromium-review.googlesource.com/1067525
Commit-Queue: Bugs Nash <bugsnash@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#560901}
[modify] https://crrev.com/b2bbdf25ee205f8e48cf4f19e5f1ebea00c2f2f7/third_party/blink/tools/blinkpy/web_tests/update_flaky_expectations.py
[modify] https://crrev.com/b2bbdf25ee205f8e48cf4f19e5f1ebea00c2f2f7/third_party/blink/tools/blinkpy/web_tests/update_flaky_expectations_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f87b94e8894a4b9c9031b4f95993d869241f0617

commit f87b94e8894a4b9c9031b4f95993d869241f0617
Author: Bugs Nash <bugsnash@chromium.org>
Date: Mon Jun 04 23:48:22 2018

Added feature to remove TestExpectations lines for unexpected passes.

Updated the pre-existing script that removes invalid flakes from
TestExpectations to also remove invalid failures.

Added --type flag to choose functionality. In the default case (no flag)
both flakes and failures and checked. Options for the flag are
{flake,fail,all}.

Renamed script to update_expectations to reflect new functionality.

Bug:  730704 
Change-Id: Iec23363fce7a8f39413a6fe42a1134ea10c614c4
Reviewed-on: https://chromium-review.googlesource.com/863002
Commit-Queue: Bugs Nash <bugsnash@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564300}
[rename] https://crrev.com/f87b94e8894a4b9c9031b4f95993d869241f0617/third_party/blink/tools/blinkpy/web_tests/update_expectations.py
[rename] https://crrev.com/f87b94e8894a4b9c9031b4f95993d869241f0617/third_party/blink/tools/blinkpy/web_tests/update_expectations_unittest.py
[rename] https://crrev.com/f87b94e8894a4b9c9031b4f95993d869241f0617/third_party/blink/tools/update_expectations.py