Thanks for filing this bug, John!

I see that you listed yourself as the owner. Feel free to fix it if you have time, but if you want to stay focused on the TBMv2 migration so that we can hit your KR, then I can take this bug instead. Feel free to assign it to me if you want.

It does give an error message when playback fails:

(ERROR) 2017-04-27 17:07:24,091 media_action.HasEventCompletedOrError:45  Detected media error while waiting for playing: Media error: error, code:4

There is an error message for anyone who looks at the output, however automated runs of the benchmark think it's successful.

Correct. Sorry, I didn't mean to imply that this wasn't a good bug. I was just noting the output line to help with my future investigation.

This is blocked on https://github.com/catapult-project/catapult/issues/3540

I heard back from the catapult bug. example command to run unit tests: 
./bin/run_tests --browser=system PlayActionTest.testPlayWithNoSelector --also-run-disabled-tests

I will start working on this again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b2a146b94632b92acd0eddbd19da0014c6720b4c

commit b2a146b94632b92acd0eddbd19da0014c6720b4c
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Tue Dec 19 18:14:51 2017

Remove broken test case.

Discovered that this was broken while fixing  crbug.com/713394 .

Bug:  713394 
Change-Id: I4fc12ebc4864872ba09674f614f35d0ad8064490
Reviewed-on: https://chromium-review.googlesource.com/833403
Reviewed-by: John Chen <johnchen@chromium.org>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Caleb Rouleau <crouleau@chromium.org>
Cr-Commit-Position: refs/heads/master@{#525082}
[modify] https://crrev.com/b2a146b94632b92acd0eddbd19da0014c6720b4c/tools/perf/page_sets/media_cases.py

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/788d0fd19737ac55103321922dce69153a46999a

commit 788d0fd19737ac55103321922dce69153a46999a
Author: Caleb Rouleau <crouleau@chromium.org>
Date: Tue Dec 19 18:22:29 2017

[Telemetry] Error when media playback fails.

This change uncovered failing tests such as detailed in
https://chromium-review.googlesource.com/c/chromium/src/+/833403

Bug:  chromium:713394 
Change-Id: I85102ddab2f866327866ce2dda1dd56ad96ead5a
Reviewed-on: https://chromium-review.googlesource.com/832921
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Commit-Queue: Caleb Rouleau <crouleau@chromium.org>

[modify] https://crrev.com/788d0fd19737ac55103321922dce69153a46999a/telemetry/telemetry/internal/actions/media_action.py