Is there a page describing how to run these tests locally to reproduce the problem? Where is the test source code? This stack says something is wrong in DevTools, but doesn't help with finding the cause.

Re #2: The tests pass when run locally. AFAIK, they only fail when run in the HW lab, when previously they were passing.

That definitely doesn't make things easier. Could you please point to the source code?

Here's some of the code that runs the failing test. I'm not sure about the devtools side of it.

- example test: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/policy_URLWhitelist/policy_URLWhitelist.py?q=package:chromeos+EnterprisePolicyTest&dr=C&l=26
- telemtry cros, responsible for talking to chrome: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/common_lib/cros/chrome.py?q=package:chromeos+chrome.py&dr
- telemtry chrome: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py

Thanks a lot, that helps.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/be7bba69f100b52764f24e5545356a09f72f540f

commit be7bba69f100b52764f24e5545356a09f72f540f
Author: kozyatinskiy <kozyatinskiy@chromium.org>
Date: Tue Apr 05 06:03:01 2016

[DevTools] Speculative chrome os tests fix

After analysis every return point of V8RuntimAgentImpl::evaluate only one suspicious return was found in InjectedScript::wrapValue.
If function.call returns empty value without exception then we return empty MaybeLocal and return from wrapEvaluateResult without setting errorString.

BUG= 599947 
R=dgozman@chromium.org

Review URL: https://codereview.chromium.org/1862533002

Cr-Commit-Position: refs/heads/master@{#385113}

[modify] https://crrev.com/be7bba69f100b52764f24e5545356a09f72f540f/third_party/WebKit/Source/platform/v8_inspector/InjectedScript.cpp

I took a look into log which was generated by test and found there next Runtime.evaluate call:
04/07 06:45:47.972 DEBUG|inspector_websocke:0097| sent [{
  "id": 8, 
  "method": "Runtime.evaluate", 
  "params": {
    "contextId": 0, 
    "expression": "document.readyState == 'complete' && document.getElementById('Email') != null", 
    "returnByValue": true
  }
}]
It looks like error. If you'd like to evaluate in default context then you can omit contextId argument. If you'd like to evaluate in specific context then first frame has id equals 1 and devtools report context in contextCreated event with correct id.
I've uploaded and sent to commit queue fix of zero contextId processing:
https://codereview.chromium.org/1867903002/. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d3e1f6b795170d0cb6a160f5193ddd2b89c812b5

commit d3e1f6b795170d0cb6a160f5193ddd2b89c812b5
Author: kozyatinskiy <kozyatinskiy@chromium.org>
Date: Thu Apr 07 19:30:44 2016

[DevTools] Fix crash when runtime.evaluate is called with contextId = 0

BUG= 599947 
R=dgozman@chromium.org

Review URL: https://codereview.chromium.org/1867903002

Cr-Commit-Position: refs/heads/master@{#385839}

[add] https://crrev.com/d3e1f6b795170d0cb6a160f5193ddd2b89c812b5/third_party/WebKit/LayoutTests/inspector-protocol/runtime/runtime-evaluate-with-context-id-equal-zero-expected.txt
[add] https://crrev.com/d3e1f6b795170d0cb6a160f5193ddd2b89c812b5/third_party/WebKit/LayoutTests/inspector-protocol/runtime/runtime-evaluate-with-context-id-equal-zero.html
[modify] https://crrev.com/d3e1f6b795170d0cb6a160f5193ddd2b89c812b5/third_party/WebKit/LayoutTests/inspector-protocol/runtime/runtime-evaluate-without-enabling-expected.txt
[modify] https://crrev.com/d3e1f6b795170d0cb6a160f5193ddd2b89c812b5/third_party/WebKit/Source/platform/v8_inspector/V8InspectorSessionImpl.cpp

It looks like this tests are still crashes but for not DevTools reason:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/59839837-chromeos-test/chromeos4-row5-rack2-host1/policy_RestoreOnStartupURLs/sysinfo/iteration.1/var/spool/crash/
Could someone from ChromeOS verify this?

The stack in the link in comment #11 seems to point to a devtools issue. This may be a different issue, though.

 0  chrome!blink::V8InspectorConnectionImpl::findInjectedScript [OwnPtr.h : 61 + 0x4]
 1  chrome!blink::V8RuntimeAgentImpl::evaluate [V8RuntimeAgentImpl.cpp : 97 + 0x1d]
 2  chrome!blink::InspectorRuntimeAgent::evaluate [InspectorRuntimeAgent.cpp : 119 + 0x10]
 3  chrome!blink::protocol::DispatcherImpl::Runtime_evaluate [Dispatcher.cpp : 1630 + 0x58]
 4  chrome!blink::protocol::DispatcherImpl::dispatch [Dispatcher.cpp : 8181 + 0x6]
 5  chrome!blink::WebDevToolsAgentImpl::dispatchMessageFromFrontend [WebDevToolsAgentImpl.cpp : 585 + 0xc]

Alexey: where did you find this error?

https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=policy_RestoreOnStartupURLs&days_back=100&releases=52

Is red. Going to samus I get the one below, but I suspect there is a variety of backtraces?

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/59834183-chromeos-test/chromeos2-row24-rack1-host21/policy_RestoreOnStartupURLs/sysinfo/iteration.1/var/spool/crash/

 3  chrome!logging::LogMessage::~LogMessage [logging.cc : 735 + 0x4]
 4  chrome!chromeos::LoginDisplayHostImpl::RenderProcessGone [login_display_host_impl.cc : 828 + 0x7]
 5  chrome!content::WebContentsImpl::RenderViewTerminated [web_contents_impl.cc : 4028 + 0x1e]
 6  chrome!content::RenderProcessHostImpl::ProcessDied [render_process_host_impl.cc : 2424 + 0x13]
 7  chrome!base::debug::TaskAnnotator::RunTask [callback.h : 397 + 0x7]
 8  chrome!base::MessageLoop::DoWork [message_loop.cc : 479 + 0x15]

Jacob, how can I determine revision of chromium for this test run? I ask because there is no V8InspectorConnectionImpl in ToT codebase.

Achuith, if you asked about #9, I've found it in test's debug log:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/59834183-chromeos-test/chromeos2-row24-rack1-host21/policy_RestoreOnStartupURLs/debug/

re #15, the chrome_version file says "Google Chrome 51.0.2701.0 unknown". Strange that V8InspectorConnectionImpl is not in the codebase, Chrome is probably too old?

Yes, it looks like Chrome was too old and doesn't have crash fix from #10.
Two last runs of tests policy_RestoreOnStartupURLs and policy_EditBookmarksEnabled were failed with:
Unhandled LoginException: Timed out going through login screen. Cryptohome not mounted. OOBE not dismissed.

Last runs were successful according https://wmatrix.googleplex.com/unfiltered?days_back=14&tests=policy_.*
Closed as fixed, feel free to reassign it back if it's still reproducible.

About half are failing again, and all on some devices, in M52: https://wmatrix.googleplex.com/unfiltered?tests=policy_.%2A&days_back=4&releases=52

There are following errors:
Unhandled LoginException: Timed out going through login screen. Cryptohome not mounted. OOBE not dismissed.
Image should be blocked.

It doesn't look like DevTools related issue. DevTools related message in test_name/debug/*.DEBUG folder looks good. Problem from #12 was fixed in #7. Please reassign it to someone more experienced with ChromeOS tests or could you provide information how to run this tests locally.

That error message sounds like  issue 582615 , but I can't find any screenshots in the results folder to confirm. I expect to land the fixes for  issue 582615  today or tomorrow.