The only similar failure I see in the past is  issue 568120 , which suggests that a big increase in trace entries can perhaps cause these sorts of 'Memory Error' failures (and maybe the locked file error is a red-herring?).

There's nothing obviously related in the v8 changelog: https://chromium.googlesource.com/v8/v8/+log/1230ed3c..10de065e

I don't think the perf buildbots bisect down into v8 rolls, do they?

ulan: as the benchmark owner, can you please investigate?  Alternately https://github.com/v8/v8/wiki/Triaging%20issues says we can assign perf issues to bmeurer or mvstanton.

This is apparently affecting only a single test, so I think it's OK to take a day or two to investigate.  But if a solution isn't clear in the next couple days we should disable the test.

> I don't think the perf buildbots bisect down into v8 rolls, do they?

It does bisect into v8 rolls. Did you try kicking off a return_code bisect?

This might be caused by https://chromium.googlesource.com/v8/v8/+/567e58390d86c54ebd527d1b95500c415f3caeb5

I think this increases the number of v8 events in the trace.

We'll try to find a fix tomorrow.

> It does bisect into v8 rolls. Did you try kicking off a return_code bisect?

Oh, cool - thanks!  I didn't kick one off for this build since there was only this single chromium CL in the range.  Sounds like ulan has this issue in-hand, but I'm happy to kick off a bisect if that'll help.

I would like to confirm my theory, how can I kick off a return_code bisect?

ulan: it's a little confusing right now since bisect is set up for perf regressions by default, and the perf dashboard doesn't know the revision of the failed build because it failed and didn't upload any data. I'll kick one off but here are the  steps:

1) Go to the graph for the perf test. You can find it from the "Results Dashboard" link on the buildbot status page (https://build.chromium.org/p/chromium.perf/builders/Win%207%20ATI%20GPU%20Perf%20%282%29/builds/4538). The graph for this one is https://chromeperf.appspot.com/report?sid=3e438e03ef4d01a8e6020c55bc90e57b75ef56bfe12ad7bf7a89384e9069eace&rev=378420
2) Click the last data point (assuming the test hasn't been fixed/isn't sending new data) and click "Bisect" in the tooltip that comes up.
3) In the dialog, change:
  Bug ID: type the bug number you want the bisect to report back to, like 590761
  Earlier revision: Last revision with data. Because the bisect is set up to check for a perf regression at this point, it'll confusingly have this number in the "later revision" field. Move it to earlier revision (in this case, it should be 378154)
  Later revision: This should be the commit pos of the last revision in the first failed build (378155 in our case)
  Bisect mode: return_code

The bisect I kicked off is: https://chromeperf.appspot.com/buildbucket_job_status/9019382868324819216

Thanks for the detailed instructions, Annie!

The bisect will be very helpful because manual run before and after the suspected V8 CL shows no difference in number of trace events.

 Issue 591081  has been merged into this issue.

Note Ethan is disabling the test in https://codereview.chromium.org/1752613003/ until it can be fixed

=== Auto-CCing suspected CL author fmeawad@chromium.org ===

Hi fmeawad@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Reland: Add Scoped Context Info (Isolate) to V8 Traces
Author  : fmeawad
Commit description:
  
This patch adds the newly added support for contexts in V8 Tracing, as well
as use it to mark all the entry points for a V8 Isolate.

Update for reland: The current tracing interface needs to be updated (AddTraceEvent),
but the embedders need to migrate to the new version before removing the old version.
(Reland of: https://codereview.chromium.org/1686233002)

The revert happened because the 2 signatures of the old and new AddTraceEvent where different
so it threw an overload-virtual error on cross arm debug. This issue is temporary, and to solve
it, I added an implementation of the old and new everywhere until the embedder implements the new.

BUG=v8:4565
LOG=N

R=jochen@chromium.org

Review URL: https://codereview.chromium.org/1704253002

Cr-Commit-Position: refs/heads/master@{#34332}
Commit  : 567e58390d86c54ebd527d1b95500c415f3caeb5
Date    : Fri Feb 26 17:25:30 2016


===== TESTED REVISIONS =====
Revision                Exit Code   Std. Dev.   Num Values  Good?
chromium@378154         0           N/A         5           good
chromium@378154,v8@6a7e8661340           N/A         5           good
chromium@378154,v8@a0fdb33f1c0           N/A         5           good
chromium@378154,v8@23cf65926e0           N/A         5           good
chromium@378154,v8@567e58390d1           N/A         5           bad
chromium@378154,v8@239ed8ffa81           N/A         5           bad
chromium@378155         1           N/A         5           bad

Bisect job ran on: winx64ati_perf_bisect
Bug ID: 590761

Test Command: python src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --also-run-disabled-tests oortonline_tbm
Test Metric: Running-frame_time_discrepancy/Running-frame_time_discrepancy
Relative Change: Zero to non-zero
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64ati_perf_bisect/builds/1255
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9019382868324819216


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!

I have removed the offending portion of the CL in https://codereview.chromium.org/1807433002/, I will re-enable the benchmark after the v8 roll.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ce376a92665ed53ae2a75507c2daf9cf7d184be4

commit ce376a92665ed53ae2a75507c2daf9cf7d184be4
Author: fmeawad <fmeawad@chromium.org>
Date: Wed Mar 16 18:48:49 2016

Re-enable oortonline benchmark on windows

The crash was due to an excessive amount of trace events logged due to
direct calls from webgl into v8 primitives. The trace events on the
v8 primitives have been removed.

BUG= 590761 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review URL: https://codereview.chromium.org/1805403003

Cr-Commit-Position: refs/heads/master@{#381505}

[modify] https://crrev.com/ce376a92665ed53ae2a75507c2daf9cf7d184be4/tools/perf/benchmarks/oortonline.py