New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 621473 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Linux , Mac
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Fix crash and re-enable page_cycler_v2.typical_25 on chromium.perf for Mac Retina Perf (1) and Linux Perf (1)

Project Member Reported by oysteine@chromium.org, Jun 20 2016

Issue description

https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Retina%20Perf%20%281%29/builds/3180
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Linux%20Perf%20%281%29/builds/12045

Link to failing step log: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Linux%20Perf%20%281%29/builds/12045/steps/page_cycler_v2.typical_25/logs/stdio


If the test is disabled, please downgrade to Pri-2.

	Thread 0 (crashed)
	 0  chrome!<name omitted> [PartitionAlloc.h : 628 + 0x0]
	    rax = 0x0000020e86601020   rdx = 0x0000000000000026
	    rcx = 0x0000000000000000   rbx = 0x000000000000000e
	    rsi = 0x0000000000000006   rdi = 0x00007ffc242dd298
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd250
	     r8 = 0x0000000000000052    r9 = 0x00007fd170ee383d
	    r10 = 0x00007fd170133b82   r11 = 0x00007fd164615c70
	    r12 = 0x0000000000000002   r13 = 0x00007fd170f09120
	    r14 = 0x00007ffc242dd2a0   r15 = 0x00007ffc242dd298
	    rip = 0x00007fd16ef4670e
	    Found by: given as instruction pointer in context
	 1  chrome!<name omitted> [StringImpl.cpp : 406 + 0x7]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd290
	    rip = 0x00007fd16ef4704c
	    Found by: stack scanning
	 2  chrome!<name omitted> [WTFString.cpp : 64 + 0x5]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd2d0
	    rip = 0x00007fd16ef54977
	    Found by: stack scanning
	 3  chrome!<name omitted> [WebString.cpp : 108 + 0x5]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd2f0
	    rip = 0x00007fd16ef389d2
	    Found by: stack scanning
	 4  chrome!content::WebURLLoaderImpl::PopulateURLResponse [WebString.h : 104 + 0x8]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd310
	    rip = 0x00007fd16d0e66d5
	    Found by: stack scanning
	 5  chrome!<name omitted> [trace_log.cc : 1291 + 0x54]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd390
	    rip = 0x00007fd16e172588
	    Found by: stack scanning
	 6  chrome!_fini + 0x20750f
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd398
	    rip = 0x00007fd1701231ab
	    Found by: stack scanning
	 7  chrome!<name omitted> [thread_local.h : 101 + 0x7]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd3e0
	    rip = 0x00007fd16e172813
	    Found by: stack scanning
	 8  chrome!<name omitted> [thread_local.h : 97 + 0x5]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd450
	    rip = 0x00007fd16e1742f2
	    Found by: stack scanning
	 9  chrome!tcmalloc::ThreadCache::FetchFromCentralCache [thread_cache.h : 218 + 0x8]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd4b0
	    rip = 0x00007fd16b5ab16d
	    Found by: stack scanning
	10  libc-2.19.so + 0x4e029
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd4b8
	    rip = 0x00007fd1644dc029
	    Found by: stack scanning
	11  chrome!_fini + 0xc4908
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd4c0
	    rip = 0x00007fd16ffe05a4
	    Found by: stack scanning
	12  chrome!_fini + 0xc4903
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd4c8
	    rip = 0x00007fd16ffe059f
	    Found by: stack scanning
	13  libc-2.19.so + 0x4a4ea
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd4d0
	    rip = 0x00007fd1644d84ea
	    Found by: stack scanning
	14  chrome!<name omitted> [char_traits.h : 257 + 0x5]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd4e0
	    rip = 0x00007fd16bcb99d8
	    Found by: stack scanning
	15  chrome + 0xb3fe00
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd590
	    rip = 0x00007fd16b5a3e00
	    Found by: stack scanning
	16  chrome!content::WebURLLoaderImpl::Context::OnReceivedResponse [web_url_loader_impl.cc : 626 + 0x16]
	    rbp = 0xffffffff00000000   rsp = 0x00007ffc242dd680
	    rip = 0x00007fd16d0e4f06
	    Found by: stack scanning
 
Cc: lukasza@chromium.org
Summary: page_cycler_v2.typical_25 crash on chromium.perf for Mac Retina Perf (1) and Linux Perf (1) (was: page_cycler_v2.typical_25 failure on chromium.perf for Mac Retina Perf (1) and Linux Perf (1))
https://codereview.chromium.org/2004653002 looks like a possible culprit. Going to try a bisect.
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Jun 20 2016


===== BISECT JOB RESULTS =====
Status: completed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N  Good?
chromium@400441  1          N/A      5  good
chromium@400460  1          N/A      5  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 621473

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.typical_25
Test Metric: pcv1-cold-firstContentfulPaint_avg/http___www.barnesandnoble.com_u_books-bestselling-books_379003057_
Relative Change: 0.00%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1344
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9009336471070051472


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5782990127366144

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Succeeded, but I forgot to enter the bug ID. Pasting result manually:

@@@STEP_LOG_LINE@json.output@{@@@
@@@STEP_LOG_LINE@json.output@  "post_data": {@@@
@@@STEP_LOG_LINE@json.output@    "aborted_reason": null, @@@
@@@STEP_LOG_LINE@json.output@    "bad_revision": "bd82f922853ea6e61c4d907f2b51786f3ed9f5a2", @@@
@@@STEP_LOG_LINE@json.output@    "bisect_bot": "linux_perf_bisect", @@@
@@@STEP_LOG_LINE@json.output@    "bug_id": -1, @@@
@@@STEP_LOG_LINE@json.output@    "buildbot_log_url": "http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6540", @@@
@@@STEP_LOG_LINE@json.output@    "change": "100.00%", @@@
@@@STEP_LOG_LINE@json.output@    "command": "src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.typical_25", @@@
@@@STEP_LOG_LINE@json.output@    "culprit_data": {@@@
@@@STEP_LOG_LINE@json.output@      "author": "lukasza", @@@
@@@STEP_LOG_LINE@json.output@      "cl": "bd82f922853ea6e61c4d907f2b51786f3ed9f5a2", @@@
@@@STEP_LOG_LINE@json.output@      "cl_date": "Fri Jun 17 18:23:17 2016", @@@
@@@STEP_LOG_LINE@json.output@      "commit_info": "\nRenderer needs to access files from HTTP POST (i.e. from XSSAuditor).\nThis CL makes sure that file access is preserved across xsite transfers.\n\nBUG=101395, 613260\nCQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_site_isolation\n\nReview-Url: https://codereview.chromium.org/2062523002\nCr-Commit-Position: refs/heads/master@{#400460}", @@@
@@@STEP_LOG_LINE@json.output@      "email": "lukasza@chromium.org", @@@
@@@STEP_LOG_LINE@json.output@      "revisions_links": [], @@@
@@@STEP_LOG_LINE@json.output@      "subject": "Fixing renderer's access to a file from HTTP POST (after a xsite transfer)."@@@
@@@STEP_LOG_LINE@json.output@    }, @@@
@@@STEP_LOG_LINE@json.output@    "good_revision": "675042893f1c9297277b8b36d193a6d1a98d26a4", @@@
@@@STEP_LOG_LINE@json.output@    "metric": "pcv1-warm-firstContentfulPaint_avg/http___www.barnesandnoble.com_u_books-bestselling-books_379003057_", @@@
@@@STEP_LOG_LINE@json.output@    "revision_data": [@@@
@@@STEP_LOG_LINE@json.output@      {@@@
@@@STEP_LOG_LINE@json.output@        "commit_hash": "675042893f1c9297277b8b36d193a6d1a98d26a4", @@@
@@@STEP_LOG_LINE@json.output@        "depot_name": "chromium", @@@
@@@STEP_LOG_LINE@json.output@        "mean_value": 1, @@@
@@@STEP_LOG_LINE@json.output@        "result": "good", @@@
@@@STEP_LOG_LINE@json.output@        "revision_string": "chromium@400441", @@@
@@@STEP_LOG_LINE@json.output@        "std_dev": null, @@@
@@@STEP_LOG_LINE@json.output@        "values": [@@@
@@@STEP_LOG_LINE@json.output@          2, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          2@@@
@@@STEP_LOG_LINE@json.output@        ]@@@
@@@STEP_LOG_LINE@json.output@      }, @@@
@@@STEP_LOG_LINE@json.output@      {@@@
@@@STEP_LOG_LINE@json.output@        "commit_hash": "c40e88210fdf15e85a553573584d217d6f8273b9", @@@
@@@STEP_LOG_LINE@json.output@        "depot_name": "chromium", @@@
@@@STEP_LOG_LINE@json.output@        "mean_value": 1, @@@
@@@STEP_LOG_LINE@json.output@        "result": "good", @@@
@@@STEP_LOG_LINE@json.output@        "revision_string": "chromium@400451", @@@
@@@STEP_LOG_LINE@json.output@        "std_dev": null, @@@
@@@STEP_LOG_LINE@json.output@        "values": [@@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0@@@
@@@STEP_LOG_LINE@json.output@        ]@@@
@@@STEP_LOG_LINE@json.output@      }, @@@
@@@STEP_LOG_LINE@json.output@      {@@@
@@@STEP_LOG_LINE@json.output@        "commit_hash": "94759c2202550721d3e49d7f7869e2952954748e", @@@
@@@STEP_LOG_LINE@json.output@        "depot_name": "chromium", @@@
@@@STEP_LOG_LINE@json.output@        "mean_value": 1, @@@
@@@STEP_LOG_LINE@json.output@        "result": "good", @@@
@@@STEP_LOG_LINE@json.output@        "revision_string": "chromium@400456", @@@
@@@STEP_LOG_LINE@json.output@        "std_dev": null, @@@
@@@STEP_LOG_LINE@json.output@        "values": [@@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          3, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          2, @@@
@@@STEP_LOG_LINE@json.output@          0@@@
@@@STEP_LOG_LINE@json.output@        ]@@@
@@@STEP_LOG_LINE@json.output@      }, @@@
@@@STEP_LOG_LINE@json.output@      {@@@
@@@STEP_LOG_LINE@json.output@        "commit_hash": "0df81511b511477d0c1990ff7e9b213ed14099b8", @@@
@@@STEP_LOG_LINE@json.output@        "depot_name": "chromium", @@@
@@@STEP_LOG_LINE@json.output@        "mean_value": 1, @@@
@@@STEP_LOG_LINE@json.output@        "result": "good", @@@
@@@STEP_LOG_LINE@json.output@        "revision_string": "chromium@400458", @@@
@@@STEP_LOG_LINE@json.output@        "std_dev": null, @@@
@@@STEP_LOG_LINE@json.output@        "values": [@@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0@@@
@@@STEP_LOG_LINE@json.output@        ]@@@
@@@STEP_LOG_LINE@json.output@      }, @@@
@@@STEP_LOG_LINE@json.output@      {@@@
@@@STEP_LOG_LINE@json.output@        "commit_hash": "80c06469b872723c27bf453b3f6ee8b06f747b27", @@@
@@@STEP_LOG_LINE@json.output@        "depot_name": "chromium", @@@
@@@STEP_LOG_LINE@json.output@        "mean_value": 1, @@@
@@@STEP_LOG_LINE@json.output@        "result": "good", @@@
@@@STEP_LOG_LINE@json.output@        "revision_string": "chromium@400459", @@@
@@@STEP_LOG_LINE@json.output@        "std_dev": null, @@@
@@@STEP_LOG_LINE@json.output@        "values": [@@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          3, @@@
@@@STEP_LOG_LINE@json.output@          2, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          1, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          2, @@@
@@@STEP_LOG_LINE@json.output@          1@@@
@@@STEP_LOG_LINE@json.output@        ]@@@
@@@STEP_LOG_LINE@json.output@      }, @@@
@@@STEP_LOG_LINE@json.output@      {@@@
@@@STEP_LOG_LINE@json.output@        "commit_hash": "bd82f922853ea6e61c4d907f2b51786f3ed9f5a2", @@@
@@@STEP_LOG_LINE@json.output@        "depot_name": "chromium", @@@
@@@STEP_LOG_LINE@json.output@        "mean_value": 0, @@@
@@@STEP_LOG_LINE@json.output@        "result": "bad", @@@
@@@STEP_LOG_LINE@json.output@        "revision_string": "chromium@400460", @@@
@@@STEP_LOG_LINE@json.output@        "std_dev": null, @@@
@@@STEP_LOG_LINE@json.output@        "values": [@@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0, @@@
@@@STEP_LOG_LINE@json.output@          0@@@
@@@STEP_LOG_LINE@json.output@        ]@@@
@@@STEP_LOG_LINE@json.output@      }@@@
@@@STEP_LOG_LINE@json.output@    ], @@@
@@@STEP_LOG_LINE@json.output@    "score": 98.0, @@@
@@@STEP_LOG_LINE@json.output@    "status": "completed", @@@
@@@STEP_LOG_LINE@json.output@    "test_type": "return_code", @@@
@@@STEP_LOG_LINE@json.output@    "try_job_id": 5871724151701504, @@@
@@@STEP_LOG_LINE@json.output@    "warnings": []@@@
@@@STEP_LOG_LINE@json.output@  }, @@@
@@@STEP_LOG_LINE@json.output@  "status_code": 200, @@@
@@@STEP_LOG_LINE@json.output@  "text": ""@@@
@@@STEP_LOG_LINE@json.output@}@@@
@@@STEP_LOG_END@json.output@@@
Cc: -lukasza@chromium.org
Owner: lukasza@chromium.org
Bisect points at https://codereview.chromium.org/2062523002 which apparently can't be cleanly reverted; disabling test and assigning to lukasza@chromium.org
Disabling test in https://codereview.chromium.org/2080223004
Project Member

Comment 8 by bugdroid1@chromium.org, Jun 21 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ea983052d8712ee0668fd4ac37e8301fa63d0a76

commit ea983052d8712ee0668fd4ac37e8301fa63d0a76
Author: oysteine <oysteine@chromium.org>
Date: Tue Jun 21 13:56:29 2016

Disabled page_cycler_v2 perf benchmark on linux and mac until crash is fixed

TBR=nednguyen
BUG= 621473 

Review-Url: https://codereview.chromium.org/2080223004
Cr-Commit-Position: refs/heads/master@{#400982}

[modify] https://crrev.com/ea983052d8712ee0668fd4ac37e8301fa63d0a76/tools/perf/benchmarks/page_cycler_v2.py

Cc: ksakamoto@chromium.org kouhei@chromium.org
Not sure why this only fails on page_cycler_v2 but not page_cycler_v1?
No repro when trying:
  $ DISPLAY=:20 tools/perf/run_benchmark run page_cycler_v2.typical_25 --browser-executable=$HOME/src/chromium4/src/out/gn_no_asan/chrome 2>&1 | tee $HOME/scratch/perf.out
with:
  $ cat out/gn_no_asan/args.gn 
  ...
  dcheck_always_on = true
  is_component_build = true
  is_debug = false
  use_goma = true

Can somebody familiar with perf/telemetry tests/bots confirm that these are the right repro steps?

Looking at the last 200 builds of Linux Perf (1) bot at https://uberchromegw.corp.google.com/i/chromium.perf/builders/Linux%20Perf%20%281%29?numbuilds=200, I see that

1) the first red build referencing "page_cycler_v2.typical_25" in the status line is:

Jun 17 11:45	9b55ff2e61b462dfb0c9b2ee86d43ea26df9a135	failure	#12045
Failed steps failed page_cycler_v2.typical_25 

The commit blamed in #c6 is not present in build 12045 - it appears on the blamelist only in build 12046.

Similarily, git shows that the commit referenced by the build has landed on Fri Jun 17 11:02:49 2016 -0700.  And git shows that the commit found by bisecting in #c6 has landed on Fri Jun 17 11:20:57 2016 -0700 (i.e. this commit was not part of the first red build and therefore cannot be blamed for the red build).

2) There were 25 red builds since the first red build (12045).  So the failure doesn't repro consistently - i.e. only in 25 out of 52 builds.  This means that results of an automated bisect have to be taken with a big dose of skepticism.

Cc: lukasza@chromium.org
Owner: ----
Based on #c11 I think https://codereview.chromium.org/2080223004 is not the root cause - removing myself from the owner field.  Not sure how to find the right owner given that bisecting a flaky failure is difficult to perform automatically.
Re: 1) Yeah you're right, sorry; I just reused the revision range of the Mac crash when I re-ran the bisect job on the linux bot.

Re: 2) That's why the bisect runs each revision a number of times (10 times in this case), to get higher certainty with flaky tests. The results were pretty straightforward, which is odd given the mismatch. We might be dealing with several crashes.

I'll relaunch another bisect with more tries and a better range.
Labels: -Pri-1 Pri-2
Summary: Fix crash and re-enable page_cycler_v2.typical_25 on chromium.perf for Mac Retina Perf (1) and Linux Perf (1) (was: page_cycler_v2.typical_25 crash on chromium.perf for Mac Retina Perf (1) and Linux Perf (1))
Labels: -Pri-2 Pri-1
Owner: lukasza@chromium.org
Status: Assigned (was: Untriaged)
lukasza: can you revert your CL or fix the problem? It's critical for us to have this benchmark running.
Cc: creis@chromium.org
Owner: nednguyen@chromium.org
nedguyen@: my CL cannot be the cause of the problem, because the problem started before my CL has landed.  My CL is first present in build 12046 (i.e. my CL appears on the blamelist of build 12046), but the first red build is 12045.  I tried to explain this in comment #c11 above.
When I say "my CL", I mean the CL that was pointed out in comment #c4 above.  If you think one of my other CLs is the cause of the problem, then please help me understand why you think this is the case.
I think this started flaking since we added additional tracing categories in PCv2, which was first landed in build #11952 [1], reverted in #11958 [2], and re-landed in #12044 [3].

All crashes except #12045 are happening inside IdentifiersFactory::addProcessIdPrefixTo, which is called from Inspector*Event only when devtools.timeline category is enabled. That explains why this fails only on PCv2.

[1] https://codereview.chromium.org/2049593002
[2] https://codereview.chromium.org/2057413002
[3] https://codereview.chromium.org/2070773002

Example stacktrace (https://uberchromegw.corp.google.com/i/chromium.perf/builders/Linux%20Perf%20%281%29/builds/12095/steps/page_cycler_v2.typical_25/logs/stdio):

	Crash reason:  SIGSEGV
	Crash address: 0x2830deffc000
	Process uptime: not available
	
	Thread 0 (crashed)
	 0  chrome!<name omitted> [StringConcatenate.cpp : 128 + 0x3b]
	    rax = 0x00002830dee14770   rdx = 0x00000000001e7880
	    rcx = 0xfffffffff9774970   rbx = 0x00007ffe43095548
	    rsi = 0x0000383b9380400c   rdi = 0x0000000000000005
	    rbp = 0x00002830dee14770   rsp = 0x00007ffe430954d8
	     r8 = 0x0000000068a9e1d0    r9 = 0x0000000068a9e1de
	    r10 = 0x0000000068a9e1de   r11 = 0x00000000068a9e1c
	    r12 = 0x00002ceae2f1b928   r13 = 0x00001654ea39b500
	    r14 = 0x0000383b9380400c   r15 = 0x0000383b93804000
	    rip = 0x00007f38b8f87eb0
	    Found by: given as instruction pointer in context
	 1  chrome!WTF::makeString<WTF::String, WTF::AtomicString> [StringConcatenate.h : 334 + 0x8]
	    rsp = 0x00007ffe430954e0   rip = 0x00007f38b71c6607
	    Found by: stack scanning
	 2  chrome!WTF::StringAppend<WTF::String, WTF::String>::operator String [StringOperators.h : 63 + 0x5]
	    rsp = 0x00007ffe43095530   rip = 0x00007f38b71c64cb
	    Found by: stack scanning
	 3  chrome!blink::IdentifiersFactory::addProcessIdPrefixTo [IdentifiersFactory.cpp : 115 + 0x8]
	    rsp = 0x00007ffe43095560   rip = 0x00007f38b788de36
	    Found by: stack scanning
	 4  chrome!blink::IdentifiersFactory::requestId [IdentifiersFactory.cpp : 72 + 0x8]
	    rsp = 0x00007ffe430955a0   rip = 0x00007f38b788deb1
	    Found by: stack scanning
	 5  chrome!blink::InspectorResourceFinishEvent::data [InspectorTraceEvents.cpp : 468 + 0x8]
	    rsp = 0x00007ffe430955b0   rip = 0x00007f38b78dee0f
	    Found by: stack scanning
	 6  chrome!<name omitted> [ProgressTracker.cpp : 279 + 0x8]
	    rsp = 0x00007ffe430955c0   rip = 0x00007f38b79150a0
	    Found by: stack scanning
	 7  chrome!blink::FrameFetchContext::dispatchDidFinishLoading [FrameFetchContext.cpp : 360 + 0x10]
	    rsp = 0x00007ffe430955e0   rip = 0x00007f38b78ff53f
	    Found by: stack scanning
	 8  chrome!<name omitted> [unique_ptr.h : 169 + 0x7]
	    rsp = 0x00007ffe43095600   rip = 0x00007f38b977123f
	    Found by: stack scanning
	 9  chrome!<name omitted> [ResourceRequest.h : 66 + 0x5]
	    rsp = 0x00007ffe43095610   rip = 0x00007f38b74a9170
	    Found by: stack scanning
	10  chrome!<name omitted> [ResourceFetcher.cpp : 924 + 0xb]
	    rsp = 0x00007ffe43095650   rip = 0x00007f38b781cef3

Cc: caseq@chromium.org
ksakamoto: Thanks, that sounds very plausible!

+caseq: Any ideas about this crash?
Cc: -ksakamoto@chromium.org nednguyen@chromium.org
Owner: ksakamoto@chromium.org
I think now I understand the reason.
IdentifiersFactory has a static WTF::String for process ID prefix [1]. It can be accidentally deleted, as a consequence of racy access to its refcount from main thread and ScriptStreamer thread [2].

[1] https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/inspector/IdentifiersFactory.cpp?rcl=0&l=45
[2] https://cs.chromium.org/chromium/src/third_party/WebKit/Source/bindings/core/v8/ScriptStreamerThread.cpp?rcl=0&l=83

 Issue 612282  looks like the same bug, judging from the stacktrace.

Created a patch: https://codereview.chromium.org/2088333003/

Project Member

Comment 23 by bugdroid1@chromium.org, Jun 24 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0751df2ff806e1ed303c2f652a090057d6768ba1

commit 0751df2ff806e1ed303c2f652a090057d6768ba1
Author: ksakamoto <ksakamoto@chromium.org>
Date: Fri Jun 24 01:48:19 2016

Make IdentifiersFactory thread-safe

IdentifiersFactory can be used not only from main thread but also from
ScriptStreamer thread, so it should not hold process ID prefix as
thread-unsafe WTF::String. This fixes crash when devtools.timeline
tracing is enabled.

This patch also makes s_lastUsedIdentifier increment atomically.

BUG= 621473 ,  612282 

Review-Url: https://codereview.chromium.org/2088333003
Cr-Commit-Position: refs/heads/master@{#401778}

[modify] https://crrev.com/0751df2ff806e1ed303c2f652a090057d6768ba1/third_party/WebKit/Source/core/inspector/IdentifiersFactory.cpp
[modify] https://crrev.com/0751df2ff806e1ed303c2f652a090057d6768ba1/third_party/WebKit/Source/core/inspector/IdentifiersFactory.h

Project Member

Comment 24 by bugdroid1@chromium.org, Jun 24 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1eff267c107b3a0d1c7411df51d28125b0a0212a

commit 1eff267c107b3a0d1c7411df51d28125b0a0212a
Author: kouhei <kouhei@chromium.org>
Date: Fri Jun 24 05:19:50 2016

[PCv2] Re-enable on desktops

BUG= 615178 ,  621473 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq

Review-Url: https://codereview.chromium.org/2091183002
Cr-Commit-Position: refs/heads/master@{#401818}

[modify] https://crrev.com/1eff267c107b3a0d1c7411df51d28125b0a0212a/tools/perf/benchmarks/page_cycler_v2.py

Status: Fixed (was: Assigned)

Sign in to add a comment