New issue
Advanced search Search tips

Issue 764561 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Mac
Pri: 1
Type: ----



Sign in to add a comment

nytimes story seems to be broken

Project Member Reported by martiniss@chromium.org, Sep 13 2017

Issue description

system_health.memory_desktop failing on 6 builders

Builders failed on: 
- Linux Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf
- Mac Pro 10.11 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Mac%20Pro%2010.11%20Perf
- Mac Retina Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Mac%20Retina%20Perf
- Win 10 High-DPI Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%2010%20High-DPI%20Perf
- Win 10 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf
- Win 7 Nvidia GPU Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20Nvidia%20GPU%20Perf


v8.runtimestats.browsing_desktop failing on 6 builders

Builders failed on: 
- Linux Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf
- Win 10 High-DPI Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%2010%20High-DPI%20Perf
- Win 10 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf
- Win 7 ATI GPU Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20ATI%20GPU%20Perf
- Win 7 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%207%20Perf
- Win 8 Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Win%208%20Perf

Looks like nytimes story is failing on most of these platforms. I'm not sure if I should disable or not; can someone take a look?


Some example tasks
https://chromium-swarm.appspot.com/task?id=388f6f26de2c1910&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=388d3b819a6a2a10&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=388f3e9843ae8210&refresh=10&show_raw=1
 
It's a crash. Luckily we have it symbolized on Windows:
https://chromium-swarm.appspot.com/task?id=388f3e4849cc8610&refresh=10&show_raw=1


********************************************************************************
	Last event: 728.988: Access violation - code c0000005 (first/second chance not available)
	  debugger time: Tue Sep 12 11:19:55.424 2017 (UTC - 7:00)
	RetAddr           : Args to Child                                                           : Call Site
	00007ffa`2d6f2e4a : 00000093`c28f4280 000000a0`cb72c6f0 00000093`c28f4280 00000000`363280b8 : chrome_child!blink::TaskRunnerHelper::Get+0xf
	00007ffa`2d6f2dc2 : 000000a0`d4311700 00000093`c28f4280 000000a0`cb72c629 000000a0`cb72c6f0 : chrome_child!blink::FontFace::GetTaskRunner+0x2a
	00007ffa`2da68e12 : 00000000`00000000 000000a0`cb72c6f0 000000a0`d4311700 00007ffa`2da67fbc : chrome_child!blink::FontFace::SetLoadStatus+0x292
	00007ffa`2da68c9d : 00000000`00000000 000000a0`cb72c900 00000000`00000000 00007ffa`2da67e3a : chrome_child!blink::CSSFontFace::Load+0x142
	00007ffa`2da689b7 : 000003f4`b11342a0 00000001`00000004 000000a0`d8580070 000000a0`cb72cbf0 : chrome_child!blink::CSSFontFace::Load+0x8d
	00007ffa`2d71dd10 : 00000093`c28f52b8 00000000`00000000 00000000`00000000 00000093`c28f52e0 : chrome_child!blink::CSSFontFace::FontLoaded+0x67
	00007ffa`2de8ff30 : 00000000`00000000 00000093`c28f52e0 00000093`c28f52e0 000002f2`db770230 : chrome_child!blink::RemoteFontFaceSource::NotifyFinished+0x2d0
	00007ffa`2de9579e : 00000000`35fd9800 000002f2`db770230 000002f2`db770230 00007ffa`2f238b65 : chrome_child!blink::Resource::NotifyFinished+0x58
	00007ffa`2de9d08f : 00000000`35fd98c0 00000093`c28f4b38 000002b2`c37ace50 000003f4`b106e950 : chrome_child!blink::ResourceFetcher::HandleLoaderError+0xbe
	00007ffa`2de9ce7b : 00007ffa`30238700 00007ffa`30238700 000000a0`cb72c8e8 000000a0`d4218e20 : chrome_child!blink::ResourceLoader::HandleError+0xc3
	00007ffa`2de9ca4c : 00000253`5993fb20 000000a0`d4311660 000000a0`d417e570 00000000`00000004 : chrome_child!blink::ResourceLoader::DidReceiveResponse+0x427
	00007ffa`2f23a749 : 00000000`00000000 00000000`00000001 00007ffa`2ffc9f68 00007ffa`304d9e80 : chrome_child!blink::ResourceLoader::DidReceiveResponse+0x14
	00007ffa`2e70394b : 000000a0`d8ac3c20 000000a0`cc12d960 000000a0`d8ac3c20 00007ffa`2e1254b6 : chrome_child!content::WebURLLoaderImpl::Context::OnReceivedResponse+0x585
	00007ffa`2e70ba2a : 000000a0`d85d7df0 00007ffa`2e0ce53e 00000000`00000000 000000a0`d85d7df0 : chrome_child!content::ResourceDispatcher::OnReceivedResponse+0x1df
	00007ffa`2dc7791d : 000000a0`cb72d6a0 000000a0`cc1838e0 000000a0`cb72d670 000000a0`cb72dc60 : chrome_child!content::URLLoaderClientImpl::OnReceiveResponse+0x7e
	00007ffa`2dac8f68 : 000000a0`d85817e8 000000a0`d85817e8 000000a0`cb72d6a0 000000a0`cb72dc60 : chrome_child!content::ThrottlingURLLoader::OnReceiveResponse+0x10d
	00007ffa`2dfd9a64 : 00000000`00000020 000000a0`d85819f4 000000a0`d81f1cf8 000000a0`cb72dc60 : chrome_child!content::mojom::URLLoaderClientStubDispatch::Accept+0x5b0
	00007ffa`2dfdad76 : 000000a0`d81f1cf0 000000a0`d81f1cf8 000000a0`d81f1cf8 000000a0`d85817d0 : chrome_child!mojo::InterfaceEndpointClient::HandleValidatedMessage+0x1cc
	00007ffa`2dfd3520 : 000000a0`d86d8f80 000000a0`cf1d0120 000000a0`cb72dd28 000000a0`cb72dd28 : chrome_child!mojo::FilterChain::Accept+0x4a
	00007ffa`2dfd1a38 : 000000a0`d86d91a8 000000a0`cb72ddf0 00000000`00000002 00000000`00000000 : chrome_child!mojo::internal::MultiplexRouter::ProcessIncomingMessage+0x1dc
	00007ffa`2dfdad76 : 000000a0`d81f1d88 000000a0`d86d8fb8 00000000`00000000 00007ffa`2dfd0000 : chrome_child!mojo::internal::MultiplexRouter::Accept+0xc8
	00007ffa`2dfd6e3c : 000000a0`d86d9128 000000a0`cb72dfc0 00000000`00000000 00007ffa`300d2888 : chrome_child!mojo::FilterChain::Accept+0x4a
	00007ffa`2dfd6c3c : 00007ffa`2fb51648 000000a0`d8c44e40 000000a0`d86d8fe0 000000a0`d867d120 : chrome_child!mojo::Connector::ReadSingleMessage+0x180
	00007ffa`2dfd609c : 000000a0`00000000 000000a0`cb72e320 00007ffa`2f82f258 000000a0`cb72e0d8 : chrome_child!mojo::Connector::ReadAllAvailableMessages+0x44
	00007ffa`2dd14d82 : 000000a0`cb72e1b8 000000a0`d867dae0 000000a0`cb72e410 000000a0`cb72e538 : chrome_child!base::internal::Invoker<base::internal::BindState<void (__cdecl mojo::Connector::*)(unsigned int) __ptr64,base::internal::UnretainedWrapper<mojo::Connector> >,void __cdecl(unsigned int)>::RunImpl<void (__cdecl mojo::Connector::*const & __ptr64)(unsigned int) __ptr64,std::tuple<base::internal::UnretainedWrapper<mojo::Connector> > const & __ptr64,0>+0x44
	00007ffa`2dd14745 : 000000a0`cb720058 00000000`00000000 00007ffa`304d9cc0 00007ffa`2caad622 : chrome_child!mojo::SimpleWatcher::DiscardReadyState+0x16
	00007ffa`2dd1507a : 00007ffa`2f7d3218 000000a0`d842f4e0 00000002`00000000 00000002`77e83f06 : chrome_child!base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::RepeatingCallback<void __cdecl(unsigned int)> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),base::RepeatingCallback<void __cdecl(unsigned int)> >,void __cdecl(unsigned int,mojo::HandleSignalsState const & __ptr64)>::RunImpl<void (__cdecl*const & __ptr64)(base::RepeatingCallback<void __cdecl(unsigned int)> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),std::tuple<base::RepeatingCallback<void __cdecl(unsigned int)> > const & __ptr64,0>+0x49
	00007ffa`2dfe02f5 : 00007ffa`2f7d3218 000000a0`d842f4e0 000000a0`cbb90000 00000002`825f1600 : chrome_child!base::internal::Invoker<base::internal::BindState<void (__cdecl*)(base::RepeatingCallback<void __cdecl(unsigned int)> const & __ptr64,unsigned int,mojo::HandleSignalsState const & __ptr64),base::RepeatingCallback<void __cdecl(unsigned int)> >,void __cdecl(unsigned int,mojo::HandleSignalsState const & __ptr64)>::Run+0x1e
	00007ffa`2dbf7e10 : 000000a0`d867dac0 00007ffa`00000000 000000a0`d842f4e0 00000000`00000000 : chrome_child!mojo::SimpleWatcher::OnHandleReady+0xc9
	00007ffa`2dfdfb72 : 000000a0`d8485be8 00007ffa`2f82f238 000000a0`d8485be0 000000a0`cc127ad8 : chrome_child!base::internal::InvokeHelper<1,void>::MakeItSo<void (__cdecl media::DecryptingAudioDecoder::*const & __ptr64)(int,enum media::Decryptor::Status,std::list<scoped_refptr<media::AudioBuffer>,std::allocator<scoped_refptr<media::AudioBuffer> > > const & __ptr64) __ptr64,base::WeakPtr<media::DecryptingAudioDecoder> const & __ptr64,int const & __ptr64,enum media::Decryptor::Status,std::list<scoped_refptr<media::AudioBuffer>,std::allocator<scoped_refptr<media::AudioBuffer> > > const & __ptr64>+0x64
	00007ffa`2cb0cd85 : 000000a0`d8485bc0 00000000`00000001 000000a0`cb72e7a0 000000a0`cf16d810 : chrome_child!base::internal::Invoker<base::internal::BindState<void (__cdecl mojo::SimpleWatcher::*)(int,unsigned int,mojo::HandleSignalsState const & __ptr64) __ptr64,base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void __cdecl(void)>::RunImpl<void (__cdecl mojo::SimpleWatcher::*const & __ptr64)(int,unsigned int,mojo::HandleSignalsState const & __ptr64) __ptr64,std::tuple<base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState> const & __ptr64,0,1,2,3>+0x4e
	00007ffa`2deab04d : 000000a0`cc0ea310 000000a0`cb72e490 00007ffa`2f84edc8 00007ffa`304d9cc0 : chrome_child!base::debug::TaskAnnotator::RunTask+0x135
	00007ffa`2deaa1ec : 000000a0`cc127be0 000000a0`cc127be0 000000a0`cc127a80 000000a0`cb72e7a0 : chrome_child!blink::scheduler::TaskQueueManager::ProcessTaskFromWorkQueue+0x491
	00007ffa`2dbbf301 : 000000a0`cb72e8b0 00007ffa`2cabd400 00000000`00000000 000000a0`cf16c430 : chrome_child!blink::scheduler::TaskQueueManager::DoWork+0x1c4
	00007ffa`2dea8c18 : 000000a0`cc122740 000000a0`cb72e9b0 00007ffa`30172110 00000000`00000000 : chrome_child!base::internal::InvokeHelper<1,void>::MakeItSo<void (__cdecl media::AudioRendererImpl::*const & __ptr64)(bool) __ptr64,base::WeakPtr<media::AudioRendererImpl> const & __ptr64,bool>+0x61
	00007ffa`2cb0cd85 : 000000a0`cc122710 00007ffa`30172110 00000000`00000000 000000a0`cc128730 : chrome_child!base::internal::Invoker<base::internal::BindState<void (__cdecl blink::scheduler::TaskQueueManager::*)(bool) __ptr64,base::WeakPtr<blink::scheduler::TaskQueueManager>,bool>,void __cdecl(void)>::RunImpl<void (__cdecl blink::scheduler::TaskQueueManager::*const & __ptr64)(bool) __ptr64,std::tuple<base::WeakPtr<blink::scheduler::TaskQueueManager>,bool> const & __ptr64,0,1>+0x44
	00007ffa`2cadb2c3 : 00000000`00000988 00000001`d15b78e4 000000a0`cc122710 440000b3`000000b4 : chrome_child!base::debug::TaskAnnotator::RunTask+0x135
	00007ffa`2cadbc53 : 00007ffa`2f82f218 00007ffa`2f82f0f0 00000002`77e83f05 00000000`00000000 : chrome_child!base::MessageLoop::RunTask+0x243
	00007ffa`2cb2c441 : 00000002`77d4bc01 000000a0`00000184 00007ffa`2f82f218 00007ffa`2f82f0f0 : chrome_child!base::MessageLoop::DoWork+0x453
	00007ffa`2caf2c59 : 00000002`77d46a4f 00000002`77d46a4f 000000a0`cb72ef60 000000a0`cb72eed0 : chrome_child!base::MessagePumpDefault::Run+0x101
	00007ffa`2e8f7d4c : 000000a0`cb72f100 000000a0`cc0e40b0 000000a0`cc0e40b0 000000a0`cc126801 : chrome_child!base::RunLoop::Run+0x69
	00007ffa`2dfc6a3b : 00000000`00000003 00000000`00000006 00007ffa`2f84c200 000000a0`cb72f4b0 : chrome_child!content::RendererMain+0x2b8
	00007ffa`2dfc6886 : 000000a0`cc0edae0 000000a0`cb72f1d0 000000a0`cc0e40b0 000000a0`00000000 : chrome_child!content::RunNamedProcessTypeMain+0x15b
	00007ffa`2dfcd7e6 : 000000a0`cb72f410 000000a0`cb72f410 ffffffff`ffffffff 00000000`00000003 : chrome_child!content::ContentMainRunnerImpl::Run+0xd6
	00007ffa`2dfc60c4 : 00000002`77a107dd 00000002`77a107dd 000000a0`cb72f570 00007ffa`2ca950b5 : chrome_child!service_manager::Main+0x2ce
	00007ffa`2dab830e : 00000002`00000000 000000a0`cb72f410 00000002`77a107dd 00000000`00000007 : chrome_child!content::ContentMain+0x2c
	*** WARNING: Unable to verify checksum for chrome.exe
	00007ff6`56f1dd13 : 00000000`00000000 00000000`00000000 00007ffa`2ca998c0 00000000`00000000 : chrome_child!ChromeMain+0xf6
	00007ff6`56f1cc9b : 00000000`00000000 ffffffff`ffffffff 00000002`77a107dd 00000000`00000000 : chrome!MainDllLoader::Launch+0x313
	
	.  0  Id: 728.988 Suspend: 0 Teb: 00007ff6`5613e000 Unfrozen
	RetAddr           : Args to Child                                                           : Call Site
	00007ffa`4d333757 : 000000a0`cb72b338 00007ffa`2e115dfa 00000000`00000000 000000a0`cb72b320 : ntdll!ZwDelayExecution+0xa
	*** WARNING: Unable to verify checksum for chrome_elf.dll
	00007ffa`3dee172c : 00000000`00000000 000000a0`00000000 00000000`00000000 00000000`00000000 : KERNELBASE!SleepEx+0xa7
	00007ffa`4d3702ce : 00000000`00000000 00000000`00000000 000000a0`cb72f890 00000000`00000001 : chrome_elf!crashpad::CrashpadClient::StartHandler+0x114c
	00007ffa`4ffb8992 : 00000000`00000000 00007ffa`4d370180 00000000`00000000 000000a0`cb72f820 : KERNELBASE!UnhandledExceptionFilter+0x14e
	00007ffa`4ffa3f66 : 000000a0`000000c0 000000a0`cb72b790 000000a0`cc1a8fe0 000000a0`cb72f820 : ntdll!memset+0x852
	00007ffa`4ffb633d : 00000000`00000000 000000a0`cb72b840 000000a0`cb72bd10 000000a0`cb72b840 : ntdll!_C_specific_handler+0x96
	00007ffa`4ff33c00 : 00007ffa`4fc94960 000000a0`cb72bd10 00007ffa`2e7a3a00 00007ffa`2c49f229 : ntdll!_chkstk+0xfd
	00007ffa`4ffb544a : 00000133`2cac1161 000000a0`cb72c460 00000093`c28f4280 00007ffa`2ed09301 : ntdll!RtlWalkFrameChain+0x1560
	00007ffa`2d803c0f : 00000128`ecefb390 00007ffa`30506588 000000a0`cb72c600 00007ffa`2d6f4cdc : ntdll!KiUserExceptionDispatcher+0x3a
	00007ffa`2d6f2e4a : 00000093`c28f4280 000000a0`cb72c6f0 00000093`c28
First failure range:

https://build.chromium.org/p/chromium.perf/builders/Win%2010%20Perf/builds/1343

I will kick off a bisect.


Project Member

Comment 5 by 42576172...@developer.gserviceaccount.com, Sep 13 2017


=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: winx64_10_perf_bisect
  Benchmark    : system_health.memory_desktop
  Metric       : memory:chrome:all_processes:dump_count:background_avg/browse_news/browse_news_nytimes

Revision             Exit Code      N
chromium@500826      1 +- N/A       20      good
chromium@500848      1 +- N/A       20      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.nytimes system_health.memory_desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8968611409252857376


For feedback, file a bug with component Speed>Bisection
The link in #4 (great to have this now!) shows even earlier failures. Kicked off a bisect with a larger range.
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Sep 13 2017


=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: winx64_10_perf_bisect
  Benchmark    : system_health.memory_desktop
  Metric       : memory:chrome:all_processes:dump_count_avg/browse_news/browse_news_cnn

Revision             Exit Code      N
chromium@500300      0 +- N/A       20      good
chromium@500826      0 +- N/A       20      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.cnn system_health.memory_desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8968563616263661776


For feedback, file a bug with component Speed>Bisection
Cc: dtu@chromium.org
Dave: can we try to use pinpoint to bisect this failure?

Comment 10 by dtu@chromium.org, Sep 14 2017

Sure, it would make a good test case for functional bisect.
Can you kick one off from the dashboard for MacBook Air? (MacBook Pro is a little swamped right now.)
Project Member

Comment 11 by 42576172...@developer.gserviceaccount.com, Sep 19 2017

๐Ÿ“ Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1493463fb80000

Comment 12 by dtu@chromium.org, Sep 19 2017

Manually kicked off a Pinpoint job. Hopefully I got the parameters right!
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Sep 19 2017

๐Ÿ“ Pinpoint job completed.
https://pinpoint-dot-chromeperf.appspot.com/job/1493463fb80000

Comment 14 by dtu@chromium.org, Sep 19 2017

I'm going to double the repeat count to 24.
Project Member

Comment 15 by 42576172...@developer.gserviceaccount.com, Sep 19 2017

๐Ÿ“ Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/12909a77b80000
Project Member

Comment 16 by 42576172...@developer.gserviceaccount.com, Sep 19 2017

๐Ÿ“ Pinpoint job completed.
https://pinpoint-dot-chromeperf.appspot.com/job/12909a77b80000

Use a real HTMLVideoElement and Document/Frame in TextTrackListTest
By japhet@chromium.org ยท Fri Sep 08 01:16:05 2017
chromium@8fd1b3cda0a769d3d2273bd246995c70f90535b0
Owner: japhet@chromium.org
Status: Assigned (was: Available)
japhet@ the bisect found your CL in #16 as the culprit for the flakiness in nytimes test.

The crash stack in #1 include TaskRunnerHelper & your CL also touches third_party/WebKit/Source/core/dom/TaskRunnerHelper.cpp, so the bisect is likely to be correct. Please take a look as this as it can cause actual Chrome crash to the user.
Is this still happening on trunk? That crash stack was a top crasher for a while, but I thought I had resolved it in https://chromium-review.googlesource.com/c/chromium/src/+/663880
Status: Fixed (was: Assigned)

Sign in to add a comment