Issue metadata
Sign in to add a comment
|
system_health.memory_mobile failure on chromium.perf: Android One Perf (1) |
||||||||||||||||||||
Issue descriptionhttps://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%281%29 Traceback (most recent call last): File "/b/build/slave/Android_One_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 85, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/build/slave/Android_One_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 319, in RunStory self._current_page.Run(self) File "/b/build/slave/Android_One_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 99, in Run self.RunPageInteractions(action_runner) File "/b/build/slave/Android_One_Perf__1_/build/src/tools/perf/page_sets/system_health/system_health_story.py", line 78, in RunPageInteractions action_runner.tab.WaitForDocumentReadyStateToBeComplete() File "/b/build/slave/Android_One_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 70, in WaitForDocumentReadyStateToBeComplete 'document.readyState == "complete"', timeout) File "/b/build/slave/Android_One_Perf__1_/build/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 136, in WaitForJavaScriptExpression e.message + '\n' + debug_message) TimeoutException: Exception thrown when trying to capture console output: TimeoutException('',)
,
Aug 4 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/93fb146c2bf9ef15b3c10ef17a2bbea2ba488433 commit 93fb146c2bf9ef15b3c10ef17a2bbea2ba488433 Author: simonhatch <simonhatch@chromium.org> Date: Thu Aug 04 13:58:35 2016 Disable system_health.memory_mobile on Android One. TBR=nednguyen@google.com BUG= 634112 Review-Url: https://codereview.chromium.org/2210533003 Cr-Commit-Position: refs/heads/master@{#409773} [modify] https://crrev.com/93fb146c2bf9ef15b3c10ef17a2bbea2ba488433/tools/perf/benchmarks/system_health.py
,
Aug 4 2016
Only page "browse:news:washingtonpost" is failing. I think we should just disable it on Android one perf? It's a crash: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_44-2016-08-04_01-31-0812831.png :-(
,
Aug 4 2016
#3: Strange, my screenshot (#1) wasn't a crash. We currently don't have a way to disable single stories on individual platforms (I'll file a bug to add that capability).
,
Aug 4 2016
Considering that the bot was green for many revisions, it seems to be chromium regression. Isn't the standard practice to bisect and revert the bad chromium revision?
,
Aug 4 2016
I remember there was a way to start bisect bot based on success/failure of a run, but I cannot find it. Does anyone have this info handy?
,
Aug 4 2016
Good point, forgot to start a bisect here, I'll get one going!
,
Aug 4 2016
It's a little complicated, see https://chromium.googlesource.com/chromium/src/+/master/tools/perf/docs/perf_bot_sheriffing.md#Handle-Test-Failures.
,
Aug 4 2016
Both browse:news:washingtonpost (https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_44-2016-08-03_17-02-4815966.png) and load:news:washingtonpost (https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_35-2016-08-03_17-02-4860840.png) are failing on Nexus 5: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%281%29/builds/3859/steps/system_health.memory_mobile/logs/stdio. It appears that both stories fail due to OOM: D/dalvikvm(19030): GC_CONCURRENT freed 506K, 4% free 18785K/19492K, paused 3ms+4ms, total 28ms In other words, this could be the same problem as (or related to) issue 629123.
,
Aug 4 2016
Thank you for starting bisect, Simon. Thank you for the link, Petr. Interesting that v8.browsing_mobile is running ok. It also runs the washingtonpost story.
,
Aug 4 2016
#10: Unfortunately, it isn't running completely ok either: https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%281%29/builds/3599/steps/v8.browsing_mobile/logs/stdio
,
Aug 4 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9005240817476021344
,
Aug 5 2016
I reproduced a crash locally in debug and release mode. The local crash in debug mode happens in https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/workers/WorkerThread.cpp?rcl=0&l=305 That line was added in https://codereview.chromium.org/2201623002, which landed close to the first failures on the bot. Unfortunately, I cannot get sensible stack trace for the release mode crash, but if I revert https://codereview.chromium.org/2209603003 and https://codereview.chromium.org/2201623002, then I can run the benchmark successfully in release mode. Assigning to the author of the suspect CL. dgozman@, could you please take a look? To reproduce: 1. ninja -C out/Debug chrome_public_apk 2. ./tools/perf/run_benchmark system_health.memory_mobile --story-filter=washingtonpost --device=android --browser-executable=$(pwd)/out/Debug/apks/ChromePublic.apk --reset-result Stack trace for debug mode crash: signal 11 (SIGSEGV), code 1, fault addr 0x0 in tid 15198 (ServiceWorker T) pid: 15115, tid: 15198, name: ServiceWorker T >>> org.chromium.chrome:sandboxed_process0 <<< signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 r0 00000000 r1 518e2610 r2 98f04de8 r3 a66f2e4f r4 b6e0ddd4 r5 98f04f1c r6 00000000 r7 518e2610 r8 98f04f04 r9 98f04f10 sl 98f04f1c fp b6e0ddd4 ip a0354e84 sp 98f04c80 lr a0d3c26d pc a0d3bfcc Stack Trace: RELADDR FUNCTION FILE:LINE 009a3fcc blink::WorkerThread::isCurrentThread() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/workers/WorkerThread.cpp:243 009a4269 blink::WorkerThread::workerInspectorController() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/workers/WorkerThread.cpp:305 008d6085 blink::InspectorInstrumentation::instrumentingAgentsFor(blink::WorkerGlobalScope*) /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/inspector/InspectorInstrumentation.cpp:151 0047919f blink::InspectorInstrumentation::didFailFetch(blink::ExecutionContext*, blink::ThreadableLoaderClient*) /usr/local/google/home/ulan/chrome/src/out/Debug/gen/blink/core/InspectorInstrumentationImpl.cpp:666 001da0ef blink::FetchManager::Loader::failed(WTF::String const&) /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/modules/fetch/FetchManager.cpp:697 001da19b blink::FetchManager::Loader::didFailAccessControlCheck(blink::ResourceError const&) /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/modules/fetch/FetchManager.cpp:395 0091b417 blink::WorkerThreadableLoader::didFail(blink::ResourceError const&) /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/loader/WorkerThreadableLoader.cpp:373 0091bc25 blink::WorkerThreadableLoader::cancel() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/loader/WorkerThreadableLoader.cpp:303 001d8491 blink::FetchManager::Loader::dispose() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/modules/fetch/FetchManager.cpp:527 001d92c5 blink::FetchManager::contextDestroyed() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/modules/fetch/FetchManager.cpp:736 005070eb blink::LifecycleNotifier<blink::ExecutionContext, blink::ContextLifecycleObserver>::notifyContextDestroyed() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/platform/LifecycleNotifier.h:102 00525aef blink::ExecutionContext::notifyContextDestroyed() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/dom/ExecutionContext.cpp:95 009a4527 blink::WorkerThread::performShutdownOnWorkerThread() /usr/local/google/home/ulan/chrome/src/third_party/WebKit/Source/core/workers/WorkerThread.cpp:585 v------> void base::internal::FunctorTraits<void (*)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), void>::Invoke<std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> > >(void (*)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >&&) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:164 v------> void base::internal::InvokeHelper<false, void>::MakeItSo<void (* const&)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> > >(void (* const&)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >&&) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:284 v------> void base::internal::Invoker<base::internal::BindState<void (*)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), base::internal::PassedWrapper<std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> > > >, void ()>::RunImpl<void (* const&)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), std::__1::tuple<base::internal::PassedWrapper<std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> > > > const&, 0u>(void (* const&)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), std::__1::tuple<base::internal::PassedWrapper<std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> > > > const&, base::IndexSequence<0u>) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:349 00023b5d base::internal::Invoker<base::internal::BindState<void (*)(std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> >), base::internal::PassedWrapper<std::__1::unique_ptr<blink::WebTaskRunner::Task, std::__1::default_delete<blink::WebTaskRunner::Task> > > >, void ()>::Run(base::internal::BindStateBase*) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:327 v------> base::Callback<void (), (base::internal::CopyMode)1>::Run() const /usr/local/google/home/ulan/chrome/src/base/callback.h:389 0007457d base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) /usr/local/google/home/ulan/chrome/src/base/debug/task_annotator.cc:51 0001c78d scheduler::TaskQueueManager::ProcessTaskFromWorkQueue(scheduler::internal::WorkQueue*, scheduler::internal::TaskQueueImpl::Task*) /usr/local/google/home/ulan/chrome/src/components/scheduler/base/task_queue_manager.cc:315 0001ca93 scheduler::TaskQueueManager::DoWork(base::TimeTicks, bool) /usr/local/google/home/ulan/chrome/src/components/scheduler/base/task_queue_manager.cc:218 v------> void base::internal::FunctorTraits<void (scheduler::TaskQueueManager::*)(base::TimeTicks, bool), void>::Invoke<base::WeakPtr<scheduler::TaskQueueManager> const&, base::TimeTicks const&, bool const&>(void (scheduler::TaskQueueManager::*)(base::TimeTicks, bool), base::WeakPtr<scheduler::TaskQueueManager> const&, base::TimeTicks const&, bool const&) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:214 v------> void base::internal::InvokeHelper<true, void>::MakeItSo<void (scheduler::TaskQueueManager::* const&)(base::TimeTicks, bool), base::WeakPtr<scheduler::TaskQueueManager> const&, base::TimeTicks const&, bool const&>(void (scheduler::TaskQueueManager::* const&)(base::TimeTicks, bool), base::WeakPtr<scheduler::TaskQueueManager> const&, base::TimeTicks const&, bool const&) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:303 v------> void base::internal::Invoker<base::internal::BindState<void (scheduler::TaskQueueManager::*)(base::TimeTicks, bool), base::WeakPtr<scheduler::TaskQueueManager>, base::TimeTicks, bool>, void ()>::RunImpl<void (scheduler::TaskQueueManager::* const&)(base::TimeTicks, bool), std::__1::tuple<base::WeakPtr<scheduler::TaskQueueManager>, base::TimeTicks, bool> const&, 0u, 1u, 2u>(void (scheduler::TaskQueueManager::* const&)(base::TimeTicks, bool), std::__1::tuple<base::WeakPtr<scheduler::TaskQueueManager>, base::TimeTicks, bool> const&, base::IndexSequence<0u, 1u, 2u>) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:349 0001ae0d base::internal::Invoker<base::internal::BindState<void (scheduler::TaskQueueManager::*)(base::TimeTicks, bool), base::WeakPtr<scheduler::TaskQueueManager>, base::TimeTicks, bool>, void ()>::Run(base::internal::BindStateBase*) /usr/local/google/home/ulan/chrome/src/base/bind_internal.h:327 v------> base::Callback<void (), (base::internal::CopyMode)1>::Run() const /usr/local/google/home/ulan/chrome/src/base/callback.h:389 0007457d base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) /usr/local/google/home/ulan/chrome/src/base/debug/task_annotator.cc:51 0008a5b1 base::MessageLoop::RunTask(base::PendingTask const&) /usr/local/google/home/ulan/chrome/src/base/message_loop/message_loop.cc:496 0008ac8b base::MessageLoop::DeferOrRunPendingTask(base::PendingTask) /usr/local/google/home/ulan/chrome/src/base/message_loop/message_loop.cc:505 0008ad89 base::MessageLoop::DoWork() /usr/local/google/home/ulan/chrome/src/base/message_loop/message_loop.cc:629 0008ce69 base::MessagePumpDefault::Run(base::MessagePump::Delegate*) /usr/local/google/home/ulan/chrome/src/base/message_loop/message_pump_default.cc:35 0008bfb7 base::MessageLoop::RunHandler() /usr/local/google/home/ulan/chrome/src/base/message_loop/message_loop.cc:459 000a2435 base::RunLoop::Run() /usr/local/google/home/ulan/chrome/src/base/run_loop.cc:35 000bef3b base::Thread::Run(base::RunLoop*) /usr/local/google/home/ulan/chrome/src/base/threading/thread.cc:228 000bfa39 base::Thread::ThreadMain() /usr/local/google/home/ulan/chrome/src/base/threading/thread.cc:301 000bb957 ThreadFunc /usr/local/google/home/ulan/chrome/src/base/threading/platform_thread_posix.cc:70 0001659f __pthread_start(void*)+30 /system/lib/libc.so 000144cb __start_thread+6
,
Aug 5 2016
+cc nednguyen FYI load:news:washingtonpost and browse:news:washingtonpost are still causing failures on Nexus 5 (both system_health.memory_mobile and v8.browsing_mobile). I think that we should disable the story on mobile completely and re-enable system_health.memory_mobile on Android One (I'll prepare a patch).
,
Aug 5 2016
#14 sgtm
,
Aug 5 2016
,
Aug 5 2016
I'm going to repurpose this bug for the OOMs that are probably unrelated to issue 634699.
,
Aug 5 2016
I don't know why the bug hasn't been autoupdated yet, but the patch disabling {load, browse}:news:washingtonpost on mobile has just landed (https://codereview.chromium.org/2222453003/).
,
Aug 5 2016
,
Aug 5 2016
===== BISECT JOB RESULTS ===== Status: failed ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@409303 0 N/A 20 good chromium@409483 1 N/A 20 bad Bisect job ran on: android_one_perf_bisect Bug ID: 634112 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_mobile Test Metric: load_tools-memory:chrome:renderer_processes:process_count_avg/load_tools-memory:chrome:renderer_processes:process_count_avg Relative Change: Zero to non-zero Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1480 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005240817476021344 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5339320752275456 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 5 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9005148667281675696
,
Aug 5 2016
===== BISECT JOB RESULTS ===== Status: failed ===== TESTED REVISIONS ===== Revision Exit Code Std Dev N Good? chromium@409303 0 N/A 20 good chromium@409483 1 N/A 20 bad Bisect job ran on: android_one_perf_bisect Bug ID: 634112 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_mobile Test Metric: load_tools-memory:chrome:renderer_processes:process_count_avg/load_tools-memory:chrome:renderer_processes:process_count_avg Relative Change: Zero to non-zero Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1480 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005240817476021344 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5339320752275456 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 8 2016
My patch disabling {load, browse}:news:washingtonpost on mobile landed on Friday: https://codereview.chromium.org/2222453003/ (no idea why it didn't auto-update this bug).
,
Aug 8 2016
> I'm going to repurpose this bug for the OOMs that are probably unrelated to issue 634699. Petr, do you have any action items in mind for this bug besides re-enabling benchmarks once crbug.com/634699 is fixed?
,
Aug 8 2016
I think we should try to re-enable the stories once issue 634699 is fixed, but I expect them to fail due to OOM. If that's indeed the case, I'll try to get an actual Android One device and try it locally to see what's going on.
,
Aug 8 2016
OK, thanks. Out of curiosity, why do you expect OOM? I was able to run the benchmark without OOM on Android One locally when I reverted the CL that was causing issue 634699. Also the bot was green before 634699.
,
Aug 8 2016
I've seen it happen a couple of times while sheriffing on Thursday and Friday. If it was due to issue 634699, that would be great :-)
,
Aug 8 2016
,
Aug 10 2016
Ulan: Could you please try to re-enable the failing stories (now that issue 634699 is fixed)?
,
Aug 11 2016
Ulan is OOO, so I'll take care of this: https://codereview.chromium.org/2240493002/
,
Aug 11 2016
,
Aug 11 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/521e1a06a15bd6451113c6bfdc881a4eef143c34 commit 521e1a06a15bd6451113c6bfdc881a4eef143c34 Author: petrcermak <petrcermak@chromium.org> Date: Thu Aug 11 17:16:22 2016 [system-health] Re-enable {load, browse}:news:washingtonpost The underlying issue (http://crbug.com/634699) has been fixed. BUG= 634112 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.perf:android_s5_perf_cq;master.tryserver.chromium.perf:linux_perf_cq;master.tryserver.chromium.perf:mac_retina_perf_cq;master.tryserver.chromium.perf:winx64_10_perf_cq Review-Url: https://codereview.chromium.org/2240493002 Cr-Commit-Position: refs/heads/master@{#411365} [modify] https://crrev.com/521e1a06a15bd6451113c6bfdc881a4eef143c34/tools/perf/page_sets/system_health/browsing_stories.py [modify] https://crrev.com/521e1a06a15bd6451113c6bfdc881a4eef143c34/tools/perf/page_sets/system_health/loading_stories.py
,
Aug 12 2016
The bot has been green (https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%281%29) since the re-enabling patch landed (https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%281%29/builds/3671), so I'm going to mark this as Fixed. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by petrcermak@chromium.org
, Aug 4 2016