All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=741831

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=775e388583cf0cfe45c6026bbd0d042b85ea2fd652ebebc63885575fcd20c7c7


Bot(s) for this bug's original alert(s):

Android Builder

I'll see if I can bisect this to a clang revision, but it'll take a while.

Thanks! I presume you know about diagnose_bloat.py and supersize? go/apk-size-regression

Yeah. Not sure how much they will help though, since this was a compiler change.

Might be easy for bisecting to just run "diagnose_bloat.py HEAD" logN times.

Culprit seems to be r306477, which enabled a new analysis pass. I'll see if I can track down a few regression cases and report them upstream.

Thank you Peter, that sounds great!

Top 40 regressions according to supersize:

~ 0)      30444 (29.7%) t@0x8af83c   +30444 (13072->43516) cc/output/shader.cc
               cc::FragmentShader::GetShaderSource const
~ 1)      50708 (49.5%) t@0x8a4858   +20264 (8272->28536) cc/output/shader.cc
               cc::VertexShader::GetShaderString const
~ 2)      56976 (55.7%) t@0x16e3100  +6268 (3532->9800) gpu/command_buffer/service/gles2_cmd_copy_texture_chromium.cc
               gpu::gles2::CopyTextureCHROMIUMResourceManager::DoCopyTextureInternal
~ 3)      60708 (59.3%) t@0x8350dc   +3732 (2356->6088) ui/gfx/color_transform.cc
               gfx::ColorTransformSkTransferFn::AppendTransferShaderSource const
~ 4)      62524 (61.1%) t@0x17362c8  +1816 (3600->5416) gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::UpdateUniforms
~ 5)      64072 (62.6%) t@0x172808c  +1548 (704->2252)  gpu/command_buffer/service/gles2_cmd_srgb_converter.cc
               gpu::gles2::SRGBConverter::InitializeSRGBConverterProgram
~ 6)      65456 (63.9%) t@0x174da00  +1384 (2080->3464) gpu/command_buffer/service/texture_manager.cc
               gpu::gles2::TextureManager::ValidateTextureParameters
~ 7)      66800 (65.3%) t@0x1738380  +1344 (1948->3292) gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::UpdateProgramOutputs
~ 8)      68084 (66.5%) t@0x7a3314   +1284 (5216->6500) third_party/skia/src/gpu/gl/GrGLInterface.cpp
               GrGLInterface::validate const
~ 9)      69288 (67.7%) t@0x309a8c   +1204 (880->2084)  base/task_scheduler/scheduler_worker_pool_impl.cc
               base::internal::SchedulerWorkerPoolImpl::SchedulerWorkerPoolImpl
~ 10)     70484 (68.9%) t@0x16d5840  +1196 (584->1780)  gpu/command_buffer/service/error_state.cc
               gpu::gles2::ErrorStateImpl::SetGLErrorInvalidParami
~ 11)     71476 (69.8%) t@0x8abf10   +992 (2692->3684)  cc/output/shader.cc
               cc::FragmentShader::SetBlendModeFunctions const
~ 12)     72460 (70.8%) t@0x1739448  +984 (1164->2148)  gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::ExecuteProgramOutputBindCalls
~ 13)     73436 (71.7%) t@0x16d5000  +976 (436->1412)   gpu/command_buffer/service/error_state.cc
               gpu::gles2::ErrorStateImpl::SetGLError
~ 14)     74348 (72.6%) t@0x2f3308   +912 (516->1428)   base/metrics/statistics_recorder.cc
               base::StatisticsRecorder::ToJSON
~ 15)     75224 (73.5%) t@0x16d6304  +876 (404->1280)   gpu/command_buffer/service/error_state.cc
               gpu::gles2::ErrorStateImpl::ClearRealGLErrors
~ 16)     76076 (74.3%) t@0x8ab864   +852 (856->1708)   cc/output/shader.cc
               cc::FragmentShader::GetShaderString const
~ 17)     76916 (75.1%) t@0x2ecd7c   +840 (596->1436)   base/metrics/histogram_delta_serialization.cc
               base::HistogramDeltaSerialization::HistogramDeltaSerialization
~ 18)     77736 (75.9%) t@0x2f0064   +820 (516->1336)   base/metrics/persistent_memory_allocator.cc
               base::PersistentMemoryAllocator::CreateTrackingHistograms
~ 19)     78544 (76.7%) t@0xee26c8   +808 (340->1148)   v8/src/asmjs/asm-types.cc
               v8::internal::wasm::AsmMinMaxType::Name
~ 20)     79336 (77.5%) t@0x2d0322   +792 (1222->2014)  base/environment.cc
               base::AlterEnvironment
~ 21)     80104 (78.3%) t@0x2ca20a   +768 (764->1532)   base/command_line.cc
               base::CommandLine::AppendSwitchNative
~ 22)     80854 (79.0%) t@0x2de8fe   +750 (324->1074)   base/location.cc
               tracked_objects::Location::ToString const
~ 23)     81550 (79.7%) t@0x17377f0  +696 (2264->2960)  gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::UpdateFragmentInputs
~ 24)     82194 (80.3%) t@0x1706974  +644 (1416->2060)  gpu/command_buffer/service/gles2_cmd_decoder.cc
               gpu::gles2::GLES2DecoderImpl::PrepareTexturesForRender
~ 25)     82822 (80.9%) t@0x17351a4  +628 (432->1060)   gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::ProcessLogInfo
~ 26)     83438 (81.5%) t@0x308874   +616 (844->1460)   base/task_scheduler/scheduler_single_thread_task_runner_manager.cc
               base::internal::SchedulerSingleThreadTaskRunnerManager::CreateSingleThreadTaskRunnerWithTraits
~ 27)     84050 (82.1%) t@0x107b490  +612 (1224->1836)  v8/src/frames.cc
               v8::internal::SafeStackFrameIterator::SafeStackFrameIterator
~ 28)     84646 (82.7%) t@0x173b29a  +596 (532->1128)   gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::DetectProgramOutputLocationBindingConflicts const
~ 29)     85234 (83.3%) t@0x1739cac  +588 (2512->3100)  gpu/command_buffer/service/program_manager.cc
               gpu::gles2::Program::Link
- 30)     84670 (82.7%) t@0x0        -564 (564->0)      v8/src/frames.cc
               v8::internal::StackFrameIteratorBase::StackFrameIteratorBase
~ 31)     85222 (83.2%) t@0x3208e0   +552 (856->1408)   base/trace_event/memory_dump_manager.cc
               base::trace_event::MemoryDumpManager::Initialize
~ 32)     85774 (83.8%) t@0x107ab28  +552 (116->668)    v8/src/frames.cc
               v8::internal::StackFrameIterator::StackFrameIterator
~ 33)     86322 (84.3%) t@0x1721328  +548 (1776->2324)  gpu/command_buffer/service/gles2_cmd_decoder_passthrough_doers.cc
               gpu::gles2::GLES2DecoderPassthroughImpl::DoGetProgramInfoCHROMIUM
~ 34)     85794 (83.8%) t@0xaf4cb4   -528 (3764->3236)  cc/surfaces/surface_aggregator.cc
               cc::SurfaceAggregator::PrewalkTree
~ 35)     86310 (84.3%) t@0x16d5f34  +516 (356->872)    gpu/command_buffer/service/error_state.cc
               gpu::gles2::ErrorStateImpl::SetGLErrorInvalidParamf
~ 36)     86802 (84.8%) t@0x2d327c   +492 (1216->1708)  base/files/file_path.cc
               base::FilePath::GetComponents const
~ 37)     87294 (85.3%) t@0x2ef1d8   +492 (460->952)    base/metrics/persistent_histogram_allocator.cc
               base::GlobalHistogramAllocator::ConstructFilePaths
~ 38)     87784 (85.8%) t@0x30179c   +490 (194->684)    base/strings/string_util.cc
               base::JoinString
~ 39)     88272 (86.2%) t@0x172d850  +488 (424->912)    gpu/command_buffer/service/logger.cc
               gpu::gles2::Logger::LogMessage

I took a closer look at the largest ones and it seems that the root cause of most of them is that r306477 causes the function std::string::append to be optimized enough that its size goes below the inlining threshold for -O2 and therefore gets inlined into a bunch of places (basically, every use of the + operator to concatenate strings together in translation units compiled with -O2 gets its own copy of operator+).

There probably isn't much that we can do here unfortunately. LLVM's inlining heuristics aren't all that good, which is a well-known problem, and it seems that we've just been unlucky enough that we've been hit by this problem for a relatively frequently-called function.

Thank you for investigating!

Marking as blocking more as a "related" than a blocking :P

The file list above is all in targets that build with -O2. At least cc/output/shader.cc doesn't feel like it should be super perf sensitive, so if we wanted we could maybe split some of our -O2 targets into core -O2 code and -Os supporting code.

That's a good idea. File it as  bug 744594