Telemetry install is slow. |
|||
Issue descriptionChrome Version: ToT OS: ChromeOS Due to dependency, the telemetry package install is often on the build_package's critical path. Installing telemetry takes relatively longer time (practically, 2-3mins, but sometimes 6mins+ (cyan-chrome-pfq)). Specifically, if chrome is not necessary to build, it is about 10-20% time on build_package (3 mins / 20mins very roughly). I think we have several items that can improve the performance. 1) Split dependency. The main reason why telemetry is on the critical path is; it depends on Chrome's source and some small binary utilities, so it depends on chromeos-chrome package. If we can split chromeos-chrome package into two; "source sync + utility build" and "build chrome", then telemetry can depend on only the first package, so that install operation can run in parallel with other packages' build/install. 2) Split dependency part 2. Or, we may want to split telemetry package as well, like "telemetry-scripts" and "telemetry-binary". Instead, chromeos-chrome package can be split into "source sync" and "binary (chrome + utilities) build". Then, telemetry script install can start earlier, too. 3) Write doins alternative. From another perspective, the main reason why telemetry is so heavy is because "doins" is slow. Telemetry has lots of files (about 17K), and for each file, doins creates several processes. If we rewrite it, we can reduce much amount of time. In my local rough test; -- with doins.py (experiment) real 0m52.543s user 3m9.080s sys 0m24.628s -- with doins (control) real 4m22.780s user 4m7.872s sys 1m43.860s The concern of this approach is maintenance, of course. Even if we take this approach, we'd need to limit the usage to telemetry, in order to avoid diverge from upstream build system. It also allows us to simplify the script, because we can drop unnecessary logic. Conceptually, "1) or 2)", and "3)" is independent. We have option to choose both, if we'd like. Any thoughts?
,
Apr 12 2017
any rewrite you try should be done in portage itself rather than any ebuild or eclass specific thing. if you can get equiv behavior but with better performance, i don't think anyone would say no there. in the mean time, if all the permissions are what you want, you could manually `dodir` and then `cp -pPR` on the files.
,
Apr 12 2017
If it's a split dependency solution (options 1 and/or 2) then it would mandate splitting the ebuild into two "sub" ebuilds, right? @hidehiko: Have you been able to ascertain whether the chromeos-chrome and telemetry packages can indeed be refactored?
,
Apr 13 2017
Thank you for comments! Re #2: achuith@, do you know a practical way to filter out unnecessary code, from third_party libs for telemetry? Can we rely on import deps only? Or is there any more deps, like data dependencies? Re #3: Thanks. If doins in portage can be accelerated, then the benefit is bigger. Two questions. 1) Does ChromeOS build system updates portage to newest? 2) my proposal is written in python, but AFAIK, portage is a build system written in bash. Is an executable python script acceptable to replace the bash for performance? (To be clear, I'd like to know the culture there, as I'm new.) Re #4: Not yet. If that looks good direction for the experts (all of you!), I'll give it a try. Note that, the biggest concern is how to deal with chromeos-chrome's cros_workon. However, I'm optimistic now.
,
Apr 13 2017
portage is written in python and bash. having doins be rewritten in python is fine. our copy of portage is here: https://chromium.googlesource.com/chromiumos/third_party/portage_tool/ it's older than upstream, but this helper hasn't changed upstream, so if you did the work there, we can send it upstream later pretty easily.
,
Apr 18 2017
Re #5: Thank you for advice! Filed another bug for its tracking. crbug.com/712659 .
,
Jan 16
,
Jan 16
|
|||
►
Sign in to add a comment |
|||
Comment 1 by derat@chromium.org
, Apr 12 2017Components: Tests>Telemetry OS>Packages
I was curious about why the package contains so many files. It looks like many of them are test data: - PNG images for canvas_bench - lots of Python scripts and .js and .sha1 files for page_sets - lots of HTML files for WebGL conformance tests - etc. There are also a lot of Python scripts that are included as part of third-party code and even some PyDoc documentation. Here's a naive attempt to provide some high-level file counts: 4889 ./third_party/catapult/third_p 4795 ./third_party/webgl/src/sdk/te 1579 ./third_party/catapult/telemet 1343 ./third_party/catapult/tracing 405 ./chrome/test/data/perf/canvas 360 ./third_party/catapult/dashboa 122 ./third_party/catapult/devil/d 86 ./third_party/protobuf/python/ 82 ./tools/perf/page_sets/data/sy 82 ./third_party/catapult/netlog_ 80 ./tools/perf/page_sets/data/to 75 ./third_party/catapult/systrac 74 ./third_party/catapult/common/ 74 ./chrome/test/data/third_party 68 ./tools/perf/page_sets/tough_a 57 ./third_party/tlslite/tlslite/ 55 ./third_party/catapult/trace_p