isolate infrastructure is very slow when dealing with many small files |
|||||
Issue descriptionBlink / WebKit tests are a huge number (120k) of very small files. This seems to be a pathological case for the current isolate infrastructure. ---- For the uploading case; * Uploading 120k files (with a cold cache) takes roughly 90 minutes. * Uploading the same 120k files again (now with a hot cache) takes roughly between 30 and 120 seconds. (It is unclear why the *huge* variance). These stats seem to be roughly the same (within a reasonable error bound) for both the Go and Python implementations of this code. The tests were performed on a Linux machine which should be the *best* performing OS for this. ---- For the downloading case; * Downloading the 120k files (with a cold local cache) take roughly 2 hours. * Downloading the same 120k files again (now with a host cache) takes roughly 30 minutes. These stats were for the Python implementation (the Go version has yet to be written at this stage). The tests were performed on a Linux machine which should be the *best* performing OS for this. ---- For comparison to "dumb tar" on the same machine; * Creating an uncompressed tarball of the same 120k files takes 30 seconds with entirely being bound by disk IO. * Adding gzip compression to the creation of the tarball seems to have minimal affect on the time (giving 1/10th the size). * Adding bzip2 compression does increase the creation time. * Extracting the tarball of the 120k takes roughly double the creation time (60 seconds) with again being entirely being bound by disk IO. * Doing `tar | nc <tcp> nc | tar` runs as fast as the extraction time. Note for replicating: * Printing the filenames to the console causes *substantial* slowdown of all these steps * Were you write the output of tar can make a big difference, you can write the output to /dev/null * The state of the Linux disk cache can also make a big difference.
,
Mar 30 2016
"was" or "is"? It seems the issue is currently still open?
,
Mar 30 2016
It looks like the following issues on "luci-py" are also related; * https://github.com/luci/luci-py/issues/141 - Partition DiskCache into subdirectories to handle >100k local items (ala git object store) * https://github.com/luci/luci-py/issues/146 - Isolate download saturate network I/O in case of mixture of large number of small files (>10k) and small number of large files I'm guessing they don't matter as you only care about the new Go implementation?
,
Apr 26 2016
,
May 27 2016
Hi everyone following this bug, I have been working this week on making luci-go's archive command much faster when uploading small files and seem to have had some success based on the Python genar stuff. On a Linux SSD; * Original archive command - Hot Cache: 30 seconds - Cold Cache: 5:30 seconds * My archive command - Hot Cache: 8 seconds - Cold Cache: 35 seconds On a Linux spinning rust; * Original archive command - Hot Cache: 4:06 seconds - Cold Cache: ~15:00 seconds * My archive command - Hot Cache: 45 seconds - Cold Cache: 1:35 seconds Still yet to test on Windows / Mac, but if they have a similar profile to the original Python tests I'm expecting even better performance improvements. Hopefully this will help not only webkit-layout-tests but skia and v8 teams too.
,
Jul 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/6952ff7c2a890381a9745558f1aeb5fa499e727b commit 6952ff7c2a890381a9745558f1aeb5fa499e727b Author: tansell <tansell@chromium.org> Date: Thu Jul 28 15:43:29 2016 luci-py: Tools for working with BSD style ar archives. ar archives are the simplest format that satisfies the following criteria; * Is an existing standard which has tools which ship on standard Linux systems. * Requires no escaping / processing of file contents. * Header can be written without needing to understand the whole file. * Extremely fast to process / generate. Other formats which were consider before selecting the ar format; * tar * zip * cpio * rar * Something protobuf based. There are a couple of drawbacks; * The ar format doesn't support special files or symlinks. * The Linux ar tool don't extract with directory paths (listing is fine). The go version is in https://codereview.chromium.org/2043623002 * https://github.com/luci/luci-go/issues/9 * https://crbug.com/598990 BUG= 598990 Review-Url: https://codereview.chromium.org/2049523004 [add] https://crrev.com/6952ff7c2a890381a9745558f1aeb5fa499e727b/client/artool [add] https://crrev.com/6952ff7c2a890381a9745558f1aeb5fa499e727b/client/libs/arfile/__init__.py [add] https://crrev.com/6952ff7c2a890381a9745558f1aeb5fa499e727b/client/libs/arfile/arfile.py [add] https://crrev.com/6952ff7c2a890381a9745558f1aeb5fa499e727b/client/libs/arfile/arfile_test.py [add] https://crrev.com/6952ff7c2a890381a9745558f1aeb5fa499e727b/client/libs/arfile/cli.py
,
Jul 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/4a9a641bd2314bd4b40607598187b0ac6305203a commit 4a9a641bd2314bd4b40607598187b0ac6305203a Author: tansell <tansell@chromium.org> Date: Thu Jul 28 16:31:59 2016 luci-py: Refactor file writing code to allow file objects. This change refactors writing files out of the cache to support file like objects as the source. This allows files to be extracted from archives or directly written from a socket. The change also makes sure that only read only files are linked (otherwise modifications could corrupt the cache). It also makes sure that read only files in the isolate actually have read only permissions. * https://github.com/luci/luci-go/issues/9 * https://crbug.com/598990 BUG= 598990 Review-Url: https://codereview.chromium.org/2186263002 [modify] https://crrev.com/4a9a641bd2314bd4b40607598187b0ac6305203a/client/cipd.py [modify] https://crrev.com/4a9a641bd2314bd4b40607598187b0ac6305203a/client/isolateserver.py [modify] https://crrev.com/4a9a641bd2314bd4b40607598187b0ac6305203a/client/run_isolated.py [modify] https://crrev.com/4a9a641bd2314bd4b40607598187b0ac6305203a/client/tests/isolateserver_test.py
,
Jul 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807 commit 6a5bfa1af2f8cd27fb5a43e7f95b969b01404807 Author: tansell <tansell@chromium.org> Date: Thu Jul 28 16:45:40 2016 luci-py/isolateserver.py: Add archive support when downloading. Add support for extracting archives which contain many small files in one group together. This drastically increases the speed of dealing with a large number of small files. * https://github.com/luci/luci-go/issues/9 * https://crbug.com/598990 BUG= 598990 DEPS=2049523004 Review-Url: https://codereview.chromium.org/2060983006 [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/appengine/isolate/doc/Design.md [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/appengine/swarming/local_smoke_test.py [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/appengine/swarming/server/bot_archive.py [add] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/appengine/swarming/swarming_bot/libs [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/client/isolated_format.py [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/client/isolateserver.py [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/client/run_isolated.py [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/client/tests/isolateserver_test.py [modify] https://crrev.com/6a5bfa1af2f8cd27fb5a43e7f95b969b01404807/client/tests/run_isolated_smoke_test.py
,
Aug 18 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a7ce77dda519428ce58039382485c7cd0cc05ab7 commit a7ce77dda519428ce58039382485c7cd0cc05ab7 Author: maruel <maruel@chromium.org> Date: Thu Aug 18 13:59:57 2016 Roll isolate go to binaries as generated by infra @ 6f3db5a82956. Which is luci-go @ e438e8c3b563e48aaccca7bd0f4997ef29174e3d TBR=tansell@chromium.org BUG= 598990 Review-Url: https://codereview.chromium.org/2256093002 Cr-Commit-Position: refs/heads/master@{#412819} [modify] https://crrev.com/a7ce77dda519428ce58039382485c7cd0cc05ab7/tools/luci-go/linux64/isolate.sha1 [modify] https://crrev.com/a7ce77dda519428ce58039382485c7cd0cc05ab7/tools/luci-go/mac64/isolate.sha1 [modify] https://crrev.com/a7ce77dda519428ce58039382485c7cd0cc05ab7/tools/luci-go/win64/isolate.exe.sha1
,
Aug 18 2016
Technically speaking, this issue should now be closed. The servers were updated two weeks ago and now the clients have been done. I'll leave to Tim to confirm first and close this bug. This should unblock moving forward with layout tests. Sorry for the delay.
,
Nov 14 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1 commit ef55a41b26e19ca89d9b9b6c993469ab58bfaff1 Author: tansell <tansell@chromium.org> Date: Mon Nov 14 02:41:11 2016 luci-py/isolateserver.py: Add support for tar archives when downloading. Currently the isolateserver.py only supports extracting ar archive which contain many small files grouped together. This CL extends the support to also include tar archive format. This support will exist while we evaluate and compare the ar and tar file formats with real workloads on real servers. It is expected that one format will be removed in the future. * https://github.com/luci/luci-go/issues/9 * https://crbug.com/598990 BUG= 598990 Review-Url: https://codereview.chromium.org/2484133002 [modify] https://crrev.com/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1/appengine/isolate/doc/Design.md [modify] https://crrev.com/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1/client/isolated_format.py [modify] https://crrev.com/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1/client/isolateserver.py [add] https://crrev.com/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1/client/tests/archive.tar [modify] https://crrev.com/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1/client/tests/isolateserver_test.py [modify] https://crrev.com/ef55a41b26e19ca89d9b9b6c993469ab58bfaff1/client/tests/run_isolated_smoke_test.py
,
Dec 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/226173eef159f64d9442d69cc40d8c9ae63b861e commit 226173eef159f64d9442d69cc40d8c9ae63b861e Author: mcgreevy <mcgreevy@chromium.org> Date: Wed Dec 07 05:12:20 2016 Log hit and miss file and byte counts from exparchive. BUG= 598990 Review-Url: https://codereview.chromium.org/2561543002 [modify] https://crrev.com/226173eef159f64d9442d69cc40d8c9ae63b861e/client/cmd/isolate/checker.go [modify] https://crrev.com/226173eef159f64d9442d69cc40d8c9ae63b861e/client/cmd/isolate/checker_test.go [modify] https://crrev.com/226173eef159f64d9442d69cc40d8c9ae63b861e/client/cmd/isolate/exp_archive.go
,
Dec 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/8336655396a49a699ff3f7d9abf469ad7b8dd2eb commit 8336655396a49a699ff3f7d9abf469ad7b8dd2eb Author: djd <djd@chromium.org> Date: Wed Dec 07 05:36:35 2016 client/isolate: add context to NewChecker func This make Checker consistent with Uploader, and allows the underyling context to be cancelled if the isolate process ends early. Also removes some punctuation from a test case that was upsetting vet. BUG= 598990 R=mcgreevy@chromium.org Review-Url: https://codereview.chromium.org/2551973004 [modify] https://crrev.com/8336655396a49a699ff3f7d9abf469ad7b8dd2eb/client/cmd/isolate/checker.go [modify] https://crrev.com/8336655396a49a699ff3f7d9abf469ad7b8dd2eb/client/cmd/isolate/checker_test.go [modify] https://crrev.com/8336655396a49a699ff3f7d9abf469ad7b8dd2eb/client/cmd/isolate/exp_archive.go
,
Dec 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/5f13498bd8d62e887d76d0ad1148106860288480 commit 5f13498bd8d62e887d76d0ad1148106860288480 Author: Michael McGreevy <mcgreevy@chromium.org> Date: Thu Dec 08 02:45:13 2016 Roll luci-go. Rolling forward to 8336655 to pick up latest exparchive changes. BUG= 598990 Change-Id: I89027796b9dc044b45f16c1f4dcdfd7ad0959b9a Reviewed-on: https://chromium-review.googlesource.com/417624 Commit-Queue: Dave Sansome <dsansome@chromium.org> Reviewed-by: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/5f13498bd8d62e887d76d0ad1148106860288480/DEPS
,
Dec 8 2016
,
Dec 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/04ad4b49050a5b0b7f38fc0b0f2c598af45c8367 commit 04ad4b49050a5b0b7f38fc0b0f2c598af45c8367 Author: mcgreevy <mcgreevy@chromium.org> Date: Thu Dec 08 05:35:58 2016 Roll isolate binaries generated at infra@5f13498 Isolate sha1s were taken from the following builds: https://build.chromium.org/p/chromium.infra/builders/infra-continuous-precise-64/builds/4926 https://build.chromium.org/p/chromium.infra/builders/infra-continuous-mac-10.10-64/builds/4025 https://build.chromium.org/p/chromium.infra/builders/infra-continuous-win-64/builds/4844 R=djd@chromium.org BUG= 598990 Review-Url: https://codereview.chromium.org/2559463005 Cr-Commit-Position: refs/heads/master@{#437188} [modify] https://crrev.com/04ad4b49050a5b0b7f38fc0b0f2c598af45c8367/tools/luci-go/linux64/isolate.sha1 [modify] https://crrev.com/04ad4b49050a5b0b7f38fc0b0f2c598af45c8367/tools/luci-go/mac64/isolate.sha1 [modify] https://crrev.com/04ad4b49050a5b0b7f38fc0b0f2c598af45c8367/tools/luci-go/win64/isolate.exe.sha1
,
Dec 12 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/a5a69040846b468ebb09aef5cc458df85575b499 commit a5a69040846b468ebb09aef5cc458df85575b499 Author: djd <djd@chromium.org> Date: Mon Dec 12 00:56:29 2016 client/isolate: correctly propagate context.Context This was overlooked in luci/luci-go@8336655 since the unti test fake doesn't use the context at all. BUG= 598990 R=mcgreevy@chromium.org Review-Url: https://codereview.chromium.org/2570433002 [modify] https://crrev.com/a5a69040846b468ebb09aef5cc458df85575b499/client/cmd/isolate/checker.go [modify] https://crrev.com/a5a69040846b468ebb09aef5cc458df85575b499/client/cmd/isolate/checker_test.go
,
Dec 13 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/f84d8594c980fc2ed8a7cedf87bba9344f570afe commit f84d8594c980fc2ed8a7cedf87bba9344f570afe Author: Dave Day <djd@chromium.org> Date: Tue Dec 13 03:41:00 2016 Roll luci-go Rolling forward to @0451078 to pick up latest exparchive changes and fixes to lhttp retry logic/logging. BUG= 598990 , 671592 Change-Id: I9540c694bb69994a58e8836d72d425c61a919b44 Reviewed-on: https://chromium-review.googlesource.com/418103 Reviewed-by: Dave Sansome <dsansome@chromium.org> Commit-Queue: Dave D <djd@chromium.org> [modify] https://crrev.com/f84d8594c980fc2ed8a7cedf87bba9344f570afe/DEPS
,
Dec 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/1104bd661b98336a2e66f72e2ace4b2997b958c6 commit 1104bd661b98336a2e66f72e2ace4b2997b958c6 Author: Dave Day <djd@chromium.org> Date: Thu Dec 15 04:46:49 2016 Roll luci-go Rolling forward to e12a0ea to pick up latest exparchive changes. BUG= 598990 Change-Id: If7486bffe1e7d589494aea41d6cb510d39f69542 Reviewed-on: https://chromium-review.googlesource.com/419882 Commit-Queue: Dave Sansome <dsansome@chromium.org> Reviewed-by: Dave Sansome <dsansome@chromium.org> [modify] https://crrev.com/1104bd661b98336a2e66f72e2ace4b2997b958c6/DEPS
,
Dec 19 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3736c107fc832c28af2961351bfa31675c4e92da commit 3736c107fc832c28af2961351bfa31675c4e92da Author: djd <djd@chromium.org> Date: Mon Dec 19 01:21:13 2016 Roll isolate binaries generated at infra@e0081b8. This includes the luci-go repo luci/luci-go@e12a0ea Isolate sha1s were taken from the following builds: https://build.chromium.org/p/chromium.infra/builders/infra-continuous-precise-64/builds/5079 https://build.chromium.org/p/chromium.infra/builders/infra-continuous-mac-10.10-64/builds/4186 https://build.chromium.org/p/chromium.infra/builders/infra-continuous-win-64/builds/5006 BUG= 671592 , 598990 Review-Url: https://codereview.chromium.org/2581823002 Cr-Commit-Position: refs/heads/master@{#439393} [modify] https://crrev.com/3736c107fc832c28af2961351bfa31675c4e92da/tools/luci-go/linux64/isolate.sha1 [modify] https://crrev.com/3736c107fc832c28af2961351bfa31675c4e92da/tools/luci-go/mac64/isolate.sha1 [modify] https://crrev.com/3736c107fc832c28af2961351bfa31675c4e92da/tools/luci-go/win64/isolate.exe.sha1
,
Dec 19 2016
The following revision refers to this bug: https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/11577104592605b1e94476b2bb75385abcc45d8e commit 11577104592605b1e94476b2bb75385abcc45d8e Author: djd <djd@chromium.org> Date: Mon Dec 19 02:56:30 2016 client/isolate: add --dump_json support to exparchive This uses the same flag name and JSON format as the batcharchive command, but will only ever emit a single digest entry. R=mcgreevy@chromium.org BUG= 598990 Review-Url: https://codereview.chromium.org/2584323002 [modify] https://crrev.com/11577104592605b1e94476b2bb75385abcc45d8e/client/cmd/isolate/exp_archive.go
,
Feb 16 2017
,
Feb 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/acf747b56b857805b0fd150cb9e9b4d51b762f90 commit acf747b56b857805b0fd150cb9e9b4d51b762f90 Author: Michael Achenbach <machenbach@chromium.org> Date: Thu Feb 16 15:47:09 2017 V8: Dogfood new exparchive feature on fyi bot BUG= 598990 Change-Id: I06ec611455a7bc82ff620f9213454838429ef554 Reviewed-on: https://chromium-review.googlesource.com/443468 Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> Commit-Queue: Michael Achenbach <machenbach@chromium.org> [modify] https://crrev.com/acf747b56b857805b0fd150cb9e9b4d51b762f90/scripts/slave/recipe_modules/v8/api.py [modify] https://crrev.com/acf747b56b857805b0fd150cb9e9b4d51b762f90/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___cfi.json
,
Feb 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/e1f7ae3328b7c23249e61dfa409237976f8811ea commit e1f7ae3328b7c23249e61dfa409237976f8811ea Author: Michael Achenbach <machenbach@chromium.org> Date: Fri Feb 17 07:35:38 2017 [swarming] Dogfood exparchive feature The new swarming archiving method requires isolate files to be copied to the build dir. BUG= chromium:598990 NOTRY=true TBR=tandrii@chromium.org,tansell@chromium.org,vogelheim@chromium.org Change-Id: I87633725d9ce715684f761ce0c8d2691642c5389 Reviewed-on: https://chromium-review.googlesource.com/444644 Reviewed-by: Michael Achenbach <machenbach@chromium.org> Commit-Queue: Michael Achenbach <machenbach@chromium.org> Cr-Commit-Position: refs/heads/master@{#43266} [modify] https://crrev.com/e1f7ae3328b7c23249e61dfa409237976f8811ea/infra/mb/mb_config.pyl
,
Feb 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/7f2771ee95dbbf699f35f612845718ac6254e3c4 commit 7f2771ee95dbbf699f35f612845718ac6254e3c4 Author: Michael Achenbach <machenbach@chromium.org> Date: Fri Feb 17 10:18:41 2017 Revert "[swarming] Dogfood exparchive feature" This reverts commit e1f7ae3328b7c23249e61dfa409237976f8811ea. Reason for revert: Breaks on the bot: https://build.chromium.org/p/client.v8/builders/V8%20Linux64%20-%20cfi/builds/8773/steps/compile/logs/stdio Original change's description: > [swarming] Dogfood exparchive feature > > The new swarming archiving method requires isolate files to be copied to the build dir. > > BUG= chromium:598990 > NOTRY=true > TBR=tandrii@chromium.org,tansell@chromium.org,vogelheim@chromium.org > > Change-Id: I87633725d9ce715684f761ce0c8d2691642c5389 > Reviewed-on: https://chromium-review.googlesource.com/444644 > Reviewed-by: Michael Achenbach <machenbach@chromium.org> > Commit-Queue: Michael Achenbach <machenbach@chromium.org> > Cr-Commit-Position: refs/heads/master@{#43266} TBR=machenbach@chromium.org,vogelheim@chromium.org,tandrii@chromium.org,tansell@chromium.org,v8-reviews@googlegroups.com NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= chromium:598990 Change-Id: Ie25205f8df69981fde223a232348534487460808 Reviewed-on: https://chromium-review.googlesource.com/444225 Commit-Queue: Michael Achenbach <machenbach@chromium.org> Reviewed-by: Michael Achenbach <machenbach@chromium.org> Cr-Commit-Position: refs/heads/master@{#43271} [modify] https://crrev.com/7f2771ee95dbbf699f35f612845718ac6254e3c4/infra/mb/mb_config.pyl
,
Feb 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/c8131a96c140da6f341343401c416fb2df531136 commit c8131a96c140da6f341343401c416fb2df531136 Author: Michael Achenbach <machenbach@chromium.org> Date: Fri Feb 17 10:21:52 2017 Revert "V8: Dogfood new exparchive feature on fyi bot" This reverts commit acf747b56b857805b0fd150cb9e9b4d51b762f90. Reason for revert: The old infrastructure only produces .isolate.gen.json files for the old batcharchive. The new step misses the normal .isolate file. Trying to generate those failed on the V8 side: https://build.chromium.org/p/client.v8/builders/V8%20Linux64%20-%20cfi/builds/8773/steps/compile/logs/stdio Possibly this might be blocked on: http://crbug.com/669910 Original change's description: > V8: Dogfood new exparchive feature on fyi bot > > BUG= 598990 > > Change-Id: I06ec611455a7bc82ff620f9213454838429ef554 > Reviewed-on: https://chromium-review.googlesource.com/443468 > Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> > Commit-Queue: Michael Achenbach <machenbach@chromium.org> > TBR=machenbach@chromium.org,tandrii@chromium.org,tansell@chromium.org,maruel@chromium.org,chromium-reviews@chromium.org NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 598990 Change-Id: I4692d9be9ffa655f2dae13e86b0ad822847b68a3 Reviewed-on: https://chromium-review.googlesource.com/444405 Reviewed-by: Michael Achenbach <machenbach@chromium.org> Commit-Queue: Michael Achenbach <machenbach@chromium.org> [modify] https://crrev.com/c8131a96c140da6f341343401c416fb2df531136/scripts/slave/recipe_modules/v8/api.py [modify] https://crrev.com/c8131a96c140da6f341343401c416fb2df531136/scripts/slave/recipes/v8.expected/full_client_v8_V8_Linux64___cfi.json
,
May 2 2017
This has been fixed. We can make further improvements, but let's have other bugs for that.
,
Jun 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/6477127e502255f62cfa126990e23122335760b9 commit 6477127e502255f62cfa126990e23122335760b9 Author: Tim 'mithro' Ansell <tansell@chromium.org> Date: Fri Jun 09 05:20:20 2017 isolate: Add support for percentage experiments for exparchive. In https://crbug.com/689735 the perfbots where converted whole sale to use exparchive instead of batcharchive. This significantly improve performance. However, we are worried that exparchive might not be faster in *all* cases, so we want to run an experiment with a percentage of runs on a builder using exparchive rather than batcharchive. BUG= 689735 , 692940 , 598990 Change-Id: Id00f35567edcdc83c77a38c8d0f86e3da1096519 Reviewed-on: https://chromium-review.googlesource.com/526874 Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org> Reviewed-by: Michael McGreevy <mcgreevy@chromium.org> [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build8.json [modify] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.py [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build7.json [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build3.json [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build1.json [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build10.json [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build6.json [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build4.json [modify] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/chromium_tests/api.py [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build9.json [modify] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/api.py [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build2.json [add] https://crrev.com/6477127e502255f62cfa126990e23122335760b9/scripts/slave/recipe_modules/isolate/examples/full.expected/use-exparchive-20percent-build5.json
,
Jun 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/69a3565e74d2187c337aada58f63427dbf48603b commit 69a3565e74d2187c337aada58f63427dbf48603b Author: Tim 'mithro' Ansell <tansell@chromium.org> Date: Thu Jun 15 05:52:30 2017 linux_chromium_rel_ng: Enable exparchive for 5% of builds. In https://crbug.com/689735 the perfbots where converted whole sale to use exparchive instead of batcharchive. This significantly improve performance. However, we are worried that exparchive might not be faster in *all* cases, so we want to run an experiment with a percentage of runs on a builder using exparchive rather than batcharchive. BUG= 689735 , 692940 , 598990 Change-Id: I2cbead4bbcb0c4f68b1ba111284b1521ff5d72fd Reviewed-on: https://chromium-review.googlesource.com/532767 Commit-Queue: Tim 'mithro' Ansell <tansell@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org> [modify] https://crrev.com/69a3565e74d2187c337aada58f63427dbf48603b/scripts/slave/recipe_modules/chromium_tests/chromium_linux.py [modify] https://crrev.com/69a3565e74d2187c337aada58f63427dbf48603b/scripts/slave/recipe_modules/chromium_tests/chromium_gpu.py [modify] https://crrev.com/69a3565e74d2187c337aada58f63427dbf48603b/scripts/slave/recipe_modules/chromium_tests/tests/api/trybot_steps.py |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by mar...@chromium.org
, Mar 30 2016