New issue
Advanced search Search tips

Issue 643243 link

Starred by 1 user
Status: Archived
Owner:
d...@chromium.org
Closed: Sep 26
Cc:
hinoka@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Optimize / tune LogDog parameters.

Project Member Reported by d...@chromium.org, Sep 1 2016 
Currently, LogDog has the following things enabled:
1) Aggregate Butler-output STDOUT/STDERR streams (effectively doubles all log information when used with Annotee).
2) Index sparseness. Currently, all logs are included in the index, increasing its size a lot more than it needs to be.
3) Butler buffer settings. How much time should Butler buffer logs before sending? There may be a bug here, make sure we're not seeing one-message-per-line issues.

Comment 1 by benhenry@chromium.org, Sep 1 2016

Status: Assigned (was: Untriaged)
Project Member

Comment 2 by bugdroid1@chromium.org, Sep 7 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/11daeac05bb27ba6a5322cb653e06285fcff7040

commit 11daeac05bb27ba6a5322cb653e06285fcff7040
Author: dnj <dnj@chromium.org>
Date: Wed Sep 07 19:15:07 2016

LogDog/Butler: Enable direct output forwarding.

Currently, Butler supports either passing a subprocess' STDOUT/STDERR
directly as a LogDog stream or completely ignoring them. This adds a
third option: forwarding them through the Butler's STDOUT/STDERR.

BUG= chromium:643243 
TEST=local
  - Ran locally, seems to work.

Review-Url: https://codereview.chromium.org/2317223002

[modify] https://crrev.com/11daeac05bb27ba6a5322cb653e06285fcff7040/logdog/client/cmd/logdog_butler/subcommand_run.go
Project Member

Comment 3 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/c88b313d2fae775acf973229bca7a00d45178781

commit c88b313d2fae775acf973229bca7a00d45178781
Author: Dan Jacques <dnj@chromium.org>
Date: Wed Sep 07 20:09:23 2016

Roll luci-go.

11daeac LogDog/Butler: Enable direct output forwarding.
6f90e9f Add counter-write and counter-read commands.
1eff02f Milo: pRPC endpoint for getting Buildbot master data

BUG= chromium:643243 
TEST=None

Change-Id: Ia224e702288ca5f08da35bb5d79a28f804bc411d
Reviewed-on: https://chromium-review.googlesource.com/382331
Reviewed-by: Benjamin Pastene <bpastene@google.com>
Commit-Queue: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/c88b313d2fae775acf973229bca7a00d45178781/DEPS
Project Member

Comment 5 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/1332a3e969641cf68df83cd9cb2f409e50385cea

commit 1332a3e969641cf68df83cd9cb2f409e50385cea
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 16:33:56 2016
Project Member

Comment 6 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/369404a67b7f7c497e36408680ad93bae9655766

commit 369404a67b7f7c497e36408680ad93bae9655766
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 16:39:46 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/0d5b7bd87099c2c5d5590c13116c6e3aa7d2c408 Canary new LogDog Butler on Infra. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2324813002
Cr-Commit-Position: refs/heads/master@{#417311}

[modify] https://crrev.com/369404a67b7f7c497e36408680ad93bae9655766/infra/config/recipes.cfg
Project Member

Comment 7 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/5922b70563f5080a3051ff87bd294477a77d71d8

commit 5922b70563f5080a3051ff87bd294477a77d71d8
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 17:13:38 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/0d5b7bd87099c2c5d5590c13116c6e3aa7d2c408 Canary new LogDog Butler on Infra. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2319403003

[modify] https://crrev.com/5922b70563f5080a3051ff87bd294477a77d71d8/infra/config/recipes.cfg
Project Member

Comment 9 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/48b323274009a26ed6b5d982571ada1a819bf29f

commit 48b323274009a26ed6b5d982571ada1a819bf29f
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 17:54:57 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/395b994270c26b85198b4ab921880d21b6fc4d9d Fix "chromium.infra" LogDog canarying. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2328463003

[modify] https://crrev.com/48b323274009a26ed6b5d982571ada1a819bf29f/infra/config/recipes.cfg
Project Member

Comment 10 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/41f725484babc76cbda92eae9366e74a681db769

commit 41f725484babc76cbda92eae9366e74a681db769
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 18:34:11 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/395b994270c26b85198b4ab921880d21b6fc4d9d Fix "chromium.infra" LogDog canarying. (dnj@chromium.org)
  https://crrev.com/156cd97e26e5aaca148b26fcf70371e30eac4ebf DevTools: Add devtools for tryserver.chromium.linux (chenwilliam@chromium.org)
  https://crrev.com/44ac4f64965b436b43201bfe7e359f73ec073b82 Updating the low end bots to a vm that triggers the job. (eyaich@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 , 642046 ,chromium:633253

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2322773003
Cr-Commit-Position: refs/heads/master@{#417350}

[modify] https://crrev.com/41f725484babc76cbda92eae9366e74a681db769/infra/config/recipes.cfg
Project Member

Comment 11 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/1f827fb3b6b4ce5185228c00ddbbe0fe53a12fa8

commit 1f827fb3b6b4ce5185228c00ddbbe0fe53a12fa8
Author: dnj <dnj@chromium.org>
Date: Thu Sep 08 19:24:18 2016

Canary seems to be working, bump actual LogDog.

BUG= chromium:643243 
TEST=canary

Review-Url: https://codereview.chromium.org/2322053002

[modify] https://crrev.com/1f827fb3b6b4ce5185228c00ddbbe0fe53a12fa8/scripts/slave/logdog_bootstrap.py
Project Member

Comment 12 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/8b07fecd105a2d4a67502ded434a5b180f8efae5

commit 8b07fecd105a2d4a67502ded434a5b180f8efae5
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 19:29:18 2016
Project Member

Comment 13 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f93ad8eba52adf66f5b9d3d7e5a449841a1fd578

commit f93ad8eba52adf66f5b9d3d7e5a449841a1fd578
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 19:35:29 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/1f827fb3b6b4ce5185228c00ddbbe0fe53a12fa8 Canary seems to be working, bump actual LogDog. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2324653004
Cr-Commit-Position: refs/heads/master@{#417364}

[modify] https://crrev.com/f93ad8eba52adf66f5b9d3d7e5a449841a1fd578/infra/config/recipes.cfg
Project Member

Comment 14 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-go.git/+/e1a482ae35e68cdbcb5385085e2d9a34219a50b8

commit e1a482ae35e68cdbcb5385085e2d9a34219a50b8
Author: dnj <dnj@chromium.org>
Date: Thu Sep 08 20:07:35 2016

LogDog/Archivist: Conditionally render data.

When archiving log stream data, LogDog's Archivist supports emitting a
rendered data stream, which creates a flat file containing all of the
log stream's data. This is useful for direct consumption, but not
required. Because it doubles the amount of archival space required, it
is gated on two conditions:

- The project specifies that streams should always be rendered, or
- The stream requests to be rendered to binary by providing an
  extension.

Currently, these conditions are ignored, and the stream is always
rendered. This patch makes the Archivist actually pay attention to this
configuration and conditionally render the stream.

BUG= chromium:643243 
TEST=unittest

Review-Url: https://codereview.chromium.org/2321173002

[modify] https://crrev.com/e1a482ae35e68cdbcb5385085e2d9a34219a50b8/logdog/server/archivist/archivist.go
[modify] https://crrev.com/e1a482ae35e68cdbcb5385085e2d9a34219a50b8/logdog/server/archivist/archivist_test.go
Project Member

Comment 15 by bugdroid1@chromium.org, Sep 8 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/813f30a8371f6512eafbf2e3bbf3526347d62256

commit 813f30a8371f6512eafbf2e3bbf3526347d62256
Author: recipe-roller <recipe-roller@chromium.org>
Date: Thu Sep 08 20:13:59 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/1f827fb3b6b4ce5185228c00ddbbe0fe53a12fa8 Canary seems to be working, bump actual LogDog. (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2326733002

[modify] https://crrev.com/813f30a8371f6512eafbf2e3bbf3526347d62256/infra/config/recipes.cfg
Project Member

Comment 16 by bugdroid1@chromium.org, Sep 9 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/4bc0aeff923a9b9b9ba1b4ea96525ef4ca5a6119

commit 4bc0aeff923a9b9b9ba1b4ea96525ef4ca5a6119
Author: dnj <dnj@chromium.org>
Date: Fri Sep 09 23:22:25 2016

Revert of Canary seems to be working, bump actual LogDog. (patchset #1 id:1 of https://codereview.chromium.org/2322053002/ )

Reason for revert:
LogDog not working, e.g.:

https://uberchromegw.corp.google.com/i/chromium/builders/Win/builds/46930

Original issue's description:
> Canary seems to be working, bump actual LogDog.
>
> BUG= chromium:643243 
> TEST=canary
>
> Committed: https://chromium.googlesource.com/chromium/tools/build/+/1f827fb3b6b4ce5185228c00ddbbe0fe53a12fa8

TBR=agable@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= chromium:643243 

Review-Url: https://codereview.chromium.org/2327103005

[modify] https://crrev.com/4bc0aeff923a9b9b9ba1b4ea96525ef4ca5a6119/scripts/slave/logdog_bootstrap.py
Project Member

Comment 17 by bugdroid1@chromium.org, Sep 9 2016 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/113076135e5adf13cd38aeee620e33b440dace0c

commit 113076135e5adf13cd38aeee620e33b440dace0c
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Sep 09 23:33:49 2016
Project Member

Comment 18 by bugdroid1@chromium.org, Sep 9 2016 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3c83f86fa4c5343193d65fc807582e421cecf809

commit 3c83f86fa4c5343193d65fc807582e421cecf809
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Sep 09 23:42:39 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/4bc0aeff923a9b9b9ba1b4ea96525ef4ca5a6119 Revert of Canary seems to be working, bump actual LogDog. (patchset #1 id:1 of https://codereview.chromium.org/2322053002/ ) (dnj@chromium.org)

TBR=martiniss@chromium.org,phajdan.jr@chromium.org
BUG= chromium:643243 

Recipe-Tryjob-Bypass-Reason: Autoroller
Bugdroid-Send-Email: False
Review-Url: https://codereview.chromium.org/2327313002
Cr-Commit-Position: refs/heads/master@{#417766}

[modify] https://crrev.com/3c83f86fa4c5343193d65fc807582e421cecf809/infra/config/recipes.cfg

Comment 19 by benhenry@chromium.org, Jan 18 2017 

Ping - please provide an update to your high priority bug. This bug is stale. Is it really P-1?

Comment 20 by hinoka@chromium.org, Sep 26

Status: Archived (was: Assigned)
Will revisit individual issues and optimize only if needed.
►

Sign in to add a comment