New issue
Advanced search Search tips

Issue 698968 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Sep 26
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Switch all masters to LogDog-only mode.

Project Member Reported by d...@chromium.org, Mar 7 2017

Issue description

This is a top-level bug for switching all BuildBot master logging over to LogDog-only mode. The advantages of this are a unified logging system and significant relief to the I/O and CPU of each master process.

Current state:
- LogDog-only mode is mature. It has been successfully running on several masters for months without major issue.
- The public waterfall template UI and Milo UI was recently updated to make LogDog-only mode not show junk links: https://screenshot.googleplex.com/eZXKEfdnXZu.png
- Several masters are already switched: https://chromium.googlesource.com/chromium/tools/build/+/b2df1caa8ced6d94b8c91decfe53bc000c48fa7b/scripts/slave/logdog-params.pyl

The remainder of the switch will be a combination of opportunity and need, based on master load and suitability.
 
Owner: d...@chromium.org
Status: Assigned (was: Untriaged)
One thing I really miss with LogDog is a quick way to download logs from a URL using my favorite command like HTTP tool (IE wget/curl). With LogDog only mode getting https://bugs.chromium.org/p/chromium/issues/detail?id=698429 fixed seems like it might be more important?
Higher up in that bug are the commands to get the logs, i.e.:

mkdir $LOGDOG_CIPD_ROOT
cd $LOGDOG_CIPD_ROOT
/path/to/depot_tools/cipd init
/path/to/depot_tools/cipd install infra/tools/luci/logdog/logdog/linux-amd64

At this point, you will have the "logdog" binary available, and can run it:

./logdog cat <path>

<path> can be one of two things:
1) A <project>/<path> tuple, like the one you would see the viewer: https://screenshot.googleplex.com/wfmB4a05Dip

e.g.:
./logdog cat chromium/bb/chromium.fyi/CrWin7Goma_dbg_/32153/+/recipes/steps/taskkill/0/stdout

2) A LogDog URL, like the one you would see in a build property: https://screenshot.googleplex.com/pQL9CJGRLvD

./logdog cat logdog://luci-logdog.appspot.com/chromium/bb/chromium.fyi/CrWin7Goma_dbg_/32153/+/recipes/annotations


We'll make this less manual but no promises (yet) to have straight HTTP programmatic access.
Following those instructions is a real step backwards -- it's gone from using an existing tools everyone is familiar with (curl <url>), to having to learn multiple steps and about things like cipd, logdog CLI, project/path tuples and other random stuff.
Yup, we do know we want to make something better here.
that said, I don't know how many people really use curl to fetch things vs. "save as"

Comment 7 by d...@chromium.org, Mar 8 2017

The plan is to have "logdog" CLI install into "depot_tools", so every developer will have it on their path. At that point, it's a one-liner.

We are still working on a harness to deploy and maintain binaries in "depot_tools", though, so this isn't in place yet.
Alright, you can try running this instead:

cit logdog cat logdog://luci-logdog.appspot.com/chromium/bb/chromium.fyi/CrWin7Goma_dbg_/32153/+/recipes/annotations

It'll spend a couple minutes doing basically a full gclient sync, but after that should be fast.
Project Member

Comment 10 by bugdroid1@chromium.org, Mar 9 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c1c68b5a49e8a9ddc42193acb95d9d86fca469c3

commit c1c68b5a49e8a9ddc42193acb95d9d86fca469c3
Author: Dirk Pranke <dpranke@chromium.org>
Date: Thu Mar 09 18:55:08 2017

Make the remaining Chromium builders LogDog-only.

This CL switches the builders on the following masters to LogDog-only
(i.e., the buildbot logs on the master will now just be stubs):

- chromium.android
- chromium.android.fyi
- chromium.chrome
- chromium.chromedriver
- chromium.chromiumos
- chromium.goma
- chromium.lkgr
- chromium.swarm
- tryserver.blink
- tryserver.chromium.android
- tryserver.chromium.angle
- tryserver.chromium.mac
- tryserver.chromium.win

After this CL lands, all of the public Chromium builders will be
LogDog-only.

R=dnj@chromium.org
BUG= 698968 

Change-Id: I8137e843df7927d89f69f378835d78bda97f1e49
Reviewed-on: https://chromium-review.googlesource.com/452677
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/c1c68b5a49e8a9ddc42193acb95d9d86fca469c3/scripts/slave/logdog-params.pyl

Project Member

Comment 11 by bugdroid1@chromium.org, Mar 9 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/c1c68b5a49e8a9ddc42193acb95d9d86fca469c3

commit c1c68b5a49e8a9ddc42193acb95d9d86fca469c3
Author: Dirk Pranke <dpranke@chromium.org>
Date: Thu Mar 09 18:55:08 2017

Make the remaining Chromium builders LogDog-only.

This CL switches the builders on the following masters to LogDog-only
(i.e., the buildbot logs on the master will now just be stubs):

- chromium.android
- chromium.android.fyi
- chromium.chrome
- chromium.chromedriver
- chromium.chromiumos
- chromium.goma
- chromium.lkgr
- chromium.swarm
- tryserver.blink
- tryserver.chromium.android
- tryserver.chromium.angle
- tryserver.chromium.mac
- tryserver.chromium.win

After this CL lands, all of the public Chromium builders will be
LogDog-only.

R=dnj@chromium.org
BUG= 698968 

Change-Id: I8137e843df7927d89f69f378835d78bda97f1e49
Reviewed-on: https://chromium-review.googlesource.com/452677
Commit-Queue: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Daniel Jacques <dnj@chromium.org>

[modify] https://crrev.com/c1c68b5a49e8a9ddc42193acb95d9d86fca469c3/scripts/slave/logdog-params.pyl

Status: Fixed (was: Assigned)
pretty sure this is done

Sign in to add a comment