New issue
Advanced search Search tips

Issue 747790 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Sep 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Upload ninja log for Goma on ChromeOS bots.

Project Member Reported by hidehiko@chromium.org, Jul 24 2017

Issue description

This will help to debug in case of mysterious chrome build errors.

 
Status: Started (was: Assigned)

Comment 2 by vapier@chromium.org, Aug 17 2017

seems like we should implement  issue 709532  instead, and then you get this "for free"
This is to upload logs to the goma's server where we upload Chrome's to.
It has some effort of visualization? Any thoughts, Shinya?

This uploads ninja log to gs, then we can have visualized tracing data.

example from chrome: 
- http://chromium-build-stats.appspot.com/ninja_log/2017/08/18/build11-m1/ninja_log.build11-m1.chrome-bot.20170818-100041.7452.gz
- trace viewer: http://chromium-build-stats.appspot.com/ninja_log/2017/08/18/build11-m1/ninja_log.build11-m1.chrome-bot.20170818-100041.7452.gz/trace.html

This is to debug "build" itself, and to analyze "build performance".
When build is low with goma, there are many possibilities: (a) one compile makes the whole build slow, (b) all compiles are equally slow, (c) link is just slow, (d) fork/exec is slow, (e) python script is slow, and so on.
When debugging build slowness, sometimes it is not enough just to have goma's compiler_proxy log. We sometimes need ninja log.

Comment 5 by vapier@chromium.org, Aug 18 2017

OK, if it's for ingesting data into another pipeline and it wants a specific format
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0ee0ee29e5cb2a2b2c2410324d528f4c477f9195

commit 0ee0ee29e5cb2a2b2c2410324d528f4c477f9195
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Tue Aug 29 09:39:08 2017

Upload ninja_log.

This CL introduces a feature to upload ninja_log and its
related meta info.
On build, ninja_log file should be created in goma's log
directory. Also, related meta info will be written into
ninja_* files.

BUG= chromium:747790 
TEST=Ran on bots.

Change-Id: Id8bb4b7851dc42e994c924f076e9bb2f41122fef
Reviewed-on: https://chromium-review.googlesource.com/618218
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/0ee0ee29e5cb2a2b2c2410324d528f4c477f9195/cbuildbot/goma_util.py
[modify] https://crrev.com/0ee0ee29e5cb2a2b2c2410324d528f4c477f9195/cbuildbot/goma_util_unittest.py
[modify] https://crrev.com/0ee0ee29e5cb2a2b2c2410324d528f4c477f9195/cbuildbot/stages/report_stages.py

Project Member

Comment 7 by bugdroid1@chromium.org, Aug 29 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/4f68b00a22f6724ed1825f386fe6096cc14cf1c2

commit 4f68b00a22f6724ed1825f386fe6096cc14cf1c2
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Tue Aug 29 09:39:07 2017

chromeos-chrome: Record ninja related info for logging.

This is preparation to upload ninja_log file to
the server. The command line, cwd and env vars will be
annotated to the meta.

BUG= chromium:747790 
TEST=Ran locally. Ran on the server.

Change-Id: Ie0cb495e2befef9a44b386b3a587051ed8f0e36e
Reviewed-on: https://chromium-review.googlesource.com/618219
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/4f68b00a22f6724ed1825f386fe6096cc14cf1c2/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild

Project Member

Comment 8 by bugdroid1@chromium.org, Sep 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/d923dfad62013fd61409a1ea537703877042d913

commit d923dfad62013fd61409a1ea537703877042d913
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Tue Sep 12 08:12:06 2017

Record ninja related info in TestSimpleChromeWorkflowStage.

Then, those info will be uploaded in a later stage.

BUG= chromium:747790 
TEST=Ran on bots.

Change-Id: Iffd89e264f93eddb6d23522aded638a294a97c39
Reviewed-on: https://chromium-review.googlesource.com/648913
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/d923dfad62013fd61409a1ea537703877042d913/cbuildbot/commands.py
[modify] https://crrev.com/d923dfad62013fd61409a1ea537703877042d913/cbuildbot/stages/chrome_stages.py

Status: Fixed (was: Started)
Done.

Comment 10 by dchan@chromium.org, Jan 22 2018

Status: Archived (was: Fixed)

Comment 11 by dchan@chromium.org, Jan 23 2018

Status: Fixed (was: Archived)

Sign in to add a comment