New issue
Advanced search Search tips

Issue 719843 link

Starred by 2 users
Status: Fixed
Owner:
shinyak@chromium.org
Closed: Feb 2018
Cc:
akes...@chromium.org
derat@chromium.org
dgarr...@chromium.org
shinyak@chromium.org
hidehiko@chromium.org
ukai@chromium.org
tikuta@chromium.org
yyanagisawa@chromium.org
sque@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

GOMA uploads too many gomacc.*.INFO.* files to server, which is slow.

Project Member Reported by hidehiko@chromium.org, May 9 2017 
In case of some unexpected failing case,
gomacc outputs gomacc.*.INFO.* log files per process, which means we have many gomacc log files.

For future investigation, these are being uploaded to cloud storage.
However, it can take time (>10mins).
Practically, those files are small, so we'd probably want to archive them, and upload it as one .tgz file, to reduce uploading duration.

Note: According to offline chat with shinyak@, in some cases, only a few gomacc log file may contain important info, so filtering won't work and it is necessary to upload all the files.

This is turned out during the goma support on ChromeOS bot, but it is not limited to ChromeOS.
Project Member

Comment 1 by bugdroid1@chromium.org, May 11 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7d62f59e99797c3f023df46680c9d8193c80949e

commit 7d62f59e99797c3f023df46680c9d8193c80949e
Author: Hidehiko Abe <hidehiko@chromium.org>
Date: Thu May 11 16:40:28 2017

Fix up goma log uploading.

This contains two goma log upload fixes.
1) Temporarily disable gomacc log uploading.
  Currently, uploading gomacc files sometimes takes very long time,
  like +10 mins. According to goma team, practically, in most cases,
  compiler_proxy logs are enough for debugging, although gomacc logs
  are needed for edge cases.
  Considering the tradeoff, this CL temporarily disables the gomacc
  log uploading, until upload time issue is resolved.
2) Propagate BUILDBOT names to chroot for annotation.
  The names were not propagated, so the annotated names were empty.

BUG=chromium:279618, chromium:719843 
TEST=Ran bots.

Change-Id: I01c0ee7629181472efe9bf847356829a390b5045
Reviewed-on: https://chromium-review.googlesource.com/499970
Commit-Ready: Hidehiko Abe <hidehiko@chromium.org>
Tested-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>

[modify] https://crrev.com/7d62f59e99797c3f023df46680c9d8193c80949e/scripts/upload_goma_info_unittest.py
[modify] https://crrev.com/7d62f59e99797c3f023df46680c9d8193c80949e/cbuildbot/stages/build_stages.py
[modify] https://crrev.com/7d62f59e99797c3f023df46680c9d8193c80949e/scripts/upload_goma_info.py

Comment 2 by tfarina@chromium.org, May 12 2017

Owner: hidehiko@chromium.org
Status: Assigned (was: Untriaged)

Comment 3 by shinyak@chromium.org, Oct 4 2017

Owner: shinyak@chromium.org
taking
Project Member

Comment 4 by bugdroid1@chromium.org, Oct 10 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0890b0faa4b9551788d3f18f0ea26e591137726e

commit 0890b0faa4b9551788d3f18f0ea26e591137726e
Author: Shinya Kawanaka <shinyak@chromium.org>
Date: Tue Oct 10 12:31:03 2017

Upload gomacc logs

Uploading gomacc logs was disabled before (revision 7d62f59) since
the number of gomacc logs is large and uploading one by one takes
long time.

In this patch, we make a tarball for gomacc logs, and upload it.

BUG= chromium:719843 
TEST=Ran goma_util_unittest

Change-Id: Ie0fe37ff34b8e4b8f163b3ba798ef67883d571b2
Reviewed-on: https://chromium-review.googlesource.com/702082
Commit-Ready: Shinya Kawanaka <shinyak@chromium.org>
Tested-by: Shinya Kawanaka <shinyak@chromium.org>
Reviewed-by: Hidehiko Abe <hidehiko@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/0890b0faa4b9551788d3f18f0ea26e591137726e/cbuildbot/goma_util.py
[modify] https://crrev.com/0890b0faa4b9551788d3f18f0ea26e591137726e/cbuildbot/goma_util_unittest.py

Comment 5 by shinyak@chromium.org, Feb 28 2018

Status: Fixed (was: Assigned)
now gomacc logs are uploaded with zipping
►

Sign in to add a comment