New issue
Advanced search Search tips

Issue 734896 link

Starred by 1 user
Status: Fixed
Owner:
shinyak@chromium.org
Closed: Jul 2017
Cc:
ukai@chromium.org
tikuta@chromium.org
yyanagisawa@chromium.org
shinyak@chromium.org
sque@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

compress gomacc logs when upload (for chrome)

Project Member Reported by shinyak@chromium.org, Jun 20 2017 
Derived from  Issue 719843 

This is for chrome builtbot version.

When a lot of gomacc logs are generated, it takes too long time to upload gomacc logs. Compress them to one file and upload it.

Comment 1 by shinyak@chromium.org, Jun 20 2017

Owner: shinyak@chromium.org
Project Member

Comment 2 by bugdroid1@chromium.org, Jun 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3b86d055d9ad254c1ccadd1820d49216cc4d5895

commit 3b86d055d9ad254c1ccadd1820d49216cc4d5895
Author: Shinya Kawanaka <shinyak@google.com>
Date: Wed Jun 21 05:53:25 2017

Zip gomacc logs before upload

When gomacc logs are generated, the number of the files is large,
often 100. Uploading one by one takes 10~20 minutes, it's too slow.

Zip them and upload the zipped file.

BUG= 734896 

Change-Id: Ib409e909633a8b600472cb1e223213677678fb10
Reviewed-on: https://chromium-review.googlesource.com/541181
Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@google.com>

[modify] https://crrev.com/3b86d055d9ad254c1ccadd1820d49216cc4d5895/scripts/slave/goma_utils.py
Project Member

Comment 3 by bugdroid1@chromium.org, Jun 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107

commit ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Jun 21 21:27:54 2017

Revert "Zip gomacc logs before upload"

This reverts commit 3b86d055d9ad254c1ccadd1820d49216cc4d5895.

Reason for revert: Looks to be causing failures on perf
https://build.chromium.org/p/chromium.perf/builders/Win%20Builder

Original change's description:
> Zip gomacc logs before upload
> 
> When gomacc logs are generated, the number of the files is large,
> often 100. Uploading one by one takes 10~20 minutes, it's too slow.
> 
> Zip them and upload the zipped file.
> 
> BUG= 734896 
> 
> Change-Id: Ib409e909633a8b600472cb1e223213677678fb10
> Reviewed-on: https://chromium-review.googlesource.com/541181
> Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>

TBR=ukai@chromium.org,dpranke@chromium.org,yyanagisawa@chromium.org,shinyak@chromium.org,tikuta@google.com,tikuta@chromium.org

Change-Id: I4c094b76e2bede6a18e0002153c8c4da360c10fe
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  734896 ,  735651 
Reviewed-on: https://chromium-review.googlesource.com/544096
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107/scripts/slave/goma_utils.py
Project Member

Comment 4 by bugdroid1@chromium.org, Jun 21 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107

commit ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Jun 21 21:27:54 2017

Revert "Zip gomacc logs before upload"

This reverts commit 3b86d055d9ad254c1ccadd1820d49216cc4d5895.

Reason for revert: Looks to be causing failures on perf
https://build.chromium.org/p/chromium.perf/builders/Win%20Builder

Original change's description:
> Zip gomacc logs before upload
> 
> When gomacc logs are generated, the number of the files is large,
> often 100. Uploading one by one takes 10~20 minutes, it's too slow.
> 
> Zip them and upload the zipped file.
> 
> BUG= 734896 
> 
> Change-Id: Ib409e909633a8b600472cb1e223213677678fb10
> Reviewed-on: https://chromium-review.googlesource.com/541181
> Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
> Reviewed-by: Takuto Ikuta <tikuta@google.com>

TBR=ukai@chromium.org,dpranke@chromium.org,yyanagisawa@chromium.org,shinyak@chromium.org,tikuta@google.com,tikuta@chromium.org

Change-Id: I4c094b76e2bede6a18e0002153c8c4da360c10fe
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  734896 ,  735651 
Reviewed-on: https://chromium-review.googlesource.com/544096
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/ca9c93fa7b512d6c1ecdf6b27f029b7137ed9107/scripts/slave/goma_utils.py

Comment 5 by shinyak@chromium.org, Jun 22 2017 

Windows!

https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_Builder%2F86238%2F%2B%2Frecipes%2Fsteps%2Fpostprocess_for_goma%2F0%2Fsteps%2Fupload_log%2F0%2Fstdout

C:\b\depot_tools\python276_bin\python.exe C:\b\rr\tmpexjtuv\rw\checkout\scripts\slave\.recipe_deps\depot_tools\gsutil.py -- -h "x-goog-meta-builderinfo:{\"slave\": \"build121-m1\", \"builder\": \"Win Builder\", \"master\": \"chromium.perf\", \"os\": \"win32\", \"clobber\": false}" cp file://c:\users\chrome~1\appdata\local\temp\tmpzmbqqg gs://chrome-goma-log/2017/06/21/build121-m1/gomacc.exe.BUILD121-M1.invalid-user.log.INFO.20170621-142515.8012.zip.gz
Copying file://c:\users\chrome~1\appdata\local\temp\tmpzmbqqg [Content-Type=application/octet-stream]...
Uploading   ...nvalid-user.log.INFO.20170621-142515.8012.zip.gz: 0 B/1.66 KiB    
Uploading   ...nvalid-user.log.INFO.20170621-142515.8012.zip.gz: 1.66 KiB/1.66 KiB    
Traceback (most recent call last):
  File "C:\b\rr\tmpexjtuv\rw\checkout\scripts\slave\upload_goma_logs.py", line 151, in <module>
Copied log file to gs://chrome-goma-log/2017/06/21/build121-m1/gomacc.exe.BUILD121-M1.invalid-user.log.INFO.20170621-142515.8012.zip.gz
    sys.exit(main())
  File "C:\b\rr\tmpexjtuv\rw\checkout\scripts\slave\upload_goma_logs.py", line 92, in main
    override_gsutil=override_gsutil
  File "C:\b\rr\tmpexjtuv\rw\checkout\scripts\slave\goma_utils.py", line 210, in UploadGomaCompilerProxyInfo
    override_gsutil=override_gsutil)
  File "C:\b\rr\tmpexjtuv\rw\checkout\scripts\slave\goma_utils.py", line 237, in UploadGomaccInfo
    os.remove(zipfile_path.name)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\users\\chrome~1\\appdata\\local\\temp\\tmpwr0imx'
step returned non-zero exit code: 1
Project Member

Comment 6 by bugdroid1@chromium.org, Jun 23 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/58bbd3825903ac987c8bef30f5f47c7b3ce51bed

commit 58bbd3825903ac987c8bef30f5f47c7b3ce51bed
Author: Shinya Kawanaka <shinyak@google.com>
Date: Fri Jun 23 02:28:40 2017

Zip gomacc logs before upload (retry)

When gomacc logs are generated, the number of the files is large,
often 100. Uploading one by one takes 10~20 minutes, it's too slow.

Zip them and upload the zipped file.

BUG= 734896 

Change-Id: I5fe4cfadd1ebc8e875900318ef0fc92acabdb181
Reviewed-on: https://chromium-review.googlesource.com/544575
Reviewed-by: Takuto Ikuta <tikuta@google.com>
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>

[modify] https://crrev.com/58bbd3825903ac987c8bef30f5f47c7b3ce51bed/scripts/slave/goma_utils.py
Project Member

Comment 7 by bugdroid1@chromium.org, Jun 23 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/35104c1f2a82940899c2a678e13bec5b6dfda575

commit 35104c1f2a82940899c2a678e13bec5b6dfda575
Author: Shinya Kawanaka <shinyak@chromium.org>
Date: Fri Jun 23 07:24:17 2017

Revert "Zip gomacc logs before upload (retry)"

This reverts commit 58bbd3825903ac987c8bef30f5f47c7b3ce51bed.

Reason for revert: Win Builder is raising exception :(

Original change's description:
> Zip gomacc logs before upload (retry)
> 
> When gomacc logs are generated, the number of the files is large,
> often 100. Uploading one by one takes 10~20 minutes, it's too slow.
> 
> Zip them and upload the zipped file.
> 
> BUG= 734896 
> 
> Change-Id: I5fe4cfadd1ebc8e875900318ef0fc92acabdb181
> Reviewed-on: https://chromium-review.googlesource.com/544575
> Reviewed-by: Takuto Ikuta <tikuta@google.com>
> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
> Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>

TBR=ukai@chromium.org,dpranke@chromium.org,yyanagisawa@chromium.org,shinyak@chromium.org,tikuta@google.com,tikuta@chromium.org

Change-Id: I383b7ec43caff8cbaf6cac7a1d8d86f7cf382bfd
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  734896 
Reviewed-on: https://chromium-review.googlesource.com/544730
Reviewed-by: Shinya Kawanaka <shinyak@chromium.org>
Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>

[modify] https://crrev.com/35104c1f2a82940899c2a678e13bec5b6dfda575/scripts/slave/goma_utils.py

Comment 8 by shinyak@chromium.org, Jun 23 2017 

Hmm, still raising exception. I've reverted the patch :/

https://build.chromium.org/p/chromium.perf/builders/Win%20Builder/builds/86756

Copied log file to gs://chrome-goma-log/2017/06/23/build63-m1/compiler_proxy.exe.BUILD63-M1.chrome-bot.log.INFO.20170622-224841.6692.gz
Visualization at http://chromium-build-stats.appspot.com/compiler_proxy_log/2017/06/23/build63-m1/compiler_proxy.exe.BUILD63-M1.chrome-bot.log.INFO.20170622-224841.6692.gz
Traceback (most recent call last):
  File "C:\b\rr\tmpxrd6g8\rw\checkout\scripts\slave\upload_goma_logs.py", line 151, in <module>
    sys.exit(main())
  File "C:\b\rr\tmpxrd6g8\rw\checkout\scripts\slave\upload_goma_logs.py", line 92, in main
    override_gsutil=override_gsutil
  File "C:\b\rr\tmpxrd6g8\rw\checkout\scripts\slave\goma_utils.py", line 210, in UploadGomaCompilerProxyInfo
    override_gsutil=override_gsutil)
  File "C:\b\rr\tmpxrd6g8\rw\checkout\scripts\slave\goma_utils.py", line 234, in UploadGomaccInfo
    override_gsutil=override_gsutil)
  File "C:\b\rr\tmpxrd6g8\rw\checkout\scripts\slave\goma_utils.py", line 146, in UploadToGomaLogGS
    with open(file_path) as f_in:
IOError: [Errno 13] Permission denied: 'c:\\users\\chrome~1\\appdata\\local\\temp\\tmpz1q9bm'
step returned non-zero exit code: 1
Project Member

Comment 9 by bugdroid1@chromium.org, Jul 10 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/3f3d5bbe0e3e450c66be29f35b885d1556b7be5b

commit 3f3d5bbe0e3e450c66be29f35b885d1556b7be5b
Author: Shinya Kawanaka <shinyak@google.com>
Date: Mon Jul 10 03:16:20 2017

Archive gomacc logs before upload (3rd try)

When gomacc logs are generated, the number of the files is large,
often 100. Uploading one by one takes 10~20 minutes, it's too slow.

Make a tarfile and upload the tar.gz file.

The previous failures were because Windows cannot re-open the opened
tempfile. We should have closed the tempfile before re-opening it.

Bug:  734896 
Change-Id: If0c2b183a819fe3646615ccdbb72e2ebe30a5dad
Reviewed-on: https://chromium-review.googlesource.com/561032
Commit-Queue: Shinya Kawanaka <shinyak@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>

[modify] https://crrev.com/3f3d5bbe0e3e450c66be29f35b885d1556b7be5b/scripts/slave/goma_utils.py

Comment 10 by shinyak@chromium.org, Jul 26 2017

Status: Fixed (was: Untriaged)
►

Sign in to add a comment