When goma module uploads ninjalog in chromium bot, we can investigate what happens in slow compile steps. Storing ninja_log from chromium builders to BigQuery so that we can find slow build tasks and make our effort for such tasks visible/trackable.
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/cb1240abde5c417405ef8a5c804580f851d8a9b6 commit cb1240abde5c417405ef8a5c804580f851d8a9b6 Author: Namiko Toriyama <namiko@google.com> Date: Thu Aug 16 04:31:37 2018 make proto file for ninjalog this protofile is used to send to BigQuery Bug:873940 Change-Id: I804a6f0d3fc2569675af1b84a4fd5a02200f9111 Reviewed-on: https://chromium-review.googlesource.com/1173934 Commit-Queue: Namiko Toriyama <namiko@google.com> Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> [add] https://crrev.com/cb1240abde5c417405ef8a5c804580f851d8a9b6/appengine/chromium_build_stats/gopath/src/ninjalog/ninjalog.proto [add] https://crrev.com/cb1240abde5c417405ef8a5c804580f851d8a9b6/appengine/chromium_build_stats/gopath/src/ninjalog/ninjalog.pb.go [add] https://crrev.com/cb1240abde5c417405ef8a5c804580f851d8a9b6/appengine/chromium_build_stats/Makefile [modify] https://crrev.com/cb1240abde5c417405ef8a5c804580f851d8a9b6/DEPS [add] https://crrev.com/cb1240abde5c417405ef8a5c804580f851d8a9b6/appengine/chromium_build_stats/gopath/src/ninjalog/gen.go
marged.(forget adding bugs number) Add make file command https://chromium-review.googlesource.com/c/infra/infra/+/1174584/9 for deploy to GAE, go build, go test Change-Id: I49427399334ac5823c1e785af41fc1d487fce632 Reviewed-on: https://chromium-review.googlesource.com/1174584 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com>
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/aabaa281660523b51690bd8a998ce4ac0ecf1522 commit aabaa281660523b51690bd8a998ce4ac0ecf1522 Author: Namiko Toriyama <namiko@google.com> Date: Mon Aug 20 04:48:34 2018 receive a request and get GCS file content. fetch GCS file contents by using pub/sub notification. Bug:873940 Change-Id: Ifa8bf4718e89afc5f4c79a06f14fbbcd67329755 Reviewed-on: https://chromium-review.googlesource.com/1175654 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> [modify] https://crrev.com/aabaa281660523b51690bd8a998ce4ac0ecf1522/go/src/infra/appengine/chromium_build_stats/app.yaml [add] https://crrev.com/aabaa281660523b51690bd8a998ce4ac0ecf1522/go/src/infra/appengine/chromium_build_stats/pubsub.go
send protocol buffer to BigQuery convert ninjalog to protocol buffer and send the protocol buffer to BigQuery. Bug: 874722 Change-Id: I641c1d707ebe4fc05984dfb3f2a2f66a7cd691ae Reviewed-on: https://chromium-review.googlesource.com/1183204 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com>
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/a2f4c1a8f0d12228fac10d2152576959bb6dba12 commit a2f4c1a8f0d12228fac10d2152576959bb6dba12 Author: Takuto Ikuta <tikuta@chromium.org> Date: Thu Aug 23 02:28:23 2018 Add build id and stepname when uploading ninja log build id and stepname will be used to select specific builds in bigquery joined with buildbucket table. This is tested on https://ci.chromium.org/swarming/task/3f79d62015f06410 Uploaded metadata has build_id and step_name https://chromium-build-stats.appspot.com/ninja_log/2018/08/22/swarm1013-c4/ninja_log.swarm1013-c4.chrome-bot.20180822-025412.29701.gz/metadata.json Bug: 873940 Change-Id: I9edb861f9574bd96a6194c33e0eec7295dd3a946 Reviewed-on: https://chromium-review.googlesource.com/1184804 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> [modify] https://crrev.com/a2f4c1a8f0d12228fac10d2152576959bb6dba12/scripts/slave/upload_goma_logs.py [modify] https://crrev.com/a2f4c1a8f0d12228fac10d2152576959bb6dba12/scripts/slave/goma_utils.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/7f99e35777495e5e722a260a650721200ddbc405 commit 7f99e35777495e5e722a260a650721200ddbc405 Author: Namiko Toriyama <namiko@google.com> Date: Fri Aug 24 09:51:05 2018 send protocol buffer to BigQuery (continuation) convert ninjalog to protocol buffer and send the protocol buffer to BigQuery. continuation: https://chromium-review.googlesource.com/c/infra/infra/+/1183204 Bug: 873940 Change-Id: I18a4d2345b6bf937ec77071689ac8dbbf849dfa7 Reviewed-on: https://chromium-review.googlesource.com/1186011 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> [modify] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog_test.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/ninja_log.go [add] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjaproto.infra_testing [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjalog.pb.go [modify] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/Makefile [add] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/app.infra_testing [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/compiler_proxy_log_test.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/app.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/app.yaml [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/bigquery.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/file.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/ninjaproto/gen.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/ninja_log_test.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjalog.proto [modify] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/deps.yaml [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/pubsub.go [modify] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog.go [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/compiler_proxy_log.go [modify] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/deps.lock [rename] https://crrev.com/7f99e35777495e5e722a260a650721200ddbc405/go/src/infra/appengine/chromium_build_stats/app/auth.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e209170a74de43acdff283a6dabb5c81ab4ecc57 commit e209170a74de43acdff283a6dabb5c81ab4ecc57 Author: Namiko Toriyama <namiko@google.com> Date: Fri Sep 07 01:52:57 2018 add admin login to request URL Currently, everyone can access the url of pubsub http handler. For security, prevent external access. Only administrator can access the url. See https://cloud.google.com/pubsub/docs/push#app-engine-standard-endpoints Bug: 873940 Change-Id: Iaf7b8c25ea7e9ad916ae77b9c80822cf454afaf9 Reviewed-on: https://chromium-review.googlesource.com/1203338 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com> Cr-Commit-Position: refs/heads/master@{#17437} [modify] https://crrev.com/e209170a74de43acdff283a6dabb5c81ab4ecc57/go/src/infra/appengine/chromium_build_stats/app/pubsub.go [modify] https://crrev.com/e209170a74de43acdff283a6dabb5c81ab4ecc57/go/src/infra/appengine/chromium_build_stats/app/app.yaml
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/67fc4e7d6491df2ff38bd673ab8d122cdd2dd26d commit 67fc4e7d6491df2ff38bd673ab8d122cdd2dd26d Author: Namiko Toriyama <namiko@google.com> Date: Fri Sep 07 04:36:52 2018 change instance class The current app dies due to insufficient memory. The current instance class F2 has only 256MB. Let me use a bit powerful instance class F4. F4 has 512MB. After the change, we rarely run out of memory. See https://cloud.google.com/appengine/docs/standard/ for the difference of instance classes. https://cloud.google.com/appengine/pricing F4 price is $0.20/hour, where F2 price is $0.10/hour. Bug: 873940 Change-Id: Ie24b001aaa823121e5e46c935a7e20966fb84e4d Reviewed-on: https://chromium-review.googlesource.com/1203337 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Cr-Commit-Position: refs/heads/master@{#17441} [modify] https://crrev.com/67fc4e7d6491df2ff38bd673ab8d122cdd2dd26d/go/src/infra/appengine/chromium_build_stats/app/app.yaml
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/1e594bf385ecc2d33664c48cbef59c839c481fd8 commit 1e594bf385ecc2d33664c48cbef59c839c481fd8 Author: Namiko Toriyama <namiko@google.com> Date: Mon Sep 10 02:44:12 2018 enable to read gzip file ninjalog are uploaded in gzip compressed. add function to open gzip files for ninjalog. if file uploaded to GCS is not ninjalog, skip reading. Bug: 873940 Change-Id: I12f91f04bbff24bd0e761281ee016ed26801e78f Reviewed-on: https://chromium-review.googlesource.com/1203335 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com> Cr-Commit-Position: refs/heads/master@{#17471} [modify] https://crrev.com/1e594bf385ecc2d33664c48cbef59c839c481fd8/go/src/infra/appengine/chromium_build_stats/app/pubsub.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e085a53160fb4e60b7e77af5082b85d99a6dea86 commit e085a53160fb4e60b7e77af5082b85d99a6dea86 Author: Namiko Toriyama <namiko@google.com> Date: Wed Sep 12 01:01:51 2018 add bqschemaupdater command when convert proto type and reflect schema on BiqQuery table, we execute this command. this is document of bqschemaupdater command in detail https://chromium.googlesource.com/infra/luci/luci-go/+/master/tools/cmd/bqschemaupdater Bug: 873940 Change-Id: I7c061e2b645ab912735e6202e0896126af4fc00a Reviewed-on: https://chromium-review.googlesource.com/1215270 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com> Cr-Commit-Position: refs/heads/master@{#17518} [modify] https://crrev.com/e085a53160fb4e60b7e77af5082b85d99a6dea86/go/src/infra/appengine/chromium_build_stats/Makefile
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/dda40c3ae840102e8dba79fe45eb7f96c7c0fb77 commit dda40c3ae840102e8dba79fe45eb7f96c7c0fb77 Author: Namiko Toriyama <namiko@google.com> Date: Wed Sep 12 07:49:35 2018 modify outputs in ninjaTask Now, when there are more than two tasks, add one task to output and the other to outputs. If there is one task, adds to only output. we want to see all tasks in outputs. so, append output to outputs. Bug: 873940 Change-Id: Ia009afd29723a1b358c3b5e7c1b3508f40c6f341 Reviewed-on: https://chromium-review.googlesource.com/1218243 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Cr-Commit-Position: refs/heads/master@{#17520} [modify] https://crrev.com/dda40c3ae840102e8dba79fe45eb7f96c7c0fb77/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog.go [modify] https://crrev.com/dda40c3ae840102e8dba79fe45eb7f96c7c0fb77/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog_test.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/fbea50b06a31f1567aeb8e1a724519d2c3fa5fab commit fbea50b06a31f1567aeb8e1a724519d2c3fa5fab Author: Namiko Toriyama <namiko@google.com> Date: Thu Sep 13 09:20:51 2018 add struct member to metadata in ninjalog Currently, there are empty items in BigQuery table schema. Therefore, we add buildID, stepname and jobs to metadata. Bug: 873940 Change-Id: I9fdc04ba1f5c4a6ffd7758d1db44df7798244279 Reviewed-on: https://chromium-review.googlesource.com/1212624 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Cr-Commit-Position: refs/heads/master@{#17560} [modify] https://crrev.com/fbea50b06a31f1567aeb8e1a724519d2c3fa5fab/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog.go [modify] https://crrev.com/fbea50b06a31f1567aeb8e1a724519d2c3fa5fab/go/src/infra/appengine/chromium_build_stats/app/pubsub.go [modify] https://crrev.com/fbea50b06a31f1567aeb8e1a724519d2c3fa5fab/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog_test.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/965b8e9f4d8c37e5bd45c9a029a7e73f69ef3b3b commit 965b8e9f4d8c37e5bd45c9a029a7e73f69ef3b3b Author: Namiko Toriyama <namiko@google.com> Date: Thu Sep 13 09:33:05 2018 convert time duration type to float type Actually, we can use duration to query for BigQuery table. But duration makes query longer due to it has seconds and nanoseconds field. e.g. taking tasks took more than one seconds SELECT * FROM <table> WHERE (log_entry.end_duration.seconds + log_entry.start_duration.nanoseconds / 1000000000) - (log_entry.start_duration.seconds + log_entry.start_duration.nanoseconds / 1000000000) > 1 We can write above query like below if we use float. SELECT * FROM <table> WHERE log_entry.end_duration_sec - log_entry.start_duration_sec > 1 Bug: 873940 Change-Id: Ic58c15d734f14abf639db530d9bd0beac75409d9 Reviewed-on: https://chromium-review.googlesource.com/1218246 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Cr-Commit-Position: refs/heads/master@{#17561} [modify] https://crrev.com/965b8e9f4d8c37e5bd45c9a029a7e73f69ef3b3b/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog.go [modify] https://crrev.com/965b8e9f4d8c37e5bd45c9a029a7e73f69ef3b3b/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog_test.go [modify] https://crrev.com/965b8e9f4d8c37e5bd45c9a029a7e73f69ef3b3b/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjalog.proto [modify] https://crrev.com/965b8e9f4d8c37e5bd45c9a029a7e73f69ef3b3b/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjalog.pb.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/f46f28423db263d487d90a13bf9e9d23c0f9c9ca commit f46f28423db263d487d90a13bf9e9d23c0f9c9ca Author: Namiko Toriyama <namiko@google.com> Date: Fri Sep 14 08:30:42 2018 download files without decompressing gzip automatically When cloud storage knows file's content encoding is gzip, it unzips automatically. So, download files without decompressing gzip automatically. Cloud storage library is below. https://godoc.org/cloud.google.com/go/storage#ObjectHandle.ReadCompressed Bug: 873940 Change-Id: I13183b69d6926d9ab9839d35707ad6edf1ef7170 Reviewed-on: https://chromium-review.googlesource.com/1226573 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com> Cr-Commit-Position: refs/heads/master@{#17594} [modify] https://crrev.com/f46f28423db263d487d90a13bf9e9d23c0f9c9ca/go/src/infra/appengine/chromium_build_stats/app/pubsub.go
namiko@, staging project seems to running app engine instances more than we expect. Could you set some limitation for instance scaling and store data to BQ for only 10% of request? https://cloud.google.com/appengine/docs/standard/go/config/appref#scaling_elements
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/260d2abdf68e3ad5353514cce711c330c5a6a22e commit 260d2abdf68e3ad5353514cce711c330c5a6a22e Author: Namiko Toriyama <namiko@google.com> Date: Tue Sep 18 08:16:24 2018 set max instance of the project With many requests, so many instances start up in the GCP. The number of the instances sometimes exceeds 100. We set max instances to save the cost of the instance. Bug: 873940 Change-Id: Ic8cb9059280b543dc6baa2e6bc98ba378b85ebc2 Reviewed-on: https://chromium-review.googlesource.com/1229540 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Cr-Commit-Position: refs/heads/master@{#17657} [modify] https://crrev.com/260d2abdf68e3ad5353514cce711c330c5a6a22e/go/src/infra/appengine/chromium_build_stats/app/app.yaml
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/36f913a266b6e8498efd5ff2206b9692afe3501f commit 36f913a266b6e8498efd5ff2206b9692afe3501f Author: Namiko Toriyama <namiko@google.com> Date: Tue Sep 18 08:25:14 2018 enable to read long token Currently, if one line exceeds 64*1024, get an error. we can see the error in the end of ninjalog. Therefore, specify the buffer to use for scanner. godoc is below. https://golang.org/pkg/bufio/#Scanner.Buffer Bug: 873940 Change-Id: If59b6e8ebc9fec9473b55040e78728ba814e3e54 Reviewed-on: https://chromium-review.googlesource.com/1226576 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Cr-Commit-Position: refs/heads/master@{#17658} [modify] https://crrev.com/36f913a266b6e8498efd5ff2206b9692afe3501f/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/a8d4b04380c1ff6d3f8672bf04ab6632392b7611 commit a8d4b04380c1ff6d3f8672bf04ab6632392b7611 Author: Namiko Toriyama <namiko@google.com> Date: Wed Sep 19 07:53:32 2018 samples 10% of requests With a large number of requests, BigQuery processing becomes heavy. When there are many requests, receives more than 15 request per second. Reduce processing by sampling 10% of requests. This is both to save app engine cost and big query insertion/storage cost while we are experimenting. Bug: 873940 Change-Id: I266f3226d40d36e4a52ee14fa4ded5feb00599ce Reviewed-on: https://chromium-review.googlesource.com/1229544 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Cr-Commit-Position: refs/heads/master@{#17702} [modify] https://crrev.com/a8d4b04380c1ff6d3f8672bf04ab6632392b7611/go/src/infra/appengine/chromium_build_stats/app/pubsub.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/731d1ac4d7a62328d5e73bfb9adbc7de6b32f235 commit 731d1ac4d7a62328d5e73bfb9adbc7de6b32f235 Author: Namiko Toriyama <namiko@google.com> Date: Wed Sep 19 09:14:54 2018 add timestamp to proto file Currently, it is difficult to search by time in BigQuery . By adding timestamp type to proto type, it makes searching by time easier, Because ptype.Timestamp is supported in BigQuery timestamp doc is below https://github.com/golang/protobuf/blob/master/ptypes/timestamp.go Bug: 873940 Change-Id: Ie3ca3e4e73d5785e4d4f83d19ca3f1e11eda3699 Reviewed-on: https://chromium-review.googlesource.com/1233134 Commit-Queue: Namiko Toriyama <namiko@google.com> Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Yoshisato Yanagisawa <yyanagisawa@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Cr-Commit-Position: refs/heads/master@{#17703} [modify] https://crrev.com/731d1ac4d7a62328d5e73bfb9adbc7de6b32f235/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog.go [modify] https://crrev.com/731d1ac4d7a62328d5e73bfb9adbc7de6b32f235/go/src/infra/appengine/chromium_build_stats/ninjalog/ninjalog_test.go [modify] https://crrev.com/731d1ac4d7a62328d5e73bfb9adbc7de6b32f235/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjalog.proto [modify] https://crrev.com/731d1ac4d7a62328d5e73bfb9adbc7de6b32f235/go/src/infra/appengine/chromium_build_stats/ninjaproto/ninjalog.pb.go
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/6ac025e0e45d70044ee8e8b1fc5dc8d80363a8cb commit 6ac025e0e45d70044ee8e8b1fc5dc8d80363a8cb Author: Namiko Toriyama <namiko@google.com> Date: Fri Sep 28 06:42:57 2018 add test for getFile Add test of the getfile function using google-cloud-go-testing. google-cloud-go-testing doc is below https://godoc.org/github.com/GoogleCloudPlatform/google-cloud-go-testing/storage/stiface google-cloud-go doc is below https://godoc.org/cloud.google.com/go/storage Bug: 873940 Change-Id: I4b509844df21b05e86767eaeacf90b59dd3bd6bc Reviewed-on: https://chromium-review.googlesource.com/1242503 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Commit-Queue: Namiko Toriyama <namiko@google.com> Cr-Commit-Position: refs/heads/master@{#17936} [modify] https://crrev.com/6ac025e0e45d70044ee8e8b1fc5dc8d80363a8cb/go/src/infra/appengine/chromium_build_stats/app/pubsub.go [modify] https://crrev.com/6ac025e0e45d70044ee8e8b1fc5dc8d80363a8cb/go/deps.lock [add] https://crrev.com/6ac025e0e45d70044ee8e8b1fc5dc8d80363a8cb/go/src/infra/appengine/chromium_build_stats/app/pubsub_test.go [modify] https://crrev.com/6ac025e0e45d70044ee8e8b1fc5dc8d80363a8cb/go/deps.yaml
I take rest of works. I will make table in chromium-build-stats, setup notification and deploy current app engine.
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/24d109f95fab745e453860475d62a0f29715a51f commit 24d109f95fab745e453860475d62a0f29715a51f Author: Takuto Ikuta <tikuta@chromium.org> Date: Wed Oct 03 05:00:50 2018 Add how to make BigQuery table in README I changed default tablename to ninjalog. Let me make staging and prod project will use new tablename after submitting this. Bug: 873940 Change-Id: Iddc51341a255e687ec8d9eb7c8950514f1572034 Reviewed-on: https://chromium-review.googlesource.com/c/1258723 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Auto-Submit: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Fumitoshi Ukai <ukai@chromium.org> Reviewed-by: Shinya Kawanaka <shinyak@chromium.org> Cr-Commit-Position: refs/heads/master@{#18019} [modify] https://crrev.com/24d109f95fab745e453860475d62a0f29715a51f/go/src/infra/appengine/chromium_build_stats/Makefile [modify] https://crrev.com/24d109f95fab745e453860475d62a0f29715a51f/go/src/infra/appengine/chromium_build_stats/app/bigquery.go [modify] https://crrev.com/24d109f95fab745e453860475d62a0f29715a51f/go/src/infra/appengine/chromium_build_stats/README.md
I setup pubsub subscription for prod project. $ gcloud --project=chromium-build-stats pubsub subscriptions create --topic=ninjalog --topic-project=chromium-archive --push-endpoint=https://chromium-build-stats.appspot.com/_ah/push-handlers/pubsub --ack-deadline=100 ninjalogSubscription Created subscription [projects/chromium-build-stats/subscriptions/ninjalogSubscription].
Prod project received ninjalog. https://pantheon.corp.google.com/bigquery?organizationId=433637338589&project=chromium-build-stats&j=bquxjob_73630fd8_1663859a98c&page=queryresults
Comment 1 by tikuta@chromium.org
, Aug 14Labels: -Pri-3 Pri-2