New issue
Advanced search Search tips

Issue 906761 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 20
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

protobuf_version:v3.6.1 causes problems for monorail

Project Member Reported by ehmaldonado@chromium.org, Nov 19

Issue description

Monorail tests are failing after https://chromium.googlesource.com/infra/infra/+/caa0bd3d76fcc9ccf0c8d999fb1fd5d77da2f4a7 
With:
"TypeError: __init__() got an unexpected keyword argument 'serialized_options'" on linux after running "make prpc_proto"

To reproduce, go to appengine/monorail and run 'maoe prpc_proto' and 'make test'.
 
Cc: mar...@chromium.org
Owner: vadimsh@chromium.org
Status: Assigned (was: Untriaged)
Looking. We need to update protobuf library in infra.git to 3.6.1 too, but I'm not sure who exactly to do this...
Yeah. I tried rolling this. It is not sufficient.
ok
Project Member

Comment 5 by bugdroid1@chromium.org, Nov 19

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0b198138877d2e7670dcc56baee83034f0d1b472

commit 0b198138877d2e7670dcc56baee83034f0d1b472
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Mon Nov 19 21:00:56 2018

Roll infra/luci/ 473a850bc..af1fb93ec (83 commits)

https://chromium.googlesource.com/infra/luci/luci-py/+log/473a850bc451..af1fb93ece58

$ git log 473a850bc..af1fb93ec --date=short --no-merges --format='%ad %ae %s'
2018-11-19 maruel protobuf: upgrade to 3.6.1 from 3.5.1
2018-11-19 akeshet [swarming] rename _maybe_pubsub_notify_via_tq
2018-11-19 akeshet swarming: plugin api: add slice_number cancellation reasons
2018-11-19 akeshet [swarming] add a dev target to Makefile for local testing
2018-11-19 jchinlee [client] Print time with tasks update message. Also increase frequency.
2018-11-19 kjlubick [swarming-ui] Address some feedback with sorting/aliases
2018-11-16 vadimsh [auth] Put additional info extracted from credentials into the auth state.
2018-11-16 bradhall Only allow repeated dimensions for 'caches'
2018-11-16 maruel Remove last references to rietveld
2018-11-15 akeshet Revert "[swarming] rename _maybe_pubsub_notify_via_tq"
2018-11-15 akeshet [swarming] rename _maybe_pubsub_notify_via_tq
2018-11-14 tikuta [isolate] Increase max_concurrent_requests
2018-11-13 akeshet [swarming] allow swarming bot to run under Chrome Remote Desktop
2018-11-12 vadimsh [swarming] Add 'swarming/bot_auth/success' metric.
2018-11-12 jbudorick swarming: process tasks in LIFO order.
2018-11-10 akeshet [swarming] add an enabled field to ExternalSchedulerConfig
2018-11-09 akeshet [swarming] use argparse in start_bot.py
2018-11-09 vadimsh [swarming] Make 'auth' field in BotGroup repeated.
2018-11-09 nodir [prpc] Add pRPC client in Python
2018-11-09 vadimsh [swarming] Preparation for making 'auth' in BotGroup proto repeated.
2018-11-08 kjlubick Fix small wording things
2018-11-07 akeshet [swarming] regenerate protos to include prpc bindings
2018-11-07 kjlubick Fix sorting bots by last_seen etc
2018-11-07 akeshet [swarming] create a representation of external scheduler in PoolConfig
2018-11-06 nodir [compile_proto] Pass --prpc-python_out
2018-11-06 kjlubick Make ui2 the default for some pages
2018-11-06 tikuta [client] Add --filepath-filter option to choose download files from isolate
2018-11-06 vadimsh [swarming] Add gce.signed_metadata_token(...) call.
2018-11-06 kjlubick Add support to cancel running tasks from UI
2018-11-06 benjaminwagner Add reboot_required to bot state.
2018-11-05 vadimsh [auth] A function to verify JWTs produced by Google auth backends.
2018-11-02 tikuta Revert "[client] Increase worker to download isolate files faster"
2018-11-01 tikuta [client] Increase worker to download isolate files faster
2018-11-01 akeshet [swarming] proto definition of external scheduler API
2018-11-01 vadimsh [isolate] Don't unexpectedly raise KeyError if the cache state file was lost.
2018-10-31 maruel swarming: add support for sorting by COMPLETED_TS
2018-10-31 akeshet [swarming] add proto definition for swarming tasks
2018-10-31 akeshet [swarming] add external scheduler options to pools.cfg
2018-10-31 maruel Fix a small error in d63fdb9a27d0e89
2018-10-31 kjlubick Make secret bytes less opaque
2018-10-31 smut [Machine Provider] Fix snapshot label match detection
2018-10-30 bpastene swarming: Roll py-adb to 412d35c842fb0917233da6fd2c6c94dfa3450593.
2018-10-30 akeshet [swarming] add "package swarming" to swarming protos
2018-10-29 kjlubick Add aliasing to new botlist
2018-10-25 kjlubick Mostly feature complete Web Components botlist
2018-10-23 smut [Swarming] Support repeatable MP dimensions, reject invalid ones
2018-10-23 smut [GCE Backend] Remove limit on number of instance templates
2018-10-23 smut [GCE Backend] Increase allowable number of instance templates
2018-10-19 smut [Swarming] Require disk_type, num_cpus, project for MP bots
2018-10-19 fmatenaar [auth] Deflake jwt minting cache test
2018-10-19 fmatenaar [auth] Fix for inconsistent result types across token minting functions
2018-10-19 bradhall Add optional-dimension argument to swarming
2018-10-18 nodir [config_service] Add service_access_group
2018-10-18 maruel [swarming] Report memory usage after each handler
2018-10-18 fmatenaar [auth] Migrate chrome infra python code to use IAM:generateAccessToken
2018-10-18 smut [Machine Provider] Advertise snapshot-related labels
2018-10-17 maruel [components] reorder utils.py
2018-10-15 maruel [swarming] fix frontend / backend split
2018-10-09 maruel [swarming] Remove enforcement of unique 'id' in TaskSlice's
2018-10-08 robertocn [run_isolated] Replace parameters in environment variables.
2018-10-05 bpastene swarming: Roll py-adb to pick up log fix.
2018-10-05 hinoka luci-config: Default home page to /#/q/
2018-10-05 bpastene swarming: Roll py-adb.
2018-10-03 kjlubick Adds the first pieces of the swarming bot-list using WebComponents.
2018-10-03 kjlubick Add sort-toggle element
2018-10-02 kjlubick Update WebComponents deps
2018-09-28 vadimsh [swarming] Return more detailed errors on service account misconfiguration.
2018-09-28 hinoka Fix luci-config UI issues caused due to incompatable npm/bower versions
2018-09-28 kjlubick Add CIPD links to task-page UI
2018-09-28 borenet Fix symlink bug in file_path.set_read_only()
2018-09-28 vadimsh Workaround a bug in task queue's stub.
2018-09-27 hinoka Move opensearch link to index.html
2018-09-27 hinoka Link the opensearch document from the front page
2018-09-27 hinoka Add opensearch to luci-config UI
2018-09-25 kjlubick Fix bad div
2018-09-25 maruel [swarming] enable capacity to outlive the last bot
2018-09-25 kjlubick Fix expiration time for task slices
2018-09-24 maruel [client] improve NamedCache logging
2018-09-24 maruel [client] fix unit test on macOS
2018-09-24 maruel [swarming] enable indexing for BotEvent.dimensions_flat
2018-09-24 kjlubick Prevent soft refresh from changing the current slice
2018-09-24 kjlubick make it more clear when a deduped task was run
2018-09-24 kjlubick Add relative paths to repro commands

Created with:
  roll-dep infra/luci

R=vadimsh@chromium.org

Bug:  906761 
Change-Id: Ibdc25d849b969da65015800868bf7f03d6cbab0d
Reviewed-on: https://chromium-review.googlesource.com/c/1343159
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19075}
[modify] https://crrev.com/0b198138877d2e7670dcc56baee83034f0d1b472/DEPS

https://chromium-review.googlesource.com/c/infra/infra/+/1342708 fixes monorail tests, but breaks some other tests :) I'm working on fixing them too.
infra/libs/buildbucket/proto/* can't find google/rpc/status.proto:

Exception occurred when listing tests. Aborting.
Traceback (most recent call last):
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/expect_tests/pipeline.py", line 521, in result_loop_single_context
    gen_loop_process(*test_gen_args)
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/expect_tests/pipeline.py", line 335, in gen_loop_process
    result_queue.put_nowait)
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/expect_tests/type_definitions.py", line 465, in gen_stage_loop
    for test in tests:
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/expect_tests/pipeline.py", line 278, in generate_tests
    gens = get_test_gens_package(testing_context, opts, subpath=subpath)
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/expect_tests/pipeline.py", line 243, in get_test_gens_package
    mod = load_module(modname)
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/expect_tests/pipeline.py", line 212, in load_module
    mod = __import__(modname)
  File "/Users/vadimsh/infra-git/infra/appengine/findit/pipelines/compile_failure/test/revert_and_notify_compile_culprit_pipeline_test.py", line 12, in <module>
    from pipelines.compile_failure import (
  File "/Users/vadimsh/infra-git/infra/appengine/findit/pipelines/compile_failure/revert_and_notify_compile_culprit_pipeline.py", line 8, in <module>
    from pipelines.create_revert_cl_pipeline import CreateRevertCLPipeline
  File "/Users/vadimsh/infra-git/infra/appengine/findit/pipelines/create_revert_cl_pipeline.py", line 11, in <module>
    from services import culprit_action
  File "/Users/vadimsh/infra-git/infra/appengine/findit/services/culprit_action.py", line 20, in <module>
    from services import ci_failure
  File "/Users/vadimsh/infra-git/infra/appengine/findit/services/ci_failure.py", line 16, in <module>
    from services import step_util
  File "/Users/vadimsh/infra-git/infra/appengine/findit/services/step_util.py", line 15, in <module>
    from common.waterfall import buildbucket_client
  File "/Users/vadimsh/infra-git/infra/appengine/findit/common/waterfall/buildbucket_client.py", line 12, in <module>
    from buildbucket_proto.rpc_pb2 import GetBuildRequest
  File "/Users/vadimsh/infra-git/infra/appengine/findit/first_party/buildbucket_proto/rpc_pb2.py", line 37, in <module>
    notification__pb2.DESCRIPTOR,
  File "/Users/vadimsh/infra-git/infra/ENV/lib/python2.7/site-packages/google/protobuf/descriptor.py", line 878, in __new__
    return _message.default_pool.AddSerializedFile(serialized_pb)
TypeError: Couldn't build proto file into descriptor pool!
Invalid proto descriptor for file "rpc.proto":
  rpc.proto: Import "google/rpc/status.proto" has not been loaded.
  buildbucket.v2.BatchResponse.Response.error: "google.rpc.Status" seems to be defined in "status.proto", which is not imported by "rpc.proto".  To use it here, please add the necessary import.

Which is very odd, since generated code has
  
from google.rpc import status_pb2 as google_dot_rpc_dot_status__pb2

which in theory should load rpc.proto's descriptor.

Also 'which is not imported by "rpc.proto"' part is total lie: https://chromium.googlesource.com/infra/luci/luci-go/+/master/buildbucket/proto/rpc.proto#13
Project Member

Comment 8 by bugdroid1@chromium.org, Nov 19

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/430746cae9ef8c19c80fbe684be4d3b0ded41a3f

commit 430746cae9ef8c19c80fbe684be4d3b0ded41a3f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Nov 19 23:14:38 2018

[proto] Fix google/rpc/*_pb2.py, it has wrong proto paths in it.

status.proto should be added to the registry as google/rpc/status.proto,
otherwise importing modules (which import it as 'google/rpc/status.proto') can't
find it.

R=nodir@chromium.org, iannucci@chromium.org, maruel@chromium.org
BUG= 906761 

Change-Id: I22781b3745937ddc451d02341cb03cba64493a16
Reviewed-on: https://chromium-review.googlesource.com/c/1342945
Auto-Submit: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/430746cae9ef8c19c80fbe684be4d3b0ded41a3f/client/third_party/google/README.swarming
[modify] https://crrev.com/430746cae9ef8c19c80fbe684be4d3b0ded41a3f/client/third_party/google/rpc/code_pb2.py
[modify] https://crrev.com/430746cae9ef8c19c80fbe684be4d3b0ded41a3f/client/third_party/google/rpc/error_details_pb2.py
[modify] https://crrev.com/430746cae9ef8c19c80fbe684be4d3b0ded41a3f/client/third_party/google/rpc/status_pb2.py

Project Member

Comment 9 by bugdroid1@chromium.org, Nov 19

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/12cd5f8292dee67610b692fd141431f5a65c544f

commit 12cd5f8292dee67610b692fd141431f5a65c544f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Nov 19 23:58:42 2018

Update protobuf lib to 3.6.1.

Existing 3.5.1 is no longer compatible with code generated by protoc 3.6.1.

Fix all the broken stuff. Also includes infra/luci DEPS roll to pick up _pb2.py
change that mutually depends on protobuf lib change:

https://chromium.googlesource.com/infra/luci/luci-py/+log/af1fb93ece58..430746cae9ef

$ git log af1fb93ec..430746cae --date=short --no-merges --format='%ad %ae %s'
2018-11-19 vadimsh [proto] Fix google/rpc/*_pb2.py, it has wrong proto paths in it.
2018-11-19 iannucci [swarming] Move isolate and cipd server settings to pools.cfg.
2018-11-19 maruel [swarming] expand BotEvent.ALLOWED_EVENTS to be more readable

R=maruel@chromium.org
BUG= 906761 

Change-Id: I41b96fc246f707647951e899348c4187b4b49e6c
Reviewed-on: https://chromium-review.googlesource.com/c/1342708
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#19077}
[modify] https://crrev.com/12cd5f8292dee67610b692fd141431f5a65c544f/infra/libs/buildbucket/swarming/test/flatten_swarmingcfg_test.py
[modify] https://crrev.com/12cd5f8292dee67610b692fd141431f5a65c544f/bootstrap/deps.pyl
[modify] https://crrev.com/12cd5f8292dee67610b692fd141431f5a65c544f/appengine/cr_rev/appengine_module/cr_rev/test/cr_rev_api_test.py
[modify] https://crrev.com/12cd5f8292dee67610b692fd141431f5a65c544f/DEPS
[modify] https://crrev.com/12cd5f8292dee67610b692fd141431f5a65c544f/infra/services/bugdroid/config_service.py

Status: Fixed (was: Assigned)
It should be working now. Requires gclient sync (to pick up 'infra/luci' roll) and gclient runhooks (to rebuild infra/ENV).
Thanks!
Would it be possible to add a presubmit step to generate monorail proto files before testing?

Sign in to add a comment