puppet run failing on chromeos-server156.cbf (sentinel) due to skylab_inventory schema update |
|||
Issue description
Debug: Executing '/bin/bash -c '/root/chromeos-admin/bin/ensure_devserver_config --environment prod /usr/local/autotest/shadow_config.ini''
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/cli/atest.py", line 90, in main
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: results = action_obj.execute()
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/cli/server.py", line 209, in execute
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: return self.execute_skylab()
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/cli/server.py", line 192, in execute_skylab
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: inventory_repo.get_data_dir())
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/skylab_inventory/text_manager.py", line 82, in load_infrastructure
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: load_text_to_proto_message(data_file_path, infrastructure)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/skylab_inventory/text_manager.py", line 53, in load_text_to_proto_message
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: text_format.Parse(f.read(), proto_message)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 445, in Parse
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: text.split('\n'), message, allow_unknown_extension, allow_field_number)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 501, in ParseLines
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: return parser.ParseLines(lines, message)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 550, in ParseLines
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: self._ParseOrMerge(lines, message)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 575, in _ParseOrMerge
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: self._MergeField(tokenizer, message)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 676, in _MergeField
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: merger(tokenizer, message, field)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 766, in _MergeMessageField
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: self._MergeField(tokenizer, sub_message)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 676, in _MergeField
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: merger(tokenizer, message, field)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 815, in _MergeScalarField
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: value = tokenizer.ConsumeEnum(field)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/local/autotest/site-packages/chromite/third_party/google/protobuf/text_format.py", line 1203, in ConsumeEnum
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: raise self.ParseError(str(e))
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: ParseError: 1155:10 : Enum type "chrome.chromeos_infra.skylab.proto.inventory.Server.Role" has no value named ROLE_SKYLAB_SUITE_PROXY.
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: Unexpected exception: 1155:10 : Enum type "chrome.chromeos_infra.skylab.proto.inventory.Server.Role" has no value named ROLE_SKYLAB_SUITE_PROXY.
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: "__main__", fname, loader, pkg_name)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: exec code in run_globals
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/root/chromeos-admin/venv/admin/scripts/ensure_devserver_config.py", line 137, in <module>
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: sys.exit(main(sys.argv))
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/root/chromeos-admin/venv/admin/scripts/ensure_devserver_config.py", line 33, in main
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: servers = serverdb.get_servers(environ=args.environment)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/root/chromeos-admin/venv/admin/serverdb.py", line 27, in get_servers
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: obj = json.loads(output)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: return _default_decoder.decode(s)
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: raise ValueError("No JSON object could be decoded")
Notice: /Stage[main]/Autotest::Shadow_config/Exec[append devserver config]/returns: ValueError: No JSON object could be decoded
,
Nov 7
I don't know how widespread this problem is. The problem is that atest (from /usr/local/autotest/cli/atest) uses a copy of infra/skylab_inventory in /usr/local/autotest/site-packages/... for the skylab inventory schema. This would be updated on build_externals. I landed a CL to fix the role used by skylab-suite drones: https://chrome-internal-review.googlesource.com/c/chromeos/infra_internal/skylab_inventory/+/709135 The role had been added to schema long ago: https://chromium-review.googlesource.com/c/chromiumos/infra/skylab_inventory/+/1171943/ But build_externals was never run on sentinel, so `atest --skylab` choked on the new enum value. There are two problems here. [1] race: One must wait for prod push after a skylab inventory schema update before updating data to use the new schema. This seems wrong. proto definitions are supposed to be forward and backward compatible. The old proto schema should be able to parse newer data (perhaps setting any unknown enum values to _INVALID value) [2] Not all servers run build_externals, so the schema was ancient. Let's fix [2] first.
,
Nov 7
Re c#2 [1] may happen because fields using that enum in the proto definition are "required". The default is only used with optional fields.
,
Nov 13
|
|||
►
Sign in to add a comment |
|||
Comment 1 by pprabhu@chromium.org
, Nov 7Labels: -Pri-2 Pri-1