[schema] There are discussions for which schema library to use.

We've seen py-schema, Cerebrus, voluptuous, colander, also considered extending our existing "Arg" framework.

The plan is to have
 - widely used, actively developed library
 - able to add description to make better docs
 - easy to write and maintain
 - easy for non-factory-code, even non-python, non-linux to work on.

Current choice is using JSON schema ( http://json-schema.org/ ) because it:
 - well defined with proper spec, and many libraries
 - having simple tools to draft schema based on input data ( http://jsonschema.net/ )
 - No dependency on factory/python code. Any programs reading JSON can use it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/ce704bc59aeb7d4a9dc0ab9253686a112de3a954

commit ce704bc59aeb7d4a9dc0ab9253686a112de3a954
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jun 03 02:31:38 2016

Add dev-python/jsonschema to chromeos-factory-deps and target-chromium-os-sdk.

JSON Schema is going to be used in factory software in both build stage
and execution stage so we need it merged in both factory dependency and
chroot SDK.

BUG= chromium:616986 
TEST=build_packages --board link; build_image --board link
CQ-DEPEND=CL:349400

Change-Id: Ic8fc090417f53f6652c7eee367419d05c7f61695
Reviewed-on: https://chromium-review.googlesource.com/349410
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[rename] https://crrev.com/ce704bc59aeb7d4a9dc0ab9253686a112de3a954/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1-r63.ebuild
[rename] https://crrev.com/ce704bc59aeb7d4a9dc0ab9253686a112de3a954/chromeos-base/chromeos-factory-deps/chromeos-factory-deps-1-r19.ebuild
[modify] https://crrev.com/ce704bc59aeb7d4a9dc0ab9253686a112de3a954/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1.ebuild
[modify] https://crrev.com/ce704bc59aeb7d4a9dc0ab9253686a112de3a954/chromeos-base/chromeos-factory-deps/chromeos-factory-deps-1.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/71022aa14566cc2e273dd992da2b4fe3b0bc75c7

commit 71022aa14566cc2e273dd992da2b4fe3b0bc75c7
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jun 03 01:55:04 2016

jsonschema: upgraded package to upstream

Upgraded dev-python/jsonschema to version 2.3.0 on amd64

BUG= chromium:616986 
TEST=sudo emerge jsonschema

Change-Id: I3fd0c8c29c47dc641a61587ca8efc7f55439f3e1
Reviewed-on: https://chromium-review.googlesource.com/349400
Commit-Ready: Ilja H. Friedel <ihf@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[add] https://crrev.com/71022aa14566cc2e273dd992da2b4fe3b0bc75c7/dev-python/jsonschema/jsonschema-2.3.0.ebuild
[add] https://crrev.com/71022aa14566cc2e273dd992da2b4fe3b0bc75c7/dev-python/jsonschema/metadata.xml
[add] https://crrev.com/71022aa14566cc2e273dd992da2b4fe3b0bc75c7/metadata/md5-cache/dev-python/jsonschema-2.3.0
[add] https://crrev.com/71022aa14566cc2e273dd992da2b4fe3b0bc75c7/dev-python/jsonschema/Manifest

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/5db78ce371ae1c09a76fdbd16fd7149051f18046

commit 5db78ce371ae1c09a76fdbd16fd7149051f18046
Author: Hung-Te Lin <hungte@chromium.org>
Date: Tue Jun 07 04:23:41 2016

utils: Add config_utils module to allow an unified way for accessing configs.

The new config_utils.py module provides some APIs to simplify (with a
robust way) reading configuration data.

 LoadConfig(): Loads config with specified name (or caller module's name).
 OverrideConfig(): Overrides a partial config on top of existing config.
 GetNamedTuple(): Returns a dict-type config in namedtuple form.

BUG= chromium:616986 
TEST=make test

Change-Id: I5424057c5c14ddb6fdf36d556dcdccaf24505be6
Reviewed-on: https://chromium-review.googlesource.com/350220
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Hung-Te Lin <hungte@chromium.org>

[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/utils/config_utils.json
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/utils/config_utils.py
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/utils/testdata/config_utils_unittest.json
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/utils/testdata/config_utils_unittest.schema.json
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/config/testdata/config_utils_unittest.json
[modify] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/tools/deps.conf
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/utils/config_utils.schema.json
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/utils/config_utils_unittest.py
[add] https://crrev.com/5db78ce371ae1c09a76fdbd16fd7149051f18046/py/config/README

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/f5b92d295d9850d6473ce6c5096ba786d45e21b4

commit f5b92d295d9850d6473ce6c5096ba786d45e21b4
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 15 06:55:22 2016

utils: Add 'default_config_dir' to LoadConfig.

To help sharing config files between different modules, we want to have
a way to override the "default config directory", for example if we know
there will be two packages always deployed together:

 |
 +- umpire/
     \- umpire.json
 +- dome/
     \- dome_config.py

Then project 'dome' can load Umpire's config easily:

 import os
 from cros.factory.utils import config_utils

 ...
 umpire_config = config_utils.LoadConfig(
   'umpire',
   default_config_dir=os.path.join(os.path.dirname(__file__), '../umpire'))

BUG= chromium:616986 
TEST=make test

Change-Id: Idae2eb0546c60b4e63e011ffe2c4001d273087d3
Reviewed-on: https://chromium-review.googlesource.com/360733
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mao Huang <littlecvr@chromium.org>

[modify] https://crrev.com/f5b92d295d9850d6473ce6c5096ba786d45e21b4/py/utils/config_utils.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/5b9f1d4a21e290072db2df355280408783c2f402

commit 5b9f1d4a21e290072db2df355280408783c2f402
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 15 06:46:31 2016

utils: Improve log processing in config_utils.

When we want to migrate more configuration data into config_utils, it
becomes a problem that config_utils calls logging functions but logging
can be only configured once. In other words, programs loading
config_utils in its initialization stage (before functions like
logging.basicConfig are called - since that may also rely on config)
will have problem setting the logging module.

In additional to revise the logging initialization of all other
programs, we also want to make config_utils more friendly by disabling
its logs by default.

People who wants to debug the real config files loaded can enable
logging by writing such file in config/config_utils.json:

{ "Logging": true }

BUG= chromium:616986 
TEST=make test

Change-Id: Ibe0f393984abc7984dbf99895ba9816d4ca9e4f2
Reviewed-on: https://chromium-review.googlesource.com/360732
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mao Huang <littlecvr@chromium.org>

[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils.json
[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils.schema.json
[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils_unittest.py
[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/5b9f1d4a21e290072db2df355280408783c2f402

commit 5b9f1d4a21e290072db2df355280408783c2f402
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 15 06:46:31 2016

utils: Improve log processing in config_utils.

When we want to migrate more configuration data into config_utils, it
becomes a problem that config_utils calls logging functions but logging
can be only configured once. In other words, programs loading
config_utils in its initialization stage (before functions like
logging.basicConfig are called - since that may also rely on config)
will have problem setting the logging module.

In additional to revise the logging initialization of all other
programs, we also want to make config_utils more friendly by disabling
its logs by default.

People who wants to debug the real config files loaded can enable
logging by writing such file in config/config_utils.json:

{ "Logging": true }

BUG= chromium:616986 
TEST=make test

Change-Id: Ibe0f393984abc7984dbf99895ba9816d4ca9e4f2
Reviewed-on: https://chromium-review.googlesource.com/360732
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Mao Huang <littlecvr@chromium.org>

[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils.json
[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils.schema.json
[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils_unittest.py
[modify] https://crrev.com/5b9f1d4a21e290072db2df355280408783c2f402/py/utils/config_utils.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/0bd55eed5ae24aacc04a6db665da8b7c6d242f08

commit 0bd55eed5ae24aacc04a6db665da8b7c6d242f08
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 15 07:51:43 2016

utils: Fix default runtime config directory for config_utils.

The default runtime config directory should be /var/factory/config, not
/var/factory/log.

BUG= chromium:616986 
TEST=make test

Change-Id: I03b2afabfa2e68729ec6362d5af1e7c2d1e76bfe
Reviewed-on: https://chromium-review.googlesource.com/360782
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Shun-Hsing Ou <shunhsingou@chromium.org>

[modify] https://crrev.com/0bd55eed5ae24aacc04a6db665da8b7c6d242f08/py/utils/config_utils.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/0bd55eed5ae24aacc04a6db665da8b7c6d242f08

commit 0bd55eed5ae24aacc04a6db665da8b7c6d242f08
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 15 07:51:43 2016

utils: Fix default runtime config directory for config_utils.

The default runtime config directory should be /var/factory/config, not
/var/factory/log.

BUG= chromium:616986 
TEST=make test

Change-Id: I03b2afabfa2e68729ec6362d5af1e7c2d1e76bfe
Reviewed-on: https://chromium-review.googlesource.com/360782
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Shun-Hsing Ou <shunhsingou@chromium.org>

[modify] https://crrev.com/0bd55eed5ae24aacc04a6db665da8b7c6d242f08/py/utils/config_utils.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/a51aa8da9e5f28e2daa0b5191ac646f68591c558

commit a51aa8da9e5f28e2daa0b5191ac646f68591c558
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 29 02:18:09 2016

tools: Add *.json config files into factory.par

When running factory software remotely (inside par), the config must be
available in the archive (factory.par).

BUG= chromium:616986 , chromium:557573 
TEST=unzip -l factory.par | grep '\.json$'

Change-Id: I16bc58024be737cdb04cdb672df1e596fcd874fe
Reviewed-on: https://chromium-review.googlesource.com/364350
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Wei-Han Chen <stimim@chromium.org>

[modify] https://crrev.com/a51aa8da9e5f28e2daa0b5191ac646f68591c558/py/tools/make_par.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/1c4e12f5bea5be1167319d7fe839e8bb6f62dd0b

commit 1c4e12f5bea5be1167319d7fe839e8bb6f62dd0b
Author: Hung-Te Lin <hungte@chromium.org>
Date: Fri Jul 29 04:27:46 2016

utils: Support loading configuration data inside PAR (python archive).

When running inside factory.par, we can't access file by standard
functions (os.path or open). Instead we have to load objects directly
from ZIP archive.

This change allows config_utils to load JSON configs using zipimport so
we can find right data even side PAR.

BUG= chromium:616986 , chromium:557573 
TEST=make test

Change-Id: I28a0c40e69c2a2b6afece38ec3a62079012dbd4e
Reviewed-on: https://chromium-review.googlesource.com/364371
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Wei-Han Chen <stimim@chromium.org>

[modify] https://crrev.com/1c4e12f5bea5be1167319d7fe839e8bb6f62dd0b/py/utils/config_utils.py

New log is working fine.