This bug is for tracking changes for porting HWID Service from google3.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/5e22dc583d4d4b34f3393334a8c3893b5adcdc95 commit 5e22dc583d4d4b34f3393334a8c3893b5adcdc95 Author: Yilun Lin <yllin@google.com> Date: Thu Mar 15 13:57:49 2018 appengine: Reanme direcotry appengine to appengine2 This CL is preparing for the incoming appengine porting change. The appengine2 will be deprecated soon. BUG= chromium:818125 TEST=make test Change-Id: Idf1a6834948555c6b521927d32ad7199e1c5b4c2 Reviewed-on: https://chromium-review.googlesource.com/963878 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [rename] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/hwid/service/appengine2/proxy.py [modify] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/setup/cros_hwidservice.sh [rename] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/hwid/service/appengine2/definition.py [rename] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/hwid/service/appengine2/app.yaml [modify] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/tools/deps.conf [rename] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/hwid/service/appengine2/README.md [rename] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/hwid/service/appengine2/factory_common.py [rename] https://crrev.com/5e22dc583d4d4b34f3393334a8c3893b5adcdc95/py/hwid/service/appengine2/__init__.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/91256b85bb0372d9e6b0713f9fe07c6b26cb1435 commit 91256b85bb0372d9e6b0713f9fe07c6b26cb1435 Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 16:20:23 2018 appengine: Porting HWID API from google3. To reuse the APIary config, which exposes AppEngine API to googleapis.com, such that both IP from interior or exterior of google domain can access the API. We decide to re-use the APIary config by porting the HWID API from google3. The source can be found in go/g3hwidapi. And also, the coding style is very different between two places, we have to modify a bit. In addition, the available python packages are different, too, so we have to replace them with the self-implemented one. - Follows factory coding style. - Replaces google3 package pyglib to class _RamSize - Adapts hwid_manager to new HWID v3 API, including - Remove unnecessary file hwid_yaml_loader.py, security.py. - Renames board names in testdata. - Adds AppEngine integration test. - Adds deploy script. - Sets app.yaml instance_class to F4_1G, which is the fastest and the largest memory machine. BUG= chromium:818125 TEST=manual test; make test; appengine/test/integration_test.py Change-Id: I970eadaebe9c8b8e2c222b891c9966fe3b42f2ca Reviewed-on: https://chromium-review.googlesource.com/963879 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/requirements.txt [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/boards.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_util_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/test/integration_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_manager.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/config.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/cron.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/appengine_test_base.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/v3-golden.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_api.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/v3-golden-after-good.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_api_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/goldeneye_ingestion.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/ingestion.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_manager_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/config_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/bigstore/TEST_PROJECT_ONE [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_util_unittest.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/deploy.sh [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/v3-golden-after-bad.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/test/factory_common.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/factory_common.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/seed_bigstore.sh [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_validator.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/memcache_adaptor_test.py [modify] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/tools/deps.conf [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_validator_unittest.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_updater.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/memcache_adaptor.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/appengine_config.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/__init__.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/test/README.md [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_api_messages.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/bigstore/boards.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/app.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/filesystem_adapter_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/v2-golden.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/boards_server.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/ingestion_test.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/app.yaml [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/filesystem_adapter.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/static/favicon.ico [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/bigstore/TEST_PROJECT_TWO [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/test/Dockerfile [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/hwid_util.py [add] https://crrev.com/91256b85bb0372d9e6b0713f9fe07c6b26cb1435/py/hwid/service/appengine/testdata/v3-golden-before.yaml
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/1f60b7de99d8a6a099e1fb300b9a414973007205 commit 1f60b7de99d8a6a099e1fb300b9a414973007205 Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 19:21:20 2018 process_utils: Workaround for AppEngine environment. HWID module references many packages outside `cros.factory.hwid.*`. These packages are using process_utils either directly import process_utils or import it in the import chain. AppEngine doesn't support some attributes, for example, `subprocess.Popen` and `subprocess.PIPE`. Using these attributes in the code raises exceptions. TEST=manual on AppEngine BUG= chromium:797645 , chromium:818125 Change-Id: Ic769a7a65147d782dbd2fd04d96d94c2013af1f0 Reviewed-on: https://chromium-review.googlesource.com/963880 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [modify] https://crrev.com/1f60b7de99d8a6a099e1fb300b9a414973007205/py/utils/process_utils.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/30c3529c98d449da6f5bc242aceb35006d08b278 commit 30c3529c98d449da6f5bc242aceb35006d08b278 Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 19:21:21 2018 appengine: Refactors config file. Use class _Config to replace attributes in the config file. TEST=integration_test; make test; BUG= chromium:818125 Change-Id: I6f0293c4f2fd20a0e9266e4a6430b6553b9971fc Reviewed-on: https://chromium-review.googlesource.com/963941 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/config_test.py [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/config.py [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/hwid_api.py [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/hwid_api_test.py [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/goldeneye_ingestion.py [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/ingestion.py [modify] https://crrev.com/30c3529c98d449da6f5bc242aceb35006d08b278/py/hwid/service/appengine/ingestion_test.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b98de72b981eafbbbf6455f7037f90215f451d1b commit b98de72b981eafbbbf6455f7037f90215f451d1b Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 19:21:22 2018 appengine: Set log level to DEBUG. Filtering logs in the AppEngine's dashboard is very easy, so it is fine just logging everything here. Also, sets log level while the AppEngine starting up. TEST=manally run on AppEngine. BUG= chromium:818125 Change-Id: Ie8ee47e532af2ff955a1a958a4c6fd53cde194d6 Reviewed-on: https://chromium-review.googlesource.com/963942 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [modify] https://crrev.com/b98de72b981eafbbbf6455f7037f90215f451d1b/py/hwid/service/appengine/appengine_config.py [modify] https://crrev.com/b98de72b981eafbbbf6455f7037f90215f451d1b/py/hwid/service/appengine/hwid_api.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/e3c84e34d84f4a119d94cdde4fb9907248141a01 commit e3c84e34d84f4a119d94cdde4fb9907248141a01 Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 19:21:23 2018 hwid_api: Make AuthCheck and logging API requests a decorator. - Adds decorator HWIDAPI. - Logs all API requests. - Moves _AuthCheck to be a global function. - Moves APIARY_USER_LIST, KNOWN_BAD_HWIDS, KNOWN_BAD_SUBSTR out to be global variables. TEST=manual BUG= chromium:818125 Change-Id: I6e7dde6ce4997ef682598e304b6d0454c5c8fe91 Reviewed-on: https://chromium-review.googlesource.com/963943 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [modify] https://crrev.com/e3c84e34d84f4a119d94cdde4fb9907248141a01/py/hwid/service/appengine/hwid_api_test.py [modify] https://crrev.com/e3c84e34d84f4a119d94cdde4fb9907248141a01/py/hwid/service/appengine/hwid_api.py
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/6415e894f7f60a9a8791bf82c90de9105709c5c6 commit 6415e894f7f60a9a8791bf82c90de9105709c5c6 Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 19:21:23 2018 appengine: Adds e2e_test. Adds AppEngine e2e tests. e2e_test loads a test config from factory-private repository, and then runs tests described in the config file. Ideally, one should run e2e_test before deploying HWID Service to prod environment. TEST=run e2e_test.py BUG= chromium:818125 Change-Id: Icd0d667ef8378347a960f3bc7f74c1556669ea90 Reviewed-on: https://chromium-review.googlesource.com/963944 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [add] https://crrev.com/6415e894f7f60a9a8791bf82c90de9105709c5c6/py/hwid/service/appengine/test/e2e_test.py [add] https://crrev.com/6415e894f7f60a9a8791bf82c90de9105709c5c6/py/hwid/service/appengine/test/e2e_test.schema.json [modify] https://crrev.com/6415e894f7f60a9a8791bf82c90de9105709c5c6/py/hwid/service/appengine/test/README.md
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/f405d7b6c1dfa9c9b2bbfba1176ff68ffaa847e7 commit f405d7b6c1dfa9c9b2bbfba1176ff68ffaa847e7 Author: Yilun Lin <yllin@google.com> Date: Mon Apr 16 19:21:24 2018 appengine: Add README.md. TEST=none BUG= chromium:818125 Change-Id: I2f14307bc588a501d3c6b1336a31f334278401e8 Reviewed-on: https://chromium-review.googlesource.com/963945 Commit-Ready: Yilun Lin <yllin@chromium.org> Tested-by: Yilun Lin <yllin@chromium.org> Reviewed-by: Yong Hong <yhong@google.com> [add] https://crrev.com/f405d7b6c1dfa9c9b2bbfba1176ff68ffaa847e7/py/hwid/service/appengine/README.md
The following revision refers to this bug: https://chrome-internal.googlesource.com/chromeos/platform/factory-private/+/2d9b0116e83b01f3db896015c38828958f603c1b commit 2d9b0116e83b01f3db896015c38828958f603c1b Author: Yilun Lin <yllin@google.com> Date: Wed Apr 18 06:34:39 2018
Is this all done?
Comment 1 by bugdroid1@chromium.org
, Mar 15 2018