factory: pre-submit checks usually fail due to unit tests |
|||||
Issue descriptionCurrently when doing "repo upload", the submit check is really hard to pass so people'll usually --no-verify or manually touch the test-passed flag. Ideally we'd like to add more presubmit checks in future, so it's important to make sure the unit tests pass. Let's set a goal to make sure - unit tests won't take too long (less than 40 or 20s) - unit tests should be stable - fix tests that can't run in parallel - change unit tests so they don't need to bind on fixed ports
,
Nov 15 2016
> - goofy unittest binds on fixed ports (pihsun has fixed this?) No it's still unfixed. The test py/hwid/v3/valid_hwid_db_unittest.py is also failing recently.
,
Nov 15 2016
I'm thinking to modify goofy_unittest to a real unittest, but keep the current one as an e2e tests, so that we can run it when needed. Just thinking if it's possible to have: 1. pre-submit runs unit tests. 2. CQ runs e2e tests for us.
,
Nov 15 2016
Re#3: that is possible. Portage supports a test stage that it can do make test. We can revise Makefile to support both 'unittest' and 'test'.
,
Nov 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/6884860f2ff9dc3be03dca9835f12103969ee2c9 commit 6884860f2ff9dc3be03dca9835f12103969ee2c9 Author: Peter Shih <pihsun@chromium.org> Date: Tue Nov 15 09:18:36 2016 fix run_tests can't kill old goofy test. Goofy imported setproctitle in py/goofy/invocation.py. By just importing that module, the file /proc/PID/environ would be messed up and filled with all zero, which cause the old test killer in run_tests.py to fail. From https://pypi.python.org/pypi/setproctitle, we can prevent the problem by setting environment variable SPT_NOENV. BUG= chromium:665306 TEST=make test TEST=make test, ctrl-c when goofy_unittest.py is running, and run make test again. Verify that previous goofy_unittest is killed. Change-Id: I819dce658c70b20b72eefe9d6ca68eb4c6fd2ead Reviewed-on: https://chromium-review.googlesource.com/411547 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/6884860f2ff9dc3be03dca9835f12103969ee2c9/py/tools/run_tests.py
,
Dec 23 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/c56c5b69b794d86a7c4b69456eb7ad992e9e220f commit c56c5b69b794d86a7c4b69456eb7ad992e9e220f Author: Peter Shih <pihsun@chromium.org> Date: Thu Dec 22 04:30:57 2016 overlord: add environment variables to change port to bind/connect Add ability to control which port overlordd/ghost to bind/connect to using environment variables OVERLORD_PORT, OVERLORD_HTTP_PORT, OVERLORD_LD_PORT, GHOST_RPC_PORT. Also fix two tests that are calling overlord to use random port, so they can be passed even if another overlord instance is running. We expect that the default port is always used, but we need this ability to make unit test not failing because of port is in used. BUG= chromium:665306 TEST=make test, and make sure all tests can be passed in the first try. TEST=manually start an overlordd instance, then run make test. check all tests still pass. Change-Id: Icf3335bc0a586cc68a07a812f998444cfdc1ebb6 Reviewed-on: https://chromium-review.googlesource.com/422791 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Ning Huang <wnhuang@chromium.org> [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/py/utils/net_utils.py [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/go/src/overlord/ghost.go [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/go/src/overlord/constants.go [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/go/src/overlord/test/overlord_e2e_unittest.py [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/py/tools/ghost.py [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/go/src/overlord/utils.go [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/py/umpire/service/umpire_service.py [modify] https://crrev.com/c56c5b69b794d86a7c4b69456eb7ad992e9e220f/py/umpire/service/overlord_unittest.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/2053b544b4374dfe90d9ded5afbea64b3323e5b6 commit 2053b544b4374dfe90d9ded5afbea64b3323e5b6 Author: Peter Shih <pihsun@chromium.org> Date: Thu Dec 22 10:12:21 2016 stop shopfloor_standalone_unittest.py being flaky The shopfloor_standalone_unittest.py invokes `make par`, which does some tar-ing on the py folder. But since we cleaned all pyc before running tests, the generated pyc by other parallel tests would cause tar to fail with "file changed as we read it", even if the pyc is in exclude lists. We can stop pyc being generated by setting environ variable PYTHONDONTWRITEBYTECODE. BUG= chromium:665306 TEST=make test TEST=make test many times, and check that shopfloor_standalone_unittest doesn't have error on first run. Change-Id: I53f0c93644a5eb85e08679790523099fd6647d12 Reviewed-on: https://chromium-review.googlesource.com/422753 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/2053b544b4374dfe90d9ded5afbea64b3323e5b6/py/test/utils/pytest_utils_unittest.py [modify] https://crrev.com/2053b544b4374dfe90d9ded5afbea64b3323e5b6/py/tools/run_tests.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b241ad7bbf1d1fe37ae3190d58b89218b903c13f commit b241ad7bbf1d1fe37ae3190d58b89218b903c13f Author: Peter Shih <pihsun@chromium.org> Date: Fri Dec 23 03:11:07 2016 fix flaky goofy_base_unittest When multiple threads are using the same generator at the same time, we may get error "ValueError: generator already executing". Add a lock to ensure that there's no race condition between threads. BUG= chromium:665306 TEST=make test TEST=run py/goofy/goofy_base_unittest.py many times, and observe no failure. Change-Id: I312df57364a89f348e3c35b72dd648b32e4d0019 Reviewed-on: https://chromium-review.googlesource.com/422754 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/b241ad7bbf1d1fe37ae3190d58b89218b903c13f/py/goofy/goofy_base_unittest.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/2a957ea60f1c69ef4352091ae978fa4fdb2458aa commit 2a957ea60f1c69ef4352091ae978fa4fdb2458aa Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 02:37:30 2016 fix umpire_service_unittest fail when there's too many files in /tmp When there's too many files in /tmp (for example, stress testing `make test` and there would be lots of test logs left), running `ls` on it is no longer a fast process... Fix this by calling `true` instead. BUG= chromium:665306 TEST=make test TEST=run py/umpire/service/umpire_service_unittest.py when /tmp has lots of file (about 200000), and make sure that the test passes. Change-Id: Ifb37d7b081683d2cf3b4c9bde57541724eb10635 Reviewed-on: https://chromium-review.googlesource.com/422800 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/2a957ea60f1c69ef4352091ae978fa4fdb2458aa/py/umpire/service/umpire_service_unittest.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/aee37bc44e4126f16e63e40438b7f3c4c974eab8 commit aee37bc44e4126f16e63e40438b7f3c4c974eab8 Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 04:54:35 2016 fix temporary file not deleted in buffer_simple_file_unittest There's some call to _CreateBuffer directly, causing the function to be called twice (once in setUp, once in subtest), and the temporary folder created in the first call would not be deleted. Fix this by delete the old temporary folder before generating a new one. Also fix lint errors in the file. BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/simple_file.*, and run py/instalog/plugins/buffer_simple_file/buffer_simple_file_unittest.py, make sure that no temp file /tmp/simple_file.* are left after the test complete. Change-Id: I35665d8d63afb70b13df37992aa0642f14400b24 Reviewed-on: https://chromium-review.googlesource.com/422801 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Joel Kitching <kitching@chromium.org> [modify] https://crrev.com/aee37bc44e4126f16e63e40438b7f3c4c974eab8/py/instalog/plugins/buffer_simple_file/buffer_simple_file_unittest.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/e14fe2b5dd5efc4c83daf04a7edf699038746655 commit e14fe2b5dd5efc4c83daf04a7edf699038746655 Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 03:32:22 2016 fix temporary file not deleted in test/event.py The temporary file generated by EventServer is not deleted by anyone when server close, fix it. BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/cros_factory_event.*, and run py/goofy/goofy_unittest.py, make sure that no temp file /tmp/cros_factory_event.* are left after the test complete. Change-Id: I91c4ea4f147444011c22bb700b35b8920e148a31 Reviewed-on: https://chromium-review.googlesource.com/423021 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/e14fe2b5dd5efc4c83daf04a7edf699038746655/py/test/event.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/db8e70d9832cf8a703caa9267660b9d54931c8eb commit db8e70d9832cf8a703caa9267660b9d54931c8eb Author: Peter Shih <pihsun@chromium.org> Date: Mon Dec 26 09:50:12 2016 remove flaky test in socket_e2e_unittest There's some race condition between this Ping in unittest and Ping in OutputSocket's Main, causing the unittest to have about 10% of fail rate in `make test`. Since the OutputSocket class Ping's function isn't designed to be called directly, and doesn't handle multithread well, and Ping is used inside Main (so also tested), we don't need testPing. BUG= chromium:665306 TEST=make test TEST=run py/instalog/plugins/socket_e2e_unittest.py many times, and observe no failure. Change-Id: Ie226cff919a7bebbb2ae92919e2e662df7127523 Reviewed-on: https://chromium-review.googlesource.com/423274 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Joel Kitching <kitching@chromium.org> [modify] https://crrev.com/db8e70d9832cf8a703caa9267660b9d54931c8eb/py/instalog/plugins/socket_e2e_unittest.py
,
Dec 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/db1566e58c0235e98654cbbb66efc468900ad20e commit db1566e58c0235e98654cbbb66efc468900ad20e Author: Peter Shih <pihsun@chromium.org> Date: Tue Dec 27 07:30:41 2016 refactor system_log_manager_unittest to not depend on time Write a fake time class and a fake queue class, and use them in testing, so the test doesn't depends on time.sleep and system load. (The test currently has about 15% fail rate in `make test`) This makes the test super fast (from 13.5 secs to 0.15 secs), and always succeed. BUG= chromium:665306 TEST=make test TEST=run py/goofy/plugins/system_log_manager_unittest.py many times, and observe no failure. Change-Id: I2def1224c390387fc42efbe32275634c8e930c53 Reviewed-on: https://chromium-review.googlesource.com/424236 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/db1566e58c0235e98654cbbb66efc468900ad20e/py/goofy/plugins/system_log_manager.py [modify] https://crrev.com/db1566e58c0235e98654cbbb66efc468900ad20e/py/goofy/plugins/system_log_manager_unittest.py
,
Dec 29 2016
Some statistics of running `make test` after CL:424235, commit 140b51952d2af4b4f5b172b5797cb567a35cae21 (not yet merged): Total count = 1336, 1st round all pass rate = 89.52% (1196), 2nd round still fail rate = 0.07% (1) Average runtime = 44.35 secs py/test/test_ui_unittest.py: Fail rate 2.99% (40 times, timeout count: 16) py/goofy/goofy_unittest.py: Fail rate 2.32% (31 times, timeout count: 31) py/umpire/service/umpire_service_unittest.py: Fail rate 2.25% (30 times) py/shopfloor/shopfloor_unittest.py: Fail rate 0.97% (13 times, timeout count: 12) py/goofy/link_manager_unittest.py: Fail rate 0.75% (10 times) py/goofy/ui_app_controller_unittest.py: Fail rate 0.60% (8 times, timeout count: 5) py/shopfloor/shopfloor_standalone_unittest.py: Fail rate 0.37% (5 times, timeout count: 1) go/src/overlord/test/overlord_e2e_unittest.py: Fail rate 0.37% (5 times, timeout count: 1) py/utils/jsonrpc_utils_unittest.py: Fail rate 0.30% (4 times) py/dkps/dkps_unittest.py: Fail rate 0.15% (2 times) py/umpire/service/overlord_unittest.py: Fail rate 0.07% (1 time) Most of the tests seems pretty stable for now. Still has many room for speed improvement though.
,
Dec 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/4138aaee3b032a49bc67d602fc8c75bef3e5a05e commit 4138aaee3b032a49bc67d602fc8c75bef3e5a05e Author: Peter Shih <pihsun@chromium.org> Date: Thu Dec 22 09:37:24 2016 test: fix a race condition between tests compiling overlord Both tests overlord_e2e_unittest and overlord_unittest compile overlord, which sometimes cause race condition since the binary may be rebuilt by another test while test is trying to run the binary. Fix by building those binary in some temporary directories. BUG= chromium:665306 TEST=make test Change-Id: I3c800bed2e0eabe0fca992efd10831ef107409a4 Reviewed-on: https://chromium-review.googlesource.com/422817 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Ning Huang <wnhuang@chromium.org> [modify] https://crrev.com/4138aaee3b032a49bc67d602fc8c75bef3e5a05e/go/src/overlord/test/overlord_e2e_unittest.py [modify] https://crrev.com/4138aaee3b032a49bc67d602fc8c75bef3e5a05e/go/src/overlord/Makefile [modify] https://crrev.com/4138aaee3b032a49bc67d602fc8c75bef3e5a05e/py/umpire/service/overlord_unittest.py
,
Dec 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/220a96d0818e957b0050347068048560e9b491f2 commit 220a96d0818e957b0050347068048560e9b491f2 Author: Peter Shih <pihsun@chromium.org> Date: Thu Dec 22 09:02:16 2016 overlord: speed up e2e unittest For overlord_e2e_unittest, most of the time spent is actually on determine whether the server is using TLS (which has a 3 seconds timeout). Add an option to ghost to skip the detection and force override whether server is using TLS. Also only do make overlord once in the test, not once in each setUp. This two together speed up the test from 18 seconds to about 3.7 seconds. BUG= chromium:665306 TEST=make test TEST=run go/src/overlord/test/overlord_e2e_unittest.py, should be completed in about 4 seconds. Change-Id: Id5bd5b592483b4f96fd794071de647bc5c5e807c Reviewed-on: https://chromium-review.googlesource.com/423267 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Ning Huang <wnhuang@chromium.org> [modify] https://crrev.com/220a96d0818e957b0050347068048560e9b491f2/go/src/overlord/test/overlord_e2e_unittest.py [modify] https://crrev.com/220a96d0818e957b0050347068048560e9b491f2/go/src/overlord/ghost.go [modify] https://crrev.com/220a96d0818e957b0050347068048560e9b491f2/go/src/overlord/cmd/ghost/main.go [modify] https://crrev.com/220a96d0818e957b0050347068048560e9b491f2/py/tools/ghost.py
,
Dec 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/6df94ed638930fa6e7b6bb5300aaea255bfba3b1 commit 6df94ed638930fa6e7b6bb5300aaea255bfba3b1 Author: Peter Shih <pihsun@chromium.org> Date: Thu Dec 29 07:56:22 2016 test: speed up n1914a_unittest We're using a mock server, so we don't need to wait for server to start up, and can set delay to 0. This improve the running time from 15 seconds to 5 seconds. BUG= chromium:665306 TEST=make test Change-Id: I31de9acb700639a52aae6dc9c2f237293fa1790f Reviewed-on: https://chromium-review.googlesource.com/423024 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/6df94ed638930fa6e7b6bb5300aaea255bfba3b1/py/test/rf/n1914a_unittest.py
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/da3b3b50e5233eee99e773a9109bef85f442b80e commit da3b3b50e5233eee99e773a9109bef85f442b80e Author: Peter Shih <pihsun@chromium.org> Date: Thu Dec 22 10:01:29 2016 umpire_server_proxy_unittest: run unittest in temporary folder Copy the testdata to some temporary folder, and run the test there, so we don't have to modify things in source tree. BUG= chromium:665306 TEST=make test Change-Id: I6f361b5fad054197f4b44c225c1509fb57f508c5 Reviewed-on: https://chromium-review.googlesource.com/422752 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/da3b3b50e5233eee99e773a9109bef85f442b80e/py/umpire/client/umpire_server_proxy_unittest.py [delete] https://crrev.com/55a2ad2203adcfe0903593a3ac263f72f647d7f2/py/umpire/client/testdata/resourcemap
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a commit 0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 05:09:24 2016 fix temporary file not deleted in run_tests.py __del__ is not guaranteed to be called for objects that still exist when the process exits, so change it to explicit calling Close(), to avoid the temporary folders being left over. Install a SIGINT handler to do proper cleanup, even if `make test` is interrupted by ctrl-c. Set TMPDIR environment variable for child processes, so child would put their tempfile inside that directory, thus would be cleaned up even if `make test` is interrupted. Also fix some place that hardcoded path '/tmp' is used instead of tempfile.gettempdir(). BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/cros_factory_root.*, and run `make test`, make sure that no temp file /tmp/cros_factory_root.* are left after the test complete. TEST=in cros_sdk, rm -rf /tmp/cros_factory_root.*, run `make test`, ctrl-c in the middle, make sure that no temp file /tmp/cros_factory_root.* are left after the test aborted. Change-Id: I4f68bc7e698a7ebfd3daf77ec60788881e3723b0 Reviewed-on: https://chromium-review.googlesource.com/422762 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a/py/tools/diff_image.py [modify] https://crrev.com/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a/py/test/testlog.py [modify] https://crrev.com/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a/py/tools/run_tests.py
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a commit 0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 05:09:24 2016 fix temporary file not deleted in run_tests.py __del__ is not guaranteed to be called for objects that still exist when the process exits, so change it to explicit calling Close(), to avoid the temporary folders being left over. Install a SIGINT handler to do proper cleanup, even if `make test` is interrupted by ctrl-c. Set TMPDIR environment variable for child processes, so child would put their tempfile inside that directory, thus would be cleaned up even if `make test` is interrupted. Also fix some place that hardcoded path '/tmp' is used instead of tempfile.gettempdir(). BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/cros_factory_root.*, and run `make test`, make sure that no temp file /tmp/cros_factory_root.* are left after the test complete. TEST=in cros_sdk, rm -rf /tmp/cros_factory_root.*, run `make test`, ctrl-c in the middle, make sure that no temp file /tmp/cros_factory_root.* are left after the test aborted. Change-Id: I4f68bc7e698a7ebfd3daf77ec60788881e3723b0 Reviewed-on: https://chromium-review.googlesource.com/422762 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a/py/tools/diff_image.py [modify] https://crrev.com/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a/py/test/testlog.py [modify] https://crrev.com/0790b9bbfbe9a95fc4e4147f0f93a7d5de79e91a/py/tools/run_tests.py
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/c527418636aff998900d6616ec546efada0bc386 commit c527418636aff998900d6616ec546efada0bc386 Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 05:18:51 2016 fix temporary file not deleted in edid_unittest We need to delete the file by ourself when using tempfile.mkstemp(). BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/tmp*, and run py/probe/functions/edid_unittest.py, make sure that no temp file /tmp/tmp* are left after the test complete. Change-Id: Ieb75a56a6b3ba1e44c35727053dd8cfd61191583 Reviewed-on: https://chromium-review.googlesource.com/422763 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/c527418636aff998900d6616ec546efada0bc386/py/probe/functions/edid_unittest.py
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/2fc91d716b780ba7f45be99a84c09db19306e37b commit 2fc91d716b780ba7f45be99a84c09db19306e37b Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 03:36:15 2016 fix temporary file not deleted in sys_utils.MountPartition When some other options is wrong (for example, source_path does not exist), the function would throw an exception after the temporary folder is created, causing the temporary folder to be left over. Reorder things so that making temporary folder is at the last step. Also catch exception if mount failed and do clean up. BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/mount_partition.*, and run `make test`, make sure that no temp file /tmp/mount_partition.* are left after the test complete. Change-Id: I0d9592d8557e15217709bf96a2b33e3581128bcc Reviewed-on: https://chromium-review.googlesource.com/423022 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/2fc91d716b780ba7f45be99a84c09db19306e37b/py/utils/sys_utils.py
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/4dc2e47060a75a165c6a01c4e6da934147dfc560 commit 4dc2e47060a75a165c6a01c4e6da934147dfc560 Author: Peter Shih <pihsun@chromium.org> Date: Wed Dec 28 05:33:28 2016 fix temporary file not deleted in time_sanitizer_unittest We're actually only using the name of NamedTemporaryFile(). (The file is deleted after that line), and didn't delete the file after we reuse it. Change the code to override unittest.TestCase.run(), and use context manager to manage the temporary folder, so it is automatically deleted on exit. BUG= chromium:665306 TEST=make test TEST=in cros_sdk, rm -rf /tmp/tmp*, and run py/tools/time_sanitizer_unittest.py, make sure that no temp file /tmp/tmp* are left after the test complete. Change-Id: Iba6ffbed0207c8518c5143a4ab200883589a5da3 Reviewed-on: https://chromium-review.googlesource.com/422764 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/4dc2e47060a75a165c6a01c4e6da934147dfc560/py/tools/time_sanitizer_unittest.py
,
Dec 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/5434c930fc9aa7b8fd5717806575c71456256bec commit 5434c930fc9aa7b8fd5717806575c71456256bec Author: Peter Shih <pihsun@chromium.org> Date: Mon Dec 26 08:45:56 2016 fix flaky overlord_e2e_unittest For the response of websocket, it can actually be cut in half of the message, or being concat with other message, causing test to fail about 10% of the time. We need to properly split using CRLF to recover the real message. BUG= chromium:665306 TEST=make test TEST=run go/src/overlord/test/overlord_e2e_unittest.py many times, and observe no failure. Change-Id: Ia39d7cb43514022a725558da9c23a8c8d5733722 Reviewed-on: https://chromium-review.googlesource.com/424235 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Ning Huang <wnhuang@chromium.org> [modify] https://crrev.com/5434c930fc9aa7b8fd5717806575c71456256bec/go/src/overlord/test/overlord_e2e_unittest.py
,
Jan 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/354e1a7264d23337811db965f5bf964eae46f087 commit 354e1a7264d23337811db965f5bf964eae46f087 Author: Peter Shih <pihsun@chromium.org> Date: Tue Jan 03 07:41:48 2017 test: speed up lan_scpi_unittest We're using a mock server, so we don't need to wait for server to start up, and can set delay to 0 and timeout to 1. This improve the running time from 10 seconds to 3.5 seconds. BUG= chromium:665306 TEST=make test Change-Id: I49bcb710f80a3e3947a5ea947ea860ef9d8d0615 Reviewed-on: https://chromium-review.googlesource.com/424457 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/354e1a7264d23337811db965f5bf964eae46f087/py/test/rf/lan_scpi_unittest.py
,
Jan 25 2017
,
Mar 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/a027452a5a80ef3ca8746a6a351874f73e8f116a commit a027452a5a80ef3ca8746a6a351874f73e8f116a Author: Peter Shih <pihsun@chromium.org> Date: Wed Mar 22 13:28:44 2017 speed up umpire_service_unittest Lower the time limit for service started / restarting monitor, so the test can be run faster. This reduced the running time from 6 seconds to 0.5 seconds. BUG= chromium:665306 TEST=make test Change-Id: I33bcb5f16cc2576dfdfcf98127aadad69a1e1641 Reviewed-on: https://chromium-review.googlesource.com/423300 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Youcheng Syu <youcheng@google.com> [modify] https://crrev.com/a027452a5a80ef3ca8746a6a351874f73e8f116a/py/umpire/service/umpire_service_unittest.py
,
Mar 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/8da6212fea93af2a8d6689973a4ed40c6f3284a3 commit 8da6212fea93af2a8d6689973a4ed40c6f3284a3 Author: Peter Shih <pihsun@chromium.org> Date: Wed Mar 29 12:45:21 2017 speed up valid_hwid_db_unittest. Most boards doesn't have HWID testdata, so if we check whether the testdata exist first, we can skip many unnecessary load of database. This speed up the unittes time from about 16 seconds to about 5.5 seconds. BUG= chromium:665306 TEST=make test Change-Id: I53288b9beb2532ac87393caba8e0cb2c74e85546 Reviewed-on: https://chromium-review.googlesource.com/461601 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/8da6212fea93af2a8d6689973a4ed40c6f3284a3/py/hwid/v3/valid_hwid_db_unittest.py
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b4a9b0ad206582cbb4c231164fbb6ff1d462b51a commit b4a9b0ad206582cbb4c231164fbb6ff1d462b51a Author: Peter Shih <pihsun@chromium.org> Date: Thu Apr 06 14:25:47 2017 speed up test_factory_flow_unittest. For BuildBoard, getting property arch needs some expensive process calls, but the property may not be used at all. Make the property not been set in initializer, but as a type_utils.LazyProperty, so the expensive operation can be avoided if it's not used at all. One of the place that doesn't use arch is test_factory_flow_unittest (only used full_name), and the running time is reduced from 8.5 seconds to 2.6 seconds. BUG= chromium:665306 TEST=make test Change-Id: I45fb96cdaa2789caba5954d429c08cfca63935fd Reviewed-on: https://chromium-review.googlesource.com/423299 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> Reviewed-by: Shun-Hsing Ou <shunhsingou@chromium.org> [modify] https://crrev.com/b4a9b0ad206582cbb4c231164fbb6ff1d462b51a/py/tools/build_board.py
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b6ba649d5cf7564ff0ebabec532a44f512be023a commit b6ba649d5cf7564ff0ebabec532a44f512be023a Author: Peter Shih <pihsun@chromium.org> Date: Thu Apr 06 14:25:47 2017 Speed up link_manager_unittest.py Refactor link_manager.py a bit, and replace all uses of time.sleep in unittest to event.wait. The time for link_manager_unittest is reduced from 12 seconds to about 2 seconds. BUG= chromium:665306 TEST=make test Change-Id: Ic9beabf390170a9bab8d7074ace03b26390261a8 Reviewed-on: https://chromium-review.googlesource.com/461784 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/b6ba649d5cf7564ff0ebabec532a44f512be023a/py/goofy/link_manager.py [modify] https://crrev.com/b6ba649d5cf7564ff0ebabec532a44f512be023a/py/goofy/link_manager_unittest.py
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/e12886c54b00b5a31b02ca136d4514e624a2967d commit e12886c54b00b5a31b02ca136d4514e624a2967d Author: Peter Shih <pihsun@chromium.org> Date: Thu Apr 06 14:25:48 2017 pytest_utils_unittest: Put test pytest modules into /tmp The temporary pytest modules was created directly inside py/test/pytests, which cause problem for make_par_unittest, since rsync would complain if there's file that are changed when rsync is running. Change to put the temporary test pytests to be in a temporary directory, and modify sys.path so that Python use the module. BUG= chromium:665306 TEST=make test Change-Id: Id6596edb568e7bf1099e949590228edf851157ad Reviewed-on: https://chromium-review.googlesource.com/464488 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/e12886c54b00b5a31b02ca136d4514e624a2967d/py/test/utils/pytest_utils_unittest.py
,
Apr 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/472185b8c7a94f3a7503df72e029da80220ef94e commit 472185b8c7a94f3a7503df72e029da80220ef94e Author: Peter Shih <pihsun@chromium.org> Date: Thu Apr 06 16:07:48 2017 Fix net_utils.FindUnusedTCPPort. The code did not properly closed the TCPServer, and was relying on GC to release the port before the returned port is actually used. Actually starting a TCPServer is also unnecessary since we only need to bind the port but not listen, and TCPServer would do both. Listening only on LOCALHOST also cause problem sometimes, since some code use the returned port to bind on 0.0.0.0. Change this to open a socket, bind it on 0.0.0.0 port 0, and return the bound port after closing it. There always would be race condition if the port is used by others between the port is returned and actually used by caller, but since almost all usage of this function is in unittest, and the chance of race condition is very small, this can be ignored. BUG= chromium:665306 TEST=make test Change-Id: Ib67e2cd75d5405a0d8a84634ce5f3727a34dd094 Reviewed-on: https://chromium-review.googlesource.com/469531 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Pi-Hsun Shih <pihsun@chromium.org> [modify] https://crrev.com/472185b8c7a94f3a7503df72e029da80220ef94e/py/utils/net_utils.py
,
Apr 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/54f6ecf82113b6b6eb758145db88ac5bb5f3a6bd commit 54f6ecf82113b6b6eb758145db88ac5bb5f3a6bd Author: Peter Shih <pihsun@chromium.org> Date: Mon Apr 17 10:30:44 2017 Overlord: fix a small race condition. The socket IO server in Overlord is used in socket server, but is initialized together with the web server. Since these two server runs in different goroutine, it's possible that the socket server is up first, and serving request before the socket IO server is initialized, causing segfault since ovl.ioserver would be nil. Fix this by moving initialization of socket IO server to be with the socket server. BUG= chromium:665306 TEST=make test Change-Id: I6a930085e7d563505946f1f76c439920313781f0 Reviewed-on: https://chromium-review.googlesource.com/471146 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Ning Huang <wnhuang@chromium.org> [modify] https://crrev.com/54f6ecf82113b6b6eb758145db88ac5bb5f3a6bd/go/src/overlord/overlord.go
,
Apr 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/3ea98ca352ab0e2551a6a1090735e506ab86bbe2 commit 3ea98ca352ab0e2551a6a1090735e506ab86bbe2 Author: Peter Shih <pihsun@chromium.org> Date: Wed Apr 19 09:32:36 2017 speed up shopfloor_unittest.py Most time of shopfloor_unittest.py and shopfloor_standalone_unittest.py is spent on running tearDown, which stop the underlying rsync daemon for both update_server and log_server, and the rsync daemon takes about 0.4 seconds to stop. Since there are about 20 test cases, and each one start and stop rsync daemon twice, this adds up to lots of time. We're not testing anything related to rsync daemon, so we can skip initialize of rsync server, and we don't have to wait it to stop. This speed up shopfloor_standalone_unittest.py from about 31 seconds to 14 seconds, and shopfloor_unittest.py from about 26 seconds to 10 seconds. This also makes the time for "make test" from 31 seconds to about 15 seconds, since these two tests are by far the slowest unittest. BUG= chromium:665306 TEST=make test Change-Id: I7cbe39b11f63e5d7f17fdc85716650ddd2650b50 Reviewed-on: https://chromium-review.googlesource.com/461778 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Joel Kitching <kitching@chromium.org> [modify] https://crrev.com/3ea98ca352ab0e2551a6a1090735e506ab86bbe2/py/shopfloor/shopfloor_unittest.py [add] https://crrev.com/3ea98ca352ab0e2551a6a1090735e506ab86bbe2/py/shopfloor/dummy_updater.py [modify] https://crrev.com/3ea98ca352ab0e2551a6a1090735e506ab86bbe2/py/shopfloor/factory_update_server.py
,
Apr 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/32e46d2e7f3211f2af2e39b9f158695d0039e084 commit 32e46d2e7f3211f2af2e39b9f158695d0039e084 Author: Peter Shih <pihsun@chromium.org> Date: Wed Apr 19 09:32:36 2017 Fix flaky dkps_unittest. Wait for the server to be ready before tests of upload things. Also use a random port instead of fixed port. BUG= chromium:665306 TEST=make test Change-Id: I7b4f5d7c4942af917d075e53114a8c4b38305b06 Reviewed-on: https://chromium-review.googlesource.com/469530 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Mao Huang <littlecvr@chromium.org> [modify] https://crrev.com/32e46d2e7f3211f2af2e39b9f158695d0039e084/py/dkps/dkps_unittest.py
,
Apr 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b79d5999781bc3f684a330cd69fafa7f14515afa commit b79d5999781bc3f684a330cd69fafa7f14515afa Author: Peter Shih <pihsun@chromium.org> Date: Wed Apr 19 09:32:36 2017 Fix flaky ui_app_controller_unittest. Wait for the connect event before doing the ShowUI test. Also there's a chance that the server shut down before our connection in Stop, so we should ignore exceptions in the attempt to connect. BUG= chromium:665306 TEST=make test TEST=Run make test 50 times, and see no failure of ui_app_controller_unittest.py. Change-Id: Ic96fe3e1da55eabce361066335f22a5f1f74edd5 Reviewed-on: https://chromium-review.googlesource.com/469532 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Joel Kitching <kitching@chromium.org> [modify] https://crrev.com/b79d5999781bc3f684a330cd69fafa7f14515afa/py/goofy/ui_app_controller.py [modify] https://crrev.com/b79d5999781bc3f684a330cd69fafa7f14515afa/py/utils/net_utils.py [modify] https://crrev.com/b79d5999781bc3f684a330cd69fafa7f14515afa/py/goofy/ui_app_controller_unittest.py
,
Apr 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b2da2a5491fd1c9184d83f8523677fdc8b261e8d commit b2da2a5491fd1c9184d83f8523677fdc8b261e8d Author: Peter Shih <pihsun@chromium.org> Date: Wed Apr 19 12:28:54 2017 Fix flaky umpire_service_unittest. Refactor umpire_service.py to fix some race conditions that cause the unittest to be flaky (Call to processEnded may be before reactor.spawnProcess return.) The option 'daemon' is removed when refactoring since no service is using it, and supporting the option would complicate the code. Also add some test cases to test that services can be correctly stopped. BUG= chromium:665306 TEST=make test TEST=./setup/cros_docker.sh umpire test CQ-DEPEND=CL:461739 Change-Id: I00bc14e736a80fdf8602df764b6d408089291bae Reviewed-on: https://chromium-review.googlesource.com/459277 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Joel Kitching <kitching@chromium.org> [modify] https://crrev.com/b2da2a5491fd1c9184d83f8523677fdc8b261e8d/py/umpire/service/umpire_service_unittest.py [modify] https://crrev.com/b2da2a5491fd1c9184d83f8523677fdc8b261e8d/py/umpire/service/instalog.py [modify] https://crrev.com/b2da2a5491fd1c9184d83f8523677fdc8b261e8d/py/umpire/service/umpire_service.py
,
May 11 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/d9fc7603c6cb0b14ed677f1486437c198d6f1d12 commit d9fc7603c6cb0b14ed677f1486437c198d6f1d12 Author: Peter Shih <pihsun@chromium.org> Date: Thu May 11 12:31:54 2017 instalog: Properly close server in input_http. The TCPServer accept socket needs to be closed by server_close(), or the port would still be open after the request handling thread is finished. This would cause about 1.5% of unittest failure because of timeout in testShutdown, since if the post thread starts slow (after the time.sleep(0.2)), request.post would still be able to connect to the port after sandbox.Stop(), but would hang forever. Also add a new unittest for this, and refactor some unittest to make it more robust. BUG= chromium:665306 TEST=make test Change-Id: I44ab3d14a97ac214f44b9aef8b30116492ff0da5 Reviewed-on: https://chromium-review.googlesource.com/497239 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Chun-Tsen Kuo <chuntsen@chromium.org> [modify] https://crrev.com/d9fc7603c6cb0b14ed677f1486437c198d6f1d12/py/instalog/plugins/input_http.py [modify] https://crrev.com/d9fc7603c6cb0b14ed677f1486437c198d6f1d12/py/instalog/plugins/input_http_unittest.py
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/876b296a39acf4835c7821f3411fd5484f77da10 commit 876b296a39acf4835c7821f3411fd5484f77da10 Author: Peter Shih <pihsun@chromium.org> Date: Fri May 12 12:28:37 2017 alternate way for test_ui.Run(blocking=False) test_ui.Run(blocking=False) has some weird problem involving python threads and signal handlers, causing race condition and deadlock about 1% ~ 2% of the time in unittest. This patch proposes an alternate way to achieve the same effect. That is always run UI event loop in main thread, run other test codes in background, and only do the test pass / fail through event loop. This solution doesn't involve signal handling, and should be more robust than the old one. BUG= chromium:665306 TEST=make test Change-Id: I46b06809efe19d3213a6a1715883caad9c4bc67d Reviewed-on: https://chromium-review.googlesource.com/431331 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/wireless_antenna.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/countdown.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/power_under_stress.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/memory_size.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/ping_test.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/ac_power.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/touchpad_hover.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/verify_components.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/factory_task.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/test_ui_unittest.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/flash_netboot.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/vswr/vswr.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/camera_fixture.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/battery_current.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/robot_movement.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/chameleon/chameleon.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/verify_components_unittest.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/rf_graphyte/rf_graphyte.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/call_shopfloor.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/suspend_resume.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/wireless_radiotap.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/doc/test_ui_api.rst [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/cellular_switch_firmware.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/robot_movement_unittest.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/wifi_throughput.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/developer_switch.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/station_entry.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/test_ui.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/station_entry_unittest.py [modify] https://crrev.com/876b296a39acf4835c7821f3411fd5484f77da10/py/test/pytests/spatial_sensor_calibration.py
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/4e6da9d0bf8a65a3e75c36134bff4e3ed6204a76 commit 4e6da9d0bf8a65a3e75c36134bff4e3ed6204a76 Author: Peter Shih <pihsun@chromium.org> Date: Fri May 12 12:28:37 2017 Fix flaky overlord_e2e_unittest. Similar issue as in CL:424235, response for shell command can also be cut to multiple packets. Buffer them all and check after the connection is closed. BUG= chromium:665306 TEST=make test TEST=run go/src/overlord/test/overlord_e2e_unittest.py many times, and observe no failure. Change-Id: I484b4b3f1ef016d994b9060b8be20c8f093095f9 Reviewed-on: https://chromium-review.googlesource.com/503988 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/4e6da9d0bf8a65a3e75c36134bff4e3ed6204a76/go/src/overlord/test/overlord_e2e_unittest.py
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/077de3538c10ec787d3f4dce2fb8b9db862023fc commit 077de3538c10ec787d3f4dce2fb8b9db862023fc Author: Peter Shih <pihsun@chromium.org> Date: Fri May 12 12:28:37 2017 Fix flaky goofy_unittest. There's a small chance that our websocket connects before the first test start. Add check for the case too. BUG= chromium:665306 TEST=make test Change-Id: I97f8337d8d893a6ac19187f1e509ecc38105d2fc Reviewed-on: https://chromium-review.googlesource.com/503989 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/077de3538c10ec787d3f4dce2fb8b9db862023fc/py/goofy/goofy_unittest.py
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/ce9490e0986237232c59cbbadd642cce4609d8d9 commit ce9490e0986237232c59cbbadd642cce4609d8d9 Author: Peter Shih <pihsun@chromium.org> Date: Fri May 12 14:53:38 2017 Speed up goofy_unittest. The event_server.shutdown() has same issue as TCPServer.shutdown(), that would take 0.5 seconds to complete. Change it to use net_utils.ShutdownTCPServer, and also change net_utils.ShutdownTCPServer to work on UnixStreamServer (which is a subclass of TCPServer). This reduce the running time of goofy_unittest from about 18 seconds to about 14 seconds, and the unittest is no longer the slowest one in `make test`. CQ-DEPEND=CL:503991 BUG= chromium:665306 TEST=make test Change-Id: I966857b6a9915ec4a3d445dc7d2d410da9fcc60a Reviewed-on: https://chromium-review.googlesource.com/503990 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/ce9490e0986237232c59cbbadd642cce4609d8d9/py/goofy/goofy.py [modify] https://crrev.com/ce9490e0986237232c59cbbadd642cce4609d8d9/py/utils/net_utils.py
,
May 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/53f94cf8e0bde1199c05003ee676834e1a7e9e03 commit 53f94cf8e0bde1199c05003ee676834e1a7e9e03 Author: Peter Shih <pihsun@chromium.org> Date: Fri May 12 14:53:38 2017 Fix locking in file_utils.FileLockContextManager The lock isn't acquired when entering the context manager, so simultaneously calling Log and Close would raise exception. Also add a unittest for this case. BUG= chromium:665306 TEST=make test Change-Id: I466c036d53e9f7ade39c51ae7b3d34bac7c2ab67 Reviewed-on: https://chromium-review.googlesource.com/503991 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/53f94cf8e0bde1199c05003ee676834e1a7e9e03/py/utils/file_utils.py [modify] https://crrev.com/53f94cf8e0bde1199c05003ee676834e1a7e9e03/py/utils/file_utils_unittest.py
,
May 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/e9ac6a0af75d08b92372bc684127651d894fa18b commit e9ac6a0af75d08b92372bc684127651d894fa18b Author: Peter Shih <pihsun@chromium.org> Date: Mon May 15 13:07:56 2017 Fix GetUnusedPort causing unittest flakyness. The original GetUnusedPort use the "bind on port 0" technique to find an unused port. Since the port got in this way would be in the ephemeral port range, the approach has some drawback: * The port would not only has race condition for other server calling GetUnusedPort, but it's also possible that the port to conflict with all outgoing connections. * A typical usage of GetUnusedPort is as follows: port = net_utils.GetUnusedPort() server = StartServerAsync(port=port) sync_utils.WaitFor(lambda: PingServer(port=port), 2) But if the server port is in ephemeral port range, there's a small chance that the PingServer() would actually connects to itself, binding the port, and the StartServerAsync() would fail. (See http://sgros.blogspot.tw/2013/08/tcp-client-self-connect.html) The most "correct" way for solving this would be enforcing that all server act correctly when directly binding to port 0, and has some way of reporting the bound port back to client, but that would be lots of work. A simple workaround is just randomly choose a port NOT in ephemeral port range, and check if it's used. Since the ephemeral port range is >= 32768 on a typical Ubuntu machine, we choose 16384~32767. This reduced the fail rate of "make test" by about 1%, from 25 fails in 1457 runs to 12 fails in 1457 runs. Also refactor shopfloor_unittest a bit so it's less flaky. BUG= chromium:665306 TEST=make test Change-Id: Idf6feda8ce60698a467898e198016ba5f78d50b3 Reviewed-on: https://chromium-review.googlesource.com/503994 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Youcheng Syu <youcheng@google.com> [modify] https://crrev.com/e9ac6a0af75d08b92372bc684127651d894fa18b/py/umpire/client/umpire_server_proxy_unittest.py [modify] https://crrev.com/e9ac6a0af75d08b92372bc684127651d894fa18b/py/utils/net_utils.py [modify] https://crrev.com/e9ac6a0af75d08b92372bc684127651d894fa18b/py/shopfloor/shopfloor_unittest.py
,
May 18 2017
Some more statistics after CL:508489: count = 3129 [3113, 16, 0] (99.49%, 0.51%, 0.00%) Average time = 16.48 seconds py/umpire/service/umpire_service_unittest.py: Fail rate 0.19% (6 times) py/test/testlog_seq_unittest.py: Fail rate 0.10% (3 times) py/instalog/plugins/input_socket_unittest.py: Fail rate 0.10% (3 times) py/shopfloor/shopfloor_standalone_unittest.py: Fail rate 0.03% (1 time) py/shopfloor/shopfloor_unittest.py: Fail rate 0.03% (1 time) go/src/overlord/test/overlord_e2e_unittest.py: Fail rate 0.03% (1 time, timeout count: 1) py/test/event_log_unittest.py: Fail rate 0.03% (1 time) py/utils/net_utils_unittest.py: Fail rate 0.03% (1 time) * Five slowest tests: py/goofy/js/goofy_js_unittest.py: 15.39 secs py/goofy/goofy_unittest.py: 14.17 secs py/shopfloor/shopfloor_standalone_unittest.py: 13.95 secs py/doc/doc_unittest.py: 11.38 secs py/shopfloor/shopfloor_unittest.py: 10.14 secs So we have 99.5% "make test" pass rate, and an average time of about 16 seconds. Would mark this issue as fixed after all changes are merged.
,
May 18 2017
I think the unit tests are pretty stable and fast enough. Really thanks for your great effort!
,
May 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/f20115f9625288d3a0849b125b1a64a13c2029f2 commit f20115f9625288d3a0849b125b1a64a13c2029f2 Author: Peter Shih <pihsun@chromium.org> Date: Fri May 19 09:25:19 2017 Speed up and fix flaky input_http_unittest. The requests package can't stream multipart/form-data file upload (https://github.com/kennethreitz/requests/issues/1584), and handling large string (even just 64MB) is really slow in Python, and sometimes would cause the test to timeout when the system is on heavy load (e.g. when running "make test"). Change all large requests from using requests package to using curl and file-based upload. This makes the test faster (from about 12~20 seconds to 9 seconds) and never fail. BUG= chromium:665306 TEST=make test Change-Id: Id647683bfac42e38b2f3e735326fd473577b2b08 Reviewed-on: https://chromium-review.googlesource.com/503992 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Chun-Tsen Kuo <chuntsen@chromium.org> [modify] https://crrev.com/f20115f9625288d3a0849b125b1a64a13c2029f2/py/instalog/plugins/input_http_unittest.py
,
May 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/03bf4064e85301382cbe37bb3585ec5ee9a35ce7 commit 03bf4064e85301382cbe37bb3585ec5ee9a35ce7 Author: Peter Shih <pihsun@chromium.org> Date: Fri May 19 09:25:18 2017 Fix all unittest flakyness issues with GetUnusedPort. After CL:503994, there's still about 0.5% chance that port got from net_utils.GetUnusedPort would be the same for two different unittest by chance, thus causing the test to fail with error "Address already in use". To completely solve this issue, we add a simple UnixStreamServer in run_tests.py, that would manage current port range. When GetUnusedPort sees that a special environment variable is set, it would try to connect to that unix domain socket and acquire a random port that is guaranteed not used by any other caller of GetUnusedPort. The fail rates drops from about 1.1% (10 out of 919 runs) to about 0.55% (15 out of 2743 runs), and there's zero test that fails with "Address already in use" in these 15 fail runs. Also renamed GetUnusedPort to FindUnusedPort, to be consistent to FindUnusedTCPPort, and merged FindConsecutiveUnusedPorts to FindUnusedPort, so the FindConsecutiveUnusedPorts can use the same mechanic to avoid port conflict. BUG= chromium:665306 TEST=make test TEST=run "make test" many times, observe that there's no error with "Address already in use". Change-Id: If08369b648190fad8cabe04e7d65c269601c0ec2 Reviewed-on: https://chromium-review.googlesource.com/508489 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Youcheng Syu <youcheng@google.com> [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/umpire/client/umpire_server_proxy_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/umpire/e2e_test/e2e_test.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/instalog/plugins/input_http_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/umpire/rpc_dut_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/umpire/rpc_cli_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/factory_flow/run_automated_tests.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/go/src/overlord/test/overlord_e2e_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/instalog/plugins/socket_e2e_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/tools/run_tests.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/umpire/daemon_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/umpire/service/overlord_unittest.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/utils/net_utils.py [modify] https://crrev.com/03bf4064e85301382cbe37bb3585ec5ee9a35ce7/py/instalog/plugins/input_socket_unittest.py
,
May 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/795787a4fc4c3482b781677cca7e1636f06854df commit 795787a4fc4c3482b781677cca7e1636f06854df Author: Peter Shih <pihsun@chromium.org> Date: Fri May 19 20:57:24 2017 goofy_js_unittest: Use the --checks-only flag. Use the --checks-only flag to skip output optimization, and fix all warnings from the new closure compiler and closure library. The speed of the unittest doesn't change much (about 17 seconds before and about 16 seconds after the change), since although the --checks-only speeds up compilation, the new closure library compiles slower than the old one. CQ-DEPEND=CL:505991 BUG= chromium:665306 TEST=make test Change-Id: I7e1db6d072355a78e200d107f73edfe0083549a1 Reviewed-on: https://chromium-review.googlesource.com/505809 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Pi-Hsun Shih <pihsun@chromium.org> [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/js/device_manager.js [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/js/i18n.js [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/plugins/status_monitor/static/status_monitor.js [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/static/Makefile [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/js/goofy_js_unittest.py [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/js/goofy.js [modify] https://crrev.com/795787a4fc4c3482b781677cca7e1636f06854df/py/goofy/js/diagnosis_tool.js
,
May 22 2017
,
Jul 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/a55bec5cdd52c1b4a0cac20c1c329e721c3021d0 commit a55bec5cdd52c1b4a0cac20c1c329e721c3021d0 Author: Peter Shih <pihsun@chromium.org> Date: Fri Jul 14 14:37:07 2017 Speed up goofy_unittest.py goofy_unittest would try to get "DUT" info, which is unnecessary when running unittest, and spends lots of time since many of them call external programs. This speed up goofy_unittest from about 20 seconds to about 13 seconds. BUG= chromium:665306 TEST=make test TEST=run goofy_unittest.py, and check that it runs faster. Change-Id: If494802d2a3b59c5048982b47c882aa55f417d57 Reviewed-on: https://chromium-review.googlesource.com/571111 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/a55bec5cdd52c1b4a0cac20c1c329e721c3021d0/py/goofy/goofy_unittest.py
,
Jul 31 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/fef3654c620b0a2ffb6ef237ecbbd6696859ba0d commit fef3654c620b0a2ffb6ef237ecbbd6696859ba0d Author: Peter Shih <pihsun@chromium.org> Date: Mon Jul 31 12:54:54 2017 Fix flaky output_http_unittest Sometimes the output_http can't connect in the first try, and would wait for 60 seconds before another attempt, causing the unittest to timeout. Reduce the retry interval to 1 second in unittest. BUG= chromium:665306 TEST=make test TEST=make test many times. Change-Id: Ib1d800b2b9358c3e5c812746c38a7744866930c7 Reviewed-on: https://chromium-review.googlesource.com/593432 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Chun-Tsen Kuo <chuntsen@chromium.org> [modify] https://crrev.com/fef3654c620b0a2ffb6ef237ecbbd6696859ba0d/py/instalog/plugins/output_http_unittest.py
,
Aug 1 2017
,
Oct 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/0a6ae8d72a9afa2cc82967599799506c0f92b248 commit 0a6ae8d72a9afa2cc82967599799506c0f92b248 Author: Peter Shih <pihsun@chromium.org> Date: Mon Oct 23 16:14:09 2017 goofy: Speed up goofy_unittest. This is the same as what CL:503990 does. goofy_server didn't cause a problem before CL:694726 since most tests in unittest wasn't using GoofyUITest. Also removed some unused variable in goofy_unittest. BUG= chromium:665306 TEST=make test, and observe that goofy_unittest speed up from about 23 seconds to 15 seconds. Change-Id: If407e734c9e7709c6519af00162342e9343e8cb6 Reviewed-on: https://chromium-review.googlesource.com/732873 Commit-Ready: Pi-Hsun Shih <pihsun@chromium.org> Tested-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Shen-En Shih <petershih@chromium.org> [modify] https://crrev.com/0a6ae8d72a9afa2cc82967599799506c0f92b248/py/goofy/goofy.py [modify] https://crrev.com/0a6ae8d72a9afa2cc82967599799506c0f92b248/py/goofy/goofy_unittest.py
,
Jan 22 2018
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by hungte@chromium.org
, Nov 15 2016