factory: Revise dut_upload |
||||||
Issue description
Currently the dut_upload will try to rsync /var/{factory,log} to shopfloor (or umpire) log/dut_upload folder.
The folder was deviceid_reimageid. However, we've seen many times that devices may reset reimageid many times (probably due to factory_restart -a), especially when the disk is almost full.
Although /var/factory is cleared, the log/ is not. It results a problem that especially in long-running devices (FRT), or stations (especially if they used "factory_restart -a" for running next DUT, before our station-based design was introduced), the var/log/*.log may be pretty huge, so each flush (rsync) will duplicate a huge dut_upload/deviceid_reimageid folder.
We tried to reduce that by
- for stations, don't put imageid in folder name
- new station-based test list that don't do factory_restart -a
But I think we should think again about the definition of reimage_id.
- When should a reimageid reset?
- What do we want for a reimageid?
- Should we just use /var/lib/dbus/machine-id for reimageid?
- Should we revise dut_upload so that the first level of folder is
deviceid/machineid/log
deviceid/reimageid/factory
Since 'log' should change per machine wiping, while reimageid seem to be something like "factory session id".
,
Apr 13 2017
,
May 18 2017
,
Jul 4 2017
Currently this is done by goofy/plugins/system_log_manager.py. I think we need to revise it so that there should be sync_device_log_paths and sync_reimage_log_paths so that /var/factory goes to sync_device_log_paths while /var/log goes to sync_reimage_log_paths.
,
Jul 14 2017
,
Jul 26 2017
Assign to chuntsen - low priority item.
,
Nov 1 2017
per discussion with littlecvr and pihsun, we feel that - /var/factory/log contains instalog stuff now that we should not upload to server via rsync - only factory.log is needed - probably not worthy building folders by reimage_id - in fact maybe we should enable this only for Create projects
,
Nov 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672 commit b4f7edad38b723b5b0a1e5fb4aa999fcb9310672 Author: Hung-Te Lin <hungte@chromium.org> Date: Thu Nov 02 07:13:01 2017 goofy: Fix duplicated useless logs filled up factory server space. /var/factory/log has Instalog and Testlog stuff that should have already been synced by flush_testlog so we should not sync it again using rsync. In other words, only 'factory.log' is needed. Meanwhile, we think it's very rare for people to debug using dut_upload folder so it's probably fine to only keep the latest data (removing reimage_id). BUG=chromium:707172 TEST=make test Change-Id: I8319426e3c79120481d046395b5c8c98aef6daf8 Reviewed-on: https://chromium-review.googlesource.com/749027 Commit-Ready: Hung-Te Lin <hungte@chromium.org> Tested-by: Hung-Te Lin <hungte@chromium.org> Reviewed-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Han Chen <stimim@chromium.org> [modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/system_log_manager.py [modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/goofy_plugin_chromeos.json [modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/system_log_manager_unittest.py [modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/goofy_plugin_station.json
,
Nov 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3 commit 431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3 Author: Hung-Te Lin <hungte@chromium.org> Date: Thu Nov 02 09:01:31 2017 goofy: Fix duplicated useless logs filled up factory server space. /var/factory/log has Instalog and Testlog stuff that should have already been synced by flush_testlog so we should not sync it again using rsync. In other words, only 'factory.log' is needed. Meanwhile, we think it's very rare for people to debug using dut_upload folder so it's probably fine to only keep the latest data (removing reimage_id). BUG=chromium:707172 TEST=make test Change-Id: I8319426e3c79120481d046395b5c8c98aef6daf8 Reviewed-on: https://chromium-review.googlesource.com/749027 Commit-Ready: Hung-Te Lin <hungte@chromium.org> Tested-by: Hung-Te Lin <hungte@chromium.org> Reviewed-by: Pi-Hsun Shih <pihsun@chromium.org> Reviewed-by: Wei-Han Chen <stimim@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/749613 Reviewed-by: Hung-Te Lin <hungte@chromium.org> Commit-Queue: Hung-Te Lin <hungte@chromium.org> Trybot-Ready: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/system_log_manager.py [modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/goofy_plugin_chromeos.json [modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/system_log_manager_unittest.py [modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/goofy_plugin_station.json |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by hungte@chromium.org
, Mar 31 2017Hmmmm definitely not machine-id, since it changes per reboot on CrOS. I think we should create a deviceid in /var/factory reimageid or installation_id or factory_id in /var/factory/state or /var/factory/log another id (not sure what name best) in /var/log/.XXXID And in dut_upload, we do: deviceid/factory_${factory_id} <= /var/factory deviceid/system_${system_id} <= /var/log