New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 707172 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Last visit > 30 days ago
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug

Blocked on:
issue 711200



Sign in to add a comment

factory: Revise dut_upload

Project Member Reported by hungte@chromium.org, Mar 31 2017

Issue description

Currently the dut_upload will try to rsync /var/{factory,log} to shopfloor (or umpire) log/dut_upload folder.

The folder was deviceid_reimageid. However, we've seen many times that devices may reset reimageid many times (probably due to factory_restart -a), especially when the disk is almost full.

Although /var/factory is cleared, the log/ is not. It results a problem that especially in long-running devices (FRT), or stations (especially if they used "factory_restart -a" for running next DUT, before our station-based design was introduced), the var/log/*.log may be pretty huge, so each flush (rsync) will duplicate a huge dut_upload/deviceid_reimageid folder.

We tried to reduce that by
 - for stations, don't put imageid in folder name
 - new station-based test list that don't do factory_restart -a

But I think we should think again about the definition of reimage_id.

 - When should a reimageid reset? 
 - What do we want for a reimageid?
 - Should we just use /var/lib/dbus/machine-id for reimageid?
 - Should we revise dut_upload so that the first level of folder is
   deviceid/machineid/log
   deviceid/reimageid/factory
 Since 'log' should change per machine wiping, while reimageid seem to be something like "factory session id".
 

Comment 1 by hungte@chromium.org, Mar 31 2017

Hmmmm definitely not machine-id, since it changes per reboot on CrOS.

I think we should create a
 deviceid in /var/factory
 reimageid or installation_id or factory_id in /var/factory/state or /var/factory/log
 another id (not sure what name best) in /var/log/.XXXID

And in dut_upload, we do:

  deviceid/factory_${factory_id} <= /var/factory
  deviceid/system_${system_id} <= /var/log

Comment 2 by hungte@chromium.org, Apr 13 2017

Blockedon: 711200

Comment 3 by hungte@chromium.org, May 18 2017

Cc: petershih@chromium.org
Currently this is done by goofy/plugins/system_log_manager.py.

I think we need to revise it so that there should be sync_device_log_paths and sync_reimage_log_paths so that /var/factory goes to sync_device_log_paths while /var/log goes to sync_reimage_log_paths.

Comment 5 by hungte@chromium.org, Jul 14 2017

Cc: -youcheng@chromium.org -petershih@chromium.org -yhong@chromium.org -chuntsen@chromium.org -kitching@chromium.org -yllin@chromium.org -stimim@chromium.org chromeos-factory-eng@google.com

Comment 6 by hungte@chromium.org, Jul 26 2017

Owner: chuntsen@chromium.org
Status: Assigned (was: Untriaged)
Assign to chuntsen - low priority item.
per discussion with littlecvr and pihsun, we feel that
 - /var/factory/log contains instalog stuff now that we should not upload to server via rsync
 - only factory.log is needed
 - probably not worthy building folders by reimage_id
 - in fact maybe we should enable this only for Create projects
Project Member

Comment 8 by bugdroid1@chromium.org, Nov 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672

commit b4f7edad38b723b5b0a1e5fb4aa999fcb9310672
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Nov 02 07:13:01 2017

goofy: Fix duplicated useless logs filled up factory server space.

/var/factory/log has Instalog and Testlog stuff that should have already
been synced by flush_testlog so we should not sync it again using rsync.
In other words, only 'factory.log' is needed.

Meanwhile, we think it's very rare for people to debug using dut_upload
folder so it's probably fine to only keep the latest data (removing
reimage_id).

BUG=chromium:707172
TEST=make test

Change-Id: I8319426e3c79120481d046395b5c8c98aef6daf8
Reviewed-on: https://chromium-review.googlesource.com/749027
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Pi-Hsun Shih <pihsun@chromium.org>
Reviewed-by: Wei-Han Chen <stimim@chromium.org>

[modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/system_log_manager.py
[modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/goofy_plugin_chromeos.json
[modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/system_log_manager_unittest.py
[modify] https://crrev.com/b4f7edad38b723b5b0a1e5fb4aa999fcb9310672/py/goofy/plugins/goofy_plugin_station.json

Project Member

Comment 9 by bugdroid1@chromium.org, Nov 2 2017

Labels: merge-merged-factory-eve-9667.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/factory/+/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3

commit 431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3
Author: Hung-Te Lin <hungte@chromium.org>
Date: Thu Nov 02 09:01:31 2017

goofy: Fix duplicated useless logs filled up factory server space.

/var/factory/log has Instalog and Testlog stuff that should have already
been synced by flush_testlog so we should not sync it again using rsync.
In other words, only 'factory.log' is needed.

Meanwhile, we think it's very rare for people to debug using dut_upload
folder so it's probably fine to only keep the latest data (removing
reimage_id).

BUG=chromium:707172
TEST=make test

Change-Id: I8319426e3c79120481d046395b5c8c98aef6daf8
Reviewed-on: https://chromium-review.googlesource.com/749027
Commit-Ready: Hung-Te Lin <hungte@chromium.org>
Tested-by: Hung-Te Lin <hungte@chromium.org>
Reviewed-by: Pi-Hsun Shih <pihsun@chromium.org>
Reviewed-by: Wei-Han Chen <stimim@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/749613
Reviewed-by: Hung-Te Lin <hungte@chromium.org>
Commit-Queue: Hung-Te Lin <hungte@chromium.org>
Trybot-Ready: Hung-Te Lin <hungte@chromium.org>

[modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/system_log_manager.py
[modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/goofy_plugin_chromeos.json
[modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/system_log_manager_unittest.py
[modify] https://crrev.com/431bd3e9cd2dcdfb80fffabd2de6dd33f6c37cd3/py/goofy/plugins/goofy_plugin_station.json

Sign in to add a comment