Repeated the tests on CrOS 55 and now old profile cleanup does not seem to be taking place at all. Only way to solve is to logout and delete user profiles manually.

What is the logic for profile deletion? Will we refuse to delete profiles that are touched in the last 24 hours (might explain above behavior since I wiped device and logged into half a dozen users in the matter of an hour or so).

It definitely seems far to easy for users to run into out of disk space errors and with no easy (and best practice) method for user or admin to resolve.

Logs from the device on CrOS 55 and no profile cleanup being performed:

https://drive.google.com/open?id=0B3BtTQPWWixOUDBGRWw3Snk1NTQ (shared with google.com)

Bernie: Can you help us triage this quickly. If profile cleanup is broken in 55 this could be a disaster in schools where devices are shared.

+1 that we should look into this ASAP please.

+Hiro since I believe in 55 the warnings should be gone.  We should also ensure the issue is just with the warnings and not with any actual low disk space issues.

+fukino and +uekawa

Can you guys look into this? Disk cleanup and user profile deletion should happen when the disk dips below 512MB.

Could repro this issue.But profile deletion is not happening.

Also could see "profile error occurred" when signed in with new user after disk space is almost full.User could no more access or download files in new user profile as disk space is full.

Device:Candy
Version:55.0.2883.87

Disk cleanup and user profile deletion job happens every hour to not put load on the system continuously scanning the file system for files to delete.



/var/log/messages Log contains at least that GCache deleting has triggered and seems to be content at that point

2016-12-13T16:35:57.368532-05:00 WARNING cryptohomed[2740]: Deleting GCache /home/.shadow/XXXX/vault/user/GCache/v1/tmp

I'm looking into this.

I guess following situation is happening:
(1). File manager refuses to download files to make disk space less than 512MB.
(2). However profile deletion never happens if disk space is more than 512MB [1].
(3). Therefore, profile deletion rarely happens.

I believe (2) was existing behavior before 54. I'm not sure when (1) was implemeted.

[1] https://cs.corp.google.com/chromeos_public/src/platform2/cryptohome/homedirs.cc?type=cs&q=platform2+/+cryptohome/homedirs.cc&sq=package:%5Echromeos_public$&l=109

@uekawa - Can you also elaborate how its kicked off "every hour". Is this a cronjob which runs at a specific minute every hour ? What if the device is always off at that particular minute. For example if it runs at 5 minutes after every hour, and classrooms always use the device between 10 minutes every hour and 50 minutes after the hour.
 
- Would file manager be able to trigger this process when it detects a low diskspace condition ? I understand the reason why its not running every minute... but it could just be tied to other events triggered when this condition is met which may be less frequent as well.

Correction for #14. I could DOWNLOAD a file for the disk space to be < 512MB.
However I could not COPY a file for the disk space to be < 512MB.

I confirmed profile deletion works if remaining disk space is < 512MB.
Chrome: 56.0.2913.0
Platform 8743.69.0 chell.

(1). Login as user A and fill diskspace to near 512MB with whatever way including copy of files. You can check Available on Storage management in Settings for remaining disk space.
(2). Then download a file from Chrome into Download and make the remaining disk space less than 512MB.
(3). Logout and login as user B, and download a small file X.
(4). Restart Chromebook.
(5). Confirmed user B's file X is deleted.
(5'). Saw /var/log/messages and find log "Freeing disk space by deleting user ..."

@oka - is reboot needed for the cleanup ? I'm still not clear what the trigger points are.

In schools its possible devices are shared between uses and no one explicitly reboots. I'm curious what happens in those scenarios.

@royans,

When the system starts, disk cleanup is triggered.
After that, periodic disk cleanup will be triggered in every hour.
So, reboot is needed to run the cleanup immediately.

Please note that if the system detects that the remaining space is less than 512MB in the cleanup process, it removes caches and profiles until it makes 1GB space.
This makes some margin for the storage use between periodic updates.

Downloading files should succeed even when the remaining space is less than 512MB.
I'm not sure about the minimum space to make the downloading successful, though.

I'm going to confirm that an older version (e.g. M49) have the same behavior.

I confirmed that M53 has the same behavior.
Signing in/out does not trigger disk cleanup. Rebooting triggers it.

Regarding the notification, from M56, it will be suppressed if the device is enterprise managed and there are more than one users.

 Issue 658788  has been merged into this issue.

Note this is actually impacting many customers. After merging the other FR here, there's 53 stars on this.

( I was posting in  Issue 658788  before merge ) -- Just wanted to note for anyone not reading that, and after reading the comments here, that I still think more aggressive cleaning should be implemented.  Even after the bug is fixed.

512MB is nothing this day and age and with users using chromebooks for audio and video and editing (wevideo and screencastify, for examples), single file sizes easily cross that boundary and the lack of available disk space prevents saving those files.

+1 to comment #24. This becomes more crucial with capability from the Play Store. The flip side of course is that we end up being more aggressive around clearing out actual user data. But that's not a problem with the clean-up logic, just a fact of physics that only X users can share a device with Y GB of space and this should be managed by the admin and enterprise and not by the software.

I'm frequently observing that the Esc + Refresh + Power method (with forced re-enrollment enabled so there is no switch to dev mode) is actually not freeing up much space. All profiles are removed but out of ~10GB total usable space its showing 8GB still in use.

In these cases if I perform a full recovery via USB or disable forced re-enrollment and get the device into dev mode, a more thorough wipe is performed and most of the space is freed up as it should be. This is a time consuming, temporary solution.

Could something other than user data be filling up the device storage? Something that isn't cleaned by the disk cleanup routine? With only a small amount of storage being cleared, the same warning comes back within a few weeks. Unfortunately I cannot troubleshoot further because once we put a device in developer mode where we might be able to look around the back end, the problem has already been fixed by the wipe that occurs in the process of switching to dev mode.

Now that its clear this is not a new issue in 55, I take back my RBS label. That being said, I think there is clear evidence that this is not a solved issue and we need more work to address this overall issue.

#26, if you observe problematic behavior, please send alt-shift-I feedback with a comment about this bug so we could see some debug logs.

#23 Regarding merge.  The existence of a bug that affects the same area as a feature request does not negate the feature request.  Many Admins would like the ability to set a policy to determine how long a profile remains on a device.  Fixing the bug would simply ensure that the new policy worked as intended.  Of the 53 individuals that had starred it, 51 came from the Feature Request.

This behavior is now affecting organizations that use Chromebooks for Testing since a cache of answers is typically stored on the Local Storage.
https://plus.google.com/u/0/104693558305645073517/posts/KAXk6CA76B3?sfc=true

Still having this issue on chrome v56.  Very frustrating.  3 years into shared chromebooks at our school district and every few months we have to go through the cycle of getting storage complaints, turning on the "erase user data" option for a week, lose all tracking for that week, and then turn it back on.

Hi Naoki - can you look into this please?

Ping. We need action on this. Customers are regularly reporting these disk full issues and enabling ephemeral mode to solve which then breaks Play Store among other things.

I have not heard back from my request on #28.

#30, 
"https://plus.google.com/u/0/104693558305645073517/posts/KAXk6CA76B3?sfc=true"
where is that message from, do we have a screenshot ?

@uekawa #34, #28
We trained our techs to send feedback as requested every time they run into this. We've surely sent dozens so far. I've sent at least 6 on my own referencing  issue 673859 . Is there another way we can send feedback so it is not missed?

Hi Junichi, Naoki said the changes to remove the warnings should have taken effect in 56 but users are still seeing it in 56.  Can you please verify that the code is in effect?  It's very important that end users do not see this warning as it's causing confusion and chaos in schools.  This is a P1 issue so please look at it as soon as possible.  Thank you.

I just want to confirm that the solution to this is not going to be simply to hide a message?  Based on comment #38 that seems to be the case.  The current implementation of "auto-cleanup" is not aggressive enough.

Please keep in mind there are other programs besides the ChromeOS that rely on having sufficient disk space.  Screencastify is a widely used app and often has space issues with the current implementation.

The solution needs to be either more aggressive auto-cleanup or, ideally, options added to the admin console where enterprise admins can specify the type of cleanup.  For example, in most shared device scenarios there would never be a need to keep more than 1 or 2 profiles saved on the device, ever.

Options should be added where admins can specify the scenario for the device...

Option A: Normal Cleanup (however you have it now) ... to be used in non-shared scenarios
Option B: Auto-Delete profiles older than X days
Option C: Keep only X number of profiles

Very helpful.  And it makes sense to be more aggressive for certain use cases.  So, one question - instead of removing profiles after a certain number are created or after a certain period of time, what if the OS automatically was more aggressive when the user or an app needed more space and purged profiles until there was enough or until there was nothing left to remove?  This might be the cleanest solution but is actually very tricky for our engineers to do since removing the partition for user B may not be able to be done while user A is logged in but our engineers are smart and they may be able to find a way. :)  In any event, if we could manage to do that, would that eliminate the need to create more admin options as you mentioned with B, C, etc.?

This sounds good but doesn't address the issue where even after a powerwash and zero profiles left on the device, it is not properly cleaned, still using most of the storage. It seems profiles are not the only thing that need to be cleaned. Something else is stuck using up storage in a lot, but not all cases where this error has occurred on our devices. (I mentioned this back in comment #26)

What would be really fabulous is if storage were automatically directed to the GDrive of the account used to enroll the device this bypassing the limits of the device's limited ssd

RE: cyrusm@chromium.org, comment #40 -- "what if the OS automatically was more aggressive when the user or an app needed more space and purged profiles"

What happens in the scenario where a program simply asks for the current available space and doesn't bother trying to save because the amount of space is too small?  e.g. I think screencastify won't even let you start recording if drive space is < 1 GB.

Re comment #31:
Could you clarify the issue? Do you see the warning message, which should be hidden from M56?

Re comment #39:
I'm not sure if these options fix the issue.
IIUC, the issue (file operation failure on low storage) occurs when 1) Other profiles have large files and 2) Current profiles fill the storage rapidly.
These options can be mitigations, but it still can happen, and aggressive removal has trade-offs.

One relatively-safe mitigation is run auto-cleanup more frequently (e.g. 1hours to 10 minutes?)
If we add an admin setting, maybe "Option C: Keep only X number of profiles" with X=1 can be an option.
If there is only one profile, user have control for all storage. It behaves similar to ephemeral mode, but is not on tmpfs.

Option C or an additional option to ephemeral mode where only 1 profile is saved would solve it.

Ideally we would not be adding admin console options to do something that should be automatic (and has worked pretty well that way in the past). I'd rather see something like "require N amount of free space at user login" instead of "keep only N profiles". Most user profiles who aren't downloading large files or using many Play Store apps won't take up room so unconditionally deleting them degrades the user experience (slower login, lost local settings, etc). However, suppose that single remaining user profile other than current user is taking up 10gb of disk space, then we still have performance issues. Deleting profiles until N free space is avaiable should balance the behavior and ensure users get a consistent free space experience.

I do agree with #41 that we might need to look at more than user profile to free up space. For instance, p2p update caching may be caching 1gb non-delta updates while on a school network that blocks all p2p connections. Admins should be able to disable p2p to prevent the update caching and free up that space.

Short term, I think #44 suggestion of reducing cleanup check time from 1 hour to 10 minutes may reduce these issues. Longer term, I think we need to do a check at login and if free disk space is below a certain threshold (500mb for non-Play store devices, 2gb for Play store devices?) then delete old profiles until that much is free.

RE: #46 Why have a certain threshold (500mb, 2gb) that is arbitrarily
chosen by Developers.  Each Domain will have different needs for the amount
of storage available on the device when users login. Does it make sense to
defined as a percentage of hard drive space or a value (mb) configurable
via policy in the cpanel?

For short term, absolutely changing the polling interval would be helpful.

Re comment #42:
No disrespect intended to the commenter, but this would be a terrible idea for most school setups where the account used to enroll the device is unrelated to any of the normal users.
Agree 100% with Comment #39.

One other thing that I believe was brought up very early in this discussion that would be very helpful would be to somehow remove the connection between the log of who has logged in and the stored profiles.  The log could just be updated when a user logs in and we could keep the device from holding onto any profile info altogether in most cases.

The reason to have an option is because you're trying to come up with a value that is best suited for all devices.  However, shared devices vs assigned/personal devices have very different expectations.  That's why it should come down to an admin option.  This way, when you know it's a shared device, you can be much more aggressive have less problems.

Regarding, "500mb for non-Play store devices, 2gb for Play store devices" ... it should be 2gb for all devices.  I keep saying it and I keep getting ignored, I feel like, but there are apps that will refuse to function is space is too small.  Screencastify will not record if there is less than 1gb of storage space available.

No disrespect taken. My thought was if we had an account used only for enrolling, that ONLY the superadmin or delegates had access to, the account could "act" as kind of an external hard drive and that possibly certain aspects could be "stored off device".

I've spoken to Omri and I'd like him to take ownership of all of this going forward. (Assigned to him.)

To be clear I'm not a fan at all of arbitrary things like polling every X hours to do cleanup.  Whatever number we pick will either be too large in some cases or too small in some cases.  We should not need polling.  We should cleanup as disk space is needed since cleanup is not a major task and is fairly instantaneous I believe?  And we certainly shouldn't introduce even more arbitrary numbers like remove after X days or remove Y profiles - we need to ensure that users always have as much space as they need to do their work - unless they are the only profile left and there's nothing more to remove.  Users already have the option to store their downloads to Google Drive by default if they so desire.  In the end, however, this is up to Omri.

Omri - some things to catch you up on:
(1) We first need a bit of bug hygiene here.  There are a lot of various feature requests and bugs that are being brought up in this one bug - to help out engineering, we need to split all this up. (e.g. #31 says they're still seeing the warning so looks like warning needs to be checked; #41 says powerwash isn't cleaning up all the storage - this would be a bad bug if it is the case; #43 is suggesting that info reported back to apps about available space is reporting back the wrong amount - it should report back the amount assuming it kills off all other partitions; etc.)

(2) We need to check if, while logged in as user C, if user C needs more space and disk is full that user A is removed and if even more space is needed user B is removed - regardless no one should get low space warnings unless there is only 1 profile left.

(3) I've also asked Krishna (cc'd on this bug) to run some tests on the above scenarios.

Thank you everyone.  We now have an assigned Product Manager and we will try to sort all this out.

Hi all, thank you so much for all the details and feedback. This is definitely a serious issue and we want to fix it ASAP.

I have read through the thread and chatted with Fukino offline. This issue indeed has many different aspects which we should track separately. In order to move forward and solve all these problems, I have split the issue into a few bugs to track work & next steps. 

Hopefully it should address most issues, please file additional bugs and mark as blocking this one if there's anything I'm missing.

I have split this to the following next steps (each has a tracking bug):
1. Change default minimum disk space to 1GB to alleviate the problem for most use-cases
→ Tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=533750 This is now implemented and released in M56

2. Create policy for controlling minimum disk space required (before performing a disk cleanup)
→ Tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=701289

3. Combine disk cleanup routine and Low disk space warning checks to save resources
→ Tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=701290

4. Fix the bug where powerwashing does not clean all data (comment #26)
→ Tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=701292

5. Change cleanup routine to happen when disk space goes below threshold (1GB by default, can be modified by policy).
→ Tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=701293

6. [lower priority] Add a policy to disable P2P caching for schools that block P2P ports to save some more space.
→ Tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=701294


Would love to get your feedback on whether the proposal of these next steps would solve the problem for you so we can move forward with implementing these changes ASAP.


P.s. For comment #26 debuggin, I collected 26 feedback reports (wow!) See full list: https://docs.google.com/spreadsheets/d/1L1WmY8gZ3HOhpzw-fychMsgXW4wyLdwDLVzKijrXOMo/edit#gid=0

Friendly ping. I see limited action on the blocking bugs, are we making any progress here?

No more M-59 releases ... I'm targeting to M-62 for now and adding Albert to find an owner for this

I we have a fix in time for M-61 then we can merge request for it 

Over to Weifang who I believe was going to look into disk space issues? Weifang feel free to assign back to me if I'm wrong.

Issue 217829 has been merged into this issue.

weifangsun@
Is there any update that can be shared with the customer?

Weifang?  Update?

weifangsun@
sorry for asking again.
is there any update?

CC: yamaguchi@
IIRC, he worked on updating the cache removal criteria.
Do you know anything related to the issue?

Here are the changes I did 1 year ago. I did not change the essential logic of cache/user removal, but did some bugfix tweak in the cryptohome based on the designs at that time.
https://bugs.chromium.org/p/chromium/issues/detail?id=533750

- making the cryptohome trigger cleanup at <1GB, for Gcache
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/382894
- keeping Android cache until <512MB
https://chromium-review.googlesource.com/c/chromiumos/platform2/+/394608

<files-triage>

https://bugs.chromium.org/p/chromium/issues/detail?id=533750 was one of the core issues.

In particular, this fix: https://chromium-review.googlesource.com/c/chromiumos/platform2/+/382894

Archiving it in favor of the Blocked-on bugs and CrOSFilesFeature-LowStorage labeled bugs. Feel free to contact me if any concerns/suggestions.

 Issue 856411  has been merged into this issue.

I called Google Admin support on a very similar issue to this.  Our School district has 4566 Chromebooks spread among 50 buildings.  With multiple user-profiles the Chromebooks are running out of disk space, wiping them one at a time is too tedious.  The policy to "Erase all local user info at sign-out" breaks our ability to audit the chromebook for bullying/threats etc in the schools.

Possible solution: ** Age based cleanup ** :- if a local account hasn't been used on the chromebook for X number of days/months, then auto-clean it up.

Our Linux/Samba file servers have an age-based auto-cleanup that has been working very well for 10 years. It would be great if Google Admin Console could implement a similar feature for Chromebook local storage.


https://bugs.chromium.org/p/chromium/issues/detail?id=856411 

There are improvements to disk cleanup process in current beta channel 68 release. Can you test against 68 beta and see if that improves your experience?