Unraid 6.9.2 - On reboot no option to Start Array, no Devices shown, following NO SPACE LEFT ON DEVICE (28)

[TheBaldFox]

Earlier this morning I was moving a movie from a desktop computer to my Unraid Server with a 4TB storage drive and a 4TB parity.  I received an error that the drive was full, which was odd because I had some 90GB of space available on the storage drive. I went into my media share folder, deleted some unwanted movies and tv shows just to make some extra room in case the issue was caused by some unkown Unraid storage data cap on the drive, and attempted to move the movie over again.... Same error.  

 

So, I attempt to log into the Unraid WebUI and am greeted by a screen reading three errors, one of which was something along the lines of "Warning session_write_close(): wrtie failed: No space left on device (28)" in one of the /usr folders.

 

I attempted to log in again, but this time I was greeted with the same error.  Somehow I was able to successfully log in on my different computer. I attempted to download a diagnostic but the page showed "downloading..." for a full 30 minutes and never pulled a diagnostic report for me to view. While waiting for the diagnostic I noticed that the cache drive was not showing up properly on /Main and figured that the issue may caused by the cache drive having gone out and that the system could not write to the drive and had a serious error.

 

I gave up on the diagnostic and rebooted (probably stupid but I didn't know what else to do) but this time the /Main page no longer showed any Array Devices, Boot Device, or any Array Operations other than Reboot and Shutdown.  

 

No shares or Users shown either as the array is obviously not started, and no option to start the array is given.

 

At this point I am at a loss on what to do.  I have never had this type of error in the past.  I have not changed configurations lately and have been running this server for 4 or 5 years with the same basic setup.  Most recent parity check was successful with no errors as well.

 

Any and all info on how to go about troubleshooting would be GREATLY appreciated.

 

Also, I have no reason to think that the storage or parity drive are faulty, and would be willing to rebuild the server from scratch if needed as long as I can be assured that the info on the drives is not affected.

 

Thanks in advance.

 

 

[TheBaldFox]

More info....

 

While attempting to troubleshoot I found that the "Fix Common Problems" plugin will not complete a scan and I get this on the plugins screen:

 

Warning: file_put_contents(): Only -1 of 5084 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 447 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 15742 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 703 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 3172 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 72767 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 2629 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 874 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139

Warning: file_put_contents(): Only -1 of 1565 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix.plugin.manager/include/ShowPlugins.phpon line 139"

 

It does seem that the Boot Device is full, which is wierd because if I recall correctly it had several gigs of space left on it the last time that I checked and I don't know what would have been writing to it.

 

What is the best method to clean the mess off of the root drive while making sure to not delete anything that shouldn't be deleted?

Edited September 29, 2021 by TheBaldFox

[Squid]

Your /tmp appears to be full. (Or you moved files from the hard drives into RAM)

 

 

 

A reboot is going to be required to fix this up

[TheBaldFox]

Hello, thanks for getting back with me.  So, this is so weird.  I woke up this morning to the kids watching a tv show playing from my plex server.  I was dumbfounded as had left the server on over night with the boot drive full and unable to operate properly.  I just checked things out and at some point during the night the server must have cleared the /tmp folder or something and basically fixed itself.  Everything is up and running for the most part as usual, however, I saw that my cache drive was still unassigned.  I have no idea why it became unassigned, and I ran a SMART test on it which returned: 

 

Num Test_Description  Status                            Remaining  LifeTime(hours)  LBA_of_first_error #

1        Short offline       Completed without error  00%          32066               -

 

I added the cache back to the pool and restarted the Array.  It seems to be working properly, but I checked the Fix Common Problems app and it found this error:

Default docker appdata location is not a cache-only share.  I set the appdata default to ONLY cache, and hopefully that will solve the issue, but now it says:

 

"Share appdata set to cache-only, but files / folders exist on the array" and I'm not sure how to go about correcting that issue.

 

Also, my boot device flash is an 8gb Cruzer and it shows that it has 7.48GB free.  

 

I've attached a diagnostic in the hopes that you can glean something useful.  Thanks for the help.

 

 

tower-diagnostics-20210930-1441.zip

[jonp]

I would definitely be concerned.  This makes no sense to me.  If the server was left in a stopped state and no drives were assigned from any of the drop downs, it is literally impossible for the server to have just "fixed itself" unless a system reboot was somehow triggered.  It is possible that your flash drive may be having issues and when you initially rebooted it, it couldn't load the super.dat file properly, thereby not showing you any drive assignments.  If this was just a temporary read-error on the flash, it is possible that a simple reboot could resolve the issue, but that would have had to be triggered by a user, a script, or power failure that upon being resolved caused the system to boot again.  That said, the system would have prompted you that an unclean shutdown occurred, requiring you to manually start the array.

 

As far as the appdata issue, you can stop your docker and VM services (from the settings page) and then trigger the mover to move those files from the array to the cache where they belong.

 

I would also consider a flash replacement:  https://wiki.unraid.net/Manual/Changing_The_Flash_Device

 

All the best,

Jon

[TheBaldFox]

Well, when I left the server out of frustration yesterday, the array was not started, would not start, and it basically just sitting there idling.  I agree that at some point it must have restarted or it was finally able to complete the reboot that I had initiated yesterday after clearing out some room... maybe?  I don't know and have never had a similar issue.

 

I'll look into moving the docker info to the cache drive.

 

What seems to be the issue with the flash drive? Age?

 

[JonathanM]

On 9/29/2021 at 2:33 PM, TheBaldFox said:

 

It does seem that the Boot Device is full, which is wierd because if I recall correctly it had several gigs of space left on it the last time that I checked and I don't know what would have been writing to it.

The boot device (USB stick) is mounted at /boot, which as you say, has plenty of free space.

 

All the messages you quoted about being out of space are referencing RAM, nothing to do with the USB free space. As Squid said, something or someone copied GB's of data into /tmp, which filled up and caused the out of space errors. /tmp is in RAM, so rebooting deletes everything copied there. If you have Plex configured to transcode to RAM without limiting how much space it can use, it's possible that did it.