No space left on device, can't login


Recommended Posts

So I've just rebooted my unRaid Server, and I've been greeted with the following

 

Warning: session_write_close(): write failed: No space left on device (28) in /usr/local/emhttp/login.php on line 33

Warning: session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php) in /usr/local/emhttp/login.php on line 33

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/emhttp/login.php:33) in /usr/local/emhttp/login.php on line 35

 

I've tried to run diagnostics, but that doesn't work properly it seems, as there's no space on device. I get the following :

 

Starting diagnostics collection...
Warning: file_put_contents(): Only 0 of 5 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 80

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 12 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 173 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 9066 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 180 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 1032 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 7450 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 7606 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 4480 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 4667 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 4240 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

Warning: file_put_contents(): Only 0 of 165 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 101

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 120

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 122

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 120

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 122
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device
echo: write error: No space left on device

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 31 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

Warning: file_put_contents(): Only 0 of 775 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 210
done.
ZIP file '/boot/logs/r2-d2-diagnostics-20200301-0944.zip' created.

 

running " du -xh --max-depth=1 /" gives me the following:
 

0       /opt
148K    /tmp
16M     /etc
1.3G    /usr
22M     /lib64
0       /mnt
20M     /sbin
0       /home
6.6M    /lib
13M     /bin
5.2M    /var
6.3G    /root
7.6G    /

 

The diagnostics file produced is just 18K, and every file in there seems empty. I can't access SMB to grab it on my Windows machine, but unzipping via SSH and looking at the contents, everything looks like 0 byte files.

 

Any help appreciated

Edited by teh0wner
Link to comment
root@R2-D2:/# ls -ltrah /root
total 12K
-rwxr-xr-x  1 root root  60 Jan 26 17:28 .bashrc*
-rw-r--r--  1 root root   0 Feb  2 21:15 .wget-hsts
lrwxrwxrwx  1 root root  30 Feb  2 21:15 .docker -> /boot/config/plugins/dockerMan/
-rwxr-xr-x  1 root root 316 Feb  2 21:15 .bash_profile*
lrwxrwxrwx  1 root root  21 Mar  1 09:05 mdcmd -> /usr/local/sbin/mdcmd*
-rw-rw-rw-  1 root root  32 Mar  1 09:05 .tmux.conf
drwx------  3 root root  60 Mar  1 09:11 .cache/
drwxr-xr-x  4 root root 200 Mar  1 09:11 ./
drwxr-xr-x  6 root root 120 Mar  1 09:33 .config/
drwxr-xr-x 20 root root 420 Mar  1 09:44 ../

Nothing pops out.

Link to comment
6 minutes ago, doron said:

There's 6.3G showing up there on your du above.

How about du -xh /root ?

 

root@R2-D2:/# du -xh /root
0       /root/.cache/rclone/vfs
0       /root/.cache/rclone
0       /root/.cache
0       /root/.config/procps
4.0K    /root/.config/sakura
4.0K    /root/.config/remmina
4.0K    /root/.config/htop
12K     /root/.config
24K     /root

Go figure..

Link to comment

Something very weird is going on...
 

root@R2-D2:/#  du -xh --max-depth=1 /
0       /opt
160K    /tmp
16M     /etc
1.3G    /usr
22M     /lib64
0       /mnt
20M     /sbin
0       /home
6.6M    /lib
13M     /bin
5.3M    /var
24K     /root
1.4G    /

and I can login just fine now.. How did root all of a sudden empty out?

 

Link to comment
  • 4 weeks later...

I'm getting the same error and my drive doesn't look anything close to being full. I'm running 6.8.3 nvidia and this ONLY happened once I upgrades to 6.8.3.

 

this is on a 16GB cruzer fit

image.png.f823b0bb5e450a65c3872f4e9bfd54e4.png

 

What got it going again was to reboot via SSH and notice that a few dirs are now smaller.

image.png.02e74d11872baae1ec6dc0c3b1851276.png

 

crazy that just 6MB kept the web gui from loading while the drive has gigs of space available...

Link to comment
Quote

[email protected]'s password:

Linux 4.19.107-Unraid.

root@Tower:~#   du -xh --max-depth=1 /

0 /opt

6.4M /tmp

12M /etc

543M /usr

22M /lib64

0 /mnt

20M /sbin

0 /home

6.6M /lib

11M /bin

4.2M /var

8.0K /root

623M /

root@Tower:~#  du -xh /root

8.0K /root

root@Tower:~# diagnostics

Starting diagnostics collection...

Warning: file_put_contents(): Only 0 of 5 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 80

 

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 12 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 16 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 13 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 312 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 942 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 11 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 15 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 14 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 11 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 85

 

Warning: file_put_contents(): Only 0 of 75 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 101

 

Warning: file_put_contents(): Only 0 of 2 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 120

 

Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 122

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

echo: write error: No space left on device

 

Warning: file_put_contents(): Only 0 of 27 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

Warning: file_put_contents(): Only 0 of 25 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

Warning: file_put_contents(): Only 0 of 29 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 160

 

Warning: file_put_contents(): Only 0 of 180 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 210

done.

ZIP file '/boot/logs/tower-diagnostics-20200323-1542.zip' created.

root@Tower:~#

I managed to SSH after a while, at first that didn't work either.

Should I just reboot it via SSH or wait to gather more data? As I said above, I had a preclear running with the Preclear plugin. 

Link to comment

Nope, now I got the exact same error again. The server freezes and I can only SSH to it. The web GUI can't be reached. This happened around 10 minutes after I started to preclear the disk. The disk is in a IcyBox cabinet connected via eSATA to the extra SATA port on the mobo. Will restart and try doing the preclear using USB 2.0 instead. Any ideas?

Link to comment
  • 4 weeks later...

This happend for me, too. How I produced this error:

Nearly fresh Unraid server with 2 parity disk, 6 data disks and 1 nvme ssd cache. While using wget through the user scripts plugin to get the files from my old NAS I disabled the ssd cache as it would produce unnecessary TBW on the SSD. But as the write speed was really low I dediced on the next day to remove the parity drives as well. So I tried to stop the array. After several minutes it was still not stopped (it was trying to unmount the user shares):

1577111502_2020-04-2112_31_41.png.7d74a68eb5b4f92b5351dbab05a5cced.png

 

I thought this was because of the still running wget script. So I removed the ethernet cable, waited several minutes and plugged it in again. Now the traffic monitor did not show anything so I successfully stopped the array.

 

Now I unselected the Parity Disks:

1619484044_2020-04-2112_33_22.png.643b5cd9358b4b57a6d83e7fca34e70d.png

 

and started the Array again. At first it looks promissing:

1748757948_2020-04-2112_35_16.png.7a8f252fdd5a3f75ed11d3efca778469.png

 

But on the next click the top left corner missed the server name and description and the disk overview as completely empty so the web clients JS API seemed to be dead now:

2020-04-21_11-56-07_02-00_1294_1407_984731_1.thumb.jpg.cdf2be051a3d42ed22b97c20b57e8fe9.jpg

 

Finally I logged out and logged in again and the known error message appeared:

Warning: session_write_close(): write failed: No space left on device (28) in /usr/local/emhttp/login.php on line 33

Warning: session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php) in /usr/local/emhttp/login.php on line 33

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/emhttp/login.php:33) in /usr/local/emhttp/login.php on line 35

 

 

"usr/local/" is on the usb flash drive, correct? Does it probably mean that the drive is defective or is there a connection regarding the removed parity drives / bug in unraid? What should be my next step?

 

2020-04-21_11-56-07_02-00_1294_1407_984731_1.jpg

Edited by mgutt
Link to comment
30 minutes ago, mgutt said:

This happend for me, too. How I produced this error:

Nearly fresh Unraid server with 2 parity disk, 6 data disks and 1 nvme ssd cache. While using wget through the user scripts plugin to get the files from my old NAS I disabled the ssd cache as it would produce unnecessary TBW on the SSD. But as the write speed was really low I dediced on the next day to remove the parity drives as well. So I tried to stop the array. After several minutes it was still not stopped (it was trying to unmount the user shares):

 

 

I thought this was because of the still running wget script. So I removed the ethernet cable, waited several minutes and plugged it in again. Now the traffic monitor did not show anything so I successfully stopped the array.

 

Now I unselected the Parity Disks:

 

 

and started the Array again. At first it looks promissing:

 

 

But on the next click the top left corner missed the server name and description and the disk overview as completely empty so the web clients JS API seemed to be dead now:

 

 

Finally I logged out and logged in again and the known error message appeared:


Warning: session_write_close(): write failed: No space left on device (28) in /usr/local/emhttp/login.php on line 33

Warning: session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php) in /usr/local/emhttp/login.php on line 33

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/emhttp/login.php:33) in /usr/local/emhttp/login.php on line 35

 

 

"usr/local/" is on the usb flash drive, correct? Does it probably mean that the drive is defective or is there a connection regarding the removed parity drives / bug in unraid? What should be my next step?

 

 

Look to be issue with your USB stick but not because /usr/local is on the stick.

Do a diskchk of your USB stick on a Windows machine to make sure the file system is ok.

Then boot your server using a different USB port, preferably USB 2.0 port.

 

 

 

 

 

 

 

 

Link to comment
3 hours ago, testdasi said:

Look to be issue with your USB stick but not because /usr/local is on the stick.

Do a diskchk of your USB stick on a Windows machine to make sure the file system is ok.

Then boot your server using a different USB port, preferably USB 2.0 port.

 

 

 

 

 

 

 

 

Ok I'll do both (new stick and use an USB 2.0 port this time). The stick was really old, so this was probably the reason.

 

P.S. after restarting the server it seem to work again, but I was not able to start the mover. It did nothing. So finally I did a fresh re-install. I hope this does not happen again.

Link to comment
1 minute ago, mgutt said:

Ok I'll do both (new stick and use an USB 2.0 port this time). The stick was really old, so this was probably the reason.

 

P.S. after restarting the server it seem to work again, but I was not able to start the mover. It did nothing. So finally I did a fresh re-install. I hope this does not happen again.

If the stick is corrupted or drops offline while the server is running (or during boot up), it always leads to strange issues so keep that in mind.

  • Like 1
Link to comment
  • 7 months later...

I just had this happen while rebuilding parity. I decided to preclear a disk as well and shortly after the ui gave (unraid Warning: session_write_close(): write failed: No space left on device). 

"du -shx /* | sort -rh" showed enogh free space on the 16gb memory stick, but df -h showed rootfs full.

I was stumped. the array was still building but the preclear disk was not showing led activity.

Found some posts about rootfs full but not and found that if a file is deleted but in use du dosnt count it.

"lsof +L1" shows such files. in my case it was

COMMAND   PID USER   FD   TYPE DEVICE      SIZE/OFF NLINK   NODE NAME
dd      29330 root    0r   BLK   8,48 0x2baa1280000     0 639212 /dev/sdd (deleted)
dd      29330 root    2w   REG    0,2   16114245632     0 663754 /tmp/.preclear/sdd/dd_output (deleted)

after killing dd using kill -9 29330 everything retured to normal but the preclear. hope this is related?

Link to comment
4 minutes ago, SimeonG said:

enogh free space on the 16gb memory stick, but df -h showed rootfs full

rootfs is in RAM not on the flash drive. The Unraid OS runs completely in RAM. Flash only has the archives of the OS and these are unpacked fresh into RAM at each boot.

 

The flash drive itself is mounted at /boot.

 

5 minutes ago, SimeonG said:

hope this is related?

This thread is pretty old. Are you asking for help?

Link to comment

No, I got it figured out, dmesg was spitting Buffer I/O error on dev sdd, had no gui so couldnt get info on what drive it was and didnt want to restart. killing the dd process got back the ui and all drive where present. Im guessing a problem whith preclear but will start a thread on it if i cant figure it out.
just wanted to post the "lsof +L1" as it found my problem. This thread helped me get started troubleshooting and seamed related to this phantom file system full. this was the first thing that came up searching for unraid Warning: session_write_close(): write failed: No space left on device). on google.

The ram drive triped me up a bit as what looking around the stick and had plenty of space, I even tried deleting some logs to clear space but it would just instantly fill agen even thogh du showed low space useage by files. Some post even claimed that mounting over a non empty directory would cause this but couldn't find anything like that. Didint notice it was an old thread sorry about that

Link to comment

Hi Guys 

 

I am getting the same isue , I can't login 

 

Quote

Warning: session_write_close(): write failed: No space left on device (28) in /usr/local/emhttp/login.php on line 33

Warning: session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php) in /usr/local/emhttp/login.php on line 33

Warning: Cannot modify header information - headers already sent by (output started at /usr/local/emhttp/login.php:33) in /usr/local/emhttp/login.php on line 35

 

To login again i have to reboot the server 

 

Can anyone help me ?

 

 

Link to comment
  • 10 months later...
On 11/29/2020 at 7:02 AM, chefmoisas said:

Hi Guys 

 

I am getting the same isue , I can't login 

 

 

To login again i have to reboot the server 

 

Can anyone help me ?

 

 

 

I transcode to my ram "/tmp/" in emby and sometimes it's not purged and fills up the ram.

my solution was to shutdown the server safely via ssh "shutdown -t 5 now" and start it back up.
then install the userscripts plugin and create a new script set to run every hour.

set the values for "transcode_dump_path" and "purge_max_minutes" 

if you run the script you should see a directory listing of your specified path.
 

#!/bin/bash
# set folder path
transcode_dump_path="/tmp/emby/transcoding-temp/"
 
# set min age of files and folders to delete
purge_max_minutes=240

ls -ltr $transcode_dump_path
find $transcode_dump_path* -mmin +$purge_max_minutes -delete > /dev/null

 

Good Luck.

Edited by mpalpha
  • Thanks 1
Link to comment
  • 1 month later...
46 minutes ago, kiowa2005 said:

everything seems to be functioning fine now.  No reboot required.

until next time.

 

Here is a script I run at boot with the User Scripts plugin. It allocates a fixed-size portion of /tmp to use for transcoding.

#!/bin/bash
mkdir /tmp/PlexRamScratch
chmod -R 777 /tmp/PlexRamScratch
mount -t tmpfs -o size=4g tmpfs /tmp/PlexRamScratch

And then map /transcode to /tmp/PlexRamScratch

 

I would give credit where it is due for this but don't remember where. Thanks

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.