No space left on device error


Recommended Posts

I've tried to install a new docker, and also to update sickrage within an existing docker and get this error for both.

 

My docker.img file is 30GB and when I ran docker images, I got:

 

REPOSITORY            TAG                IMAGE ID            CREATED            VIRTUAL SIZE

binhex/arch-sickrage  latest              816cb9f461d1        5 days ago          713 MB

gfjardim/crashplan    latest              4abc20213697        3 weeks ago        637.3 MB

gfjardim/owncloud      latest              e432c58fda24        7 weeks ago        510.6 MB

needo/deluge          latest              60e9118749e9        7 weeks ago        367 MB

needo/plex            latest              8906416ebf13        3 months ago        603.9 MB

<none>                <none>              c9813f09f163        4 months ago        215 MB

 

This doesn't seem to be 30GB.

 

I check /mnt/cache because that is where my docker.img resides and it has 120GB left.

 

I had this happen before, so I removed the docker.img and rebuilt it, reinstalling all my dockers.  However it has happened again.

 

How do I fix this?

 

thanks

david

Link to comment

just double checking your docker.img is set to 30G and not 3G

 

If you add up your VIRTUAL SIZE of your docker images it is : 3046.8 MB which is 2.9754 GB  If your docker image is 3GB instead of 30 GB i can see this no space left on device error happening

 

Goto Settings -> Docker and in the Docker Volume Info box,  what does the devid line read?

 

it should look something like this

 

devid    1 size 20.00GiB used 9.04GiB path /dev/loop0

Link to comment

Does this help?

 

Filesystem      Size  Used Avail Use% Mounted on

/dev/loop0      30G  30G  4.0K 100% /var/lib/docker

 

 

looking in /var/lib/docker/btrfs/subvolumes/* I see many (100's) of directories with long numbers as the directory name and each is 100s of MB large.

 

/var/lib/docker/btrfs/subvolumes/02b99f0f04903f89daf2ea9027919dea120354e2e6e576286b43701ebb4254ed

 

bin/  config/  dev/  home/  lib64/  mnt/  proc/  run/  srv/  tmp/  var/

boot/  data/    etc/  lib/  media/  opt/  root/  sbin/  sys/  usr/

 

I tried to stop docker, but it hung, and I can't shutdown tower because it is just sitting there, for 10 minutes, trying to unmount the disks.

 

david

Link to comment

You'll need to delete yourdocker.img and recreate from your templates.

 

Once your docker.img is full you have no means of telling which files may have been corrupted via truncation.

 

I suspect you do not have your directory mappings set correctly between host and docker and that some docker gobbled up all the space it could. Likely candidate is owncloud and your downloading programs.

 

The other possibility is you went through extensive docker install uninstall reinstall or docker program updates that filled all ypur space.

Link to comment

All my mappings are to

 

/mnt/cache/apps/*

 

Things have been running without issue for months. Not sure what happened to break things.  I had to rebuild all my containers two days ago because it was out of space, and now again.  I've not added/changed containers.

 

I'll recreate and watch the docker.img size as I do so.  I only update one container (sickrage).

 

I did try to download the Google Music Manager docker manuall, maybe that is what screwed it up.  During that pull I got the out of space error.

 

david

Link to comment

So after installing my dockers I ran the scrubber:

 

WARNING: errors detected during scrubbing, corrected.

scrub device /dev/loop0 (id 1) done

scrub started at Sat Aug 8 10:36:51 2015 and finished after 338 seconds

data_extents_scrubbed: 75459

tree_extents_scrubbed: 140933

data_bytes_scrubbed: 2867515392

tree_bytes_scrubbed: 2309046272

read_errors: 0

csum_errors: 0

verify_errors: 0

no_csum: 288

csum_discards: 8971

super_errors: 0

malloc_errors: 0

uncorrectable_errors: 0

unverified_errors: 495

corrected_errors: 0

last_physical: 6211764224

 

 

are errors expected?

 

david

Link to comment

You likely have something misconfigured or one of the dockers isnt staged properly. See if any of them have MFT of entries in the docker logs. I once had an issue where a docker wasnt able to start a program properly so it kept trying to restart it, which in turn ended up chewing through the log file like Gabe at a Buffet.

 

For reference, my 3 dockers of Eggdrop, Nzbget, and Transmission are using all of 655 Megs after 1.5 months, going by: df -h /var/lib/docker

Link to comment

My docker image is full but I can't figure out why:

/dev/loop0       50G   49G  899M  99% /var/lib/docker

 

But when I look at what is taking up the most space in /var/lib/docker I get this which doesn't add up:

 

13G     containers
2.3G    vfs
77M     tmp
23M     init
15M     unraid
6.7M    graph
1.5M    volumes
32K     linkgraph.db
8.0K    repositories-btrfs
4.0K    unraid-update-status.json

 

So I'm not really sure what my issue is. Seems like there used to be a bug where the image filled up and space wasn't reclaimed but I'm having trouble searching for it now.

Link to comment

Does this sound like the same issue? Not trying to hijack threads, but I came here with a very similar problem with a brand new Plex install from yesterday. Thanks!

 

ug 9 06:57:09 Tower shfs/user0: shfs_mkdir: assign_disk: Applications/.plexmediaserver (28) No space left on device

Aug 9 06:57:09 Tower logger: rsync: recv_generator: mkdir "/mnt/user0/Applications/.plexmediaserver" failed: No space left on device (28)

Aug 9 06:57:09 Tower logger: *** Skipping any contents from this failed directory ***

Aug 9 06:57:09 Tower logger: cd+++++++++ Applications/.plexmediaserver/

Aug 9 06:57:09 Tower logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]

Aug 9 06:57:09 Tower logger: ./Applications/.plexmediaserver/config/Plex Media Server/Media/localhost/4/076fe297b0b7106fc4b18da7f1b0325a97626df.bundle/Contents/Subtitle Contributions/com.plexapp.system

Aug 9 06:57:09 Tower shfs/user0: shfs_mkdir: assign_disk: Applications/.plexmediaserver (28) No space left on device

Aug 9 06:57:09 Tower logger: rsync: recv_generator: mkdir "/mnt/user0/Applications/.plexmediaserver" failed: No space left on device (28)

Aug 9 06:57:09 Tower logger: *** Skipping any contents from this failed directory ***

Aug 9 06:57:09 Tower logger: cd+++++++++ Applications/.plexmediaserver/

Aug 9 06:57:09 Tower logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]

Capture.PNG.441182826fdd11e4d9cd6c96ebb2f749.PNG

Link to comment

Does this sound like the same issue? Not trying to hijack threads, but I came here with a very similar problem with a brand new Plex install from yesterday. Thanks!

 

ug 9 06:57:09 Tower shfs/user0: shfs_mkdir: assign_disk: Applications/.plexmediaserver (28) No space left on device

Aug 9 06:57:09 Tower logger: rsync: recv_generator: mkdir "/mnt/user0/Applications/.plexmediaserver" failed: No space left on device (28)

Aug 9 06:57:09 Tower logger: *** Skipping any contents from this failed directory ***

Aug 9 06:57:09 Tower logger: cd+++++++++ Applications/.plexmediaserver/

Aug 9 06:57:09 Tower logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]

Aug 9 06:57:09 Tower logger: ./Applications/.plexmediaserver/config/Plex Media Server/Media/localhost/4/076fe297b0b7106fc4b18da7f1b0325a97626df.bundle/Contents/Subtitle Contributions/com.plexapp.system

Aug 9 06:57:09 Tower shfs/user0: shfs_mkdir: assign_disk: Applications/.plexmediaserver (28) No space left on device

Aug 9 06:57:09 Tower logger: rsync: recv_generator: mkdir "/mnt/user0/Applications/.plexmediaserver" failed: No space left on device (28)

Aug 9 06:57:09 Tower logger: *** Skipping any contents from this failed directory ***

Aug 9 06:57:09 Tower logger: cd+++++++++ Applications/.plexmediaserver/

Aug 9 06:57:09 Tower logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0]

There is nothing in your Host paths that would result in this docker trying to access /mnt/user0/Applications/.plexmediaserver. Are you sure you don't have something else, maybe a Plex plugin still installed?
Link to comment

You need to use /data not /mnt for the docker side unless you edited the config as well.

 

My config uses /mnt because I was using Crashplan before the docker was created. So when I moved to the docker, I just kept /mnt

 

I did notice that the first volume was empty.  Not sure how that happened, but I fixed that and am now running again.  Not sure what happens when you have a <blank> mapped to <blank>

 

david

Link to comment

Along the lines of what BRiT said, I have both of these volume mappings defined:

  /mnt  ->  /mnt

  /data ->  /mnt

 

I don't know if /data -> /mnt is needed, but I didn't delete it when adding /mnt -> /mnt.

 

You could exec into the docker and see if a /data directory exists and has content.  If so, that would confirm that the mapping is needed.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.