Solved - Cannot write to docker Image error

[gacpac]

So, I started getting this error. I'm also downloading movies to my cache drive and it's getting filled up. This never happened to me before and I'm using the latest build 6.6.5

 

I can delete the docker vdisk and recreate everything. But then after awhile it starts happening again. Right now, I have restarted my server and I'm assuming this happens because the cache drive is getting full, but then when I think about it.

 

The docker vdisk is a virtual disk of 30GB, so as long as my apps don't exceed that capacity it's not supposed to matter if my cache drive is full or not. 

 

Maybe I'm doing something wrong? If so, give me some suggestion maybe setting a minimum Cache Drive size under the Global Share settings. 

Edited December 26, 2018 by gacpac
Topic solved

[John_M]

You probably have corruption of the cache file system. Post diagnostics.

[itimpi]

You should post the system diagnostics ZIP file (obtained via Tools->Diagnostics) so that we can give some informed feedback.  

 

It sounds as if there is corruption either on your cache pool or your docker image - the diagnostics should help with checking for this.

[gacpac]

I just restarted my server. Wouldn't the logs go away?

[John_M]

Wait until the problems start, then grab diagnostics.

[gacpac]

As soon as the problem arises again I will post the diagnostics right away. For now, I have checked the file systems on all my data drives including cache, found no errors. Also increased the minimum cache drive size for Global Share settings to 3GB. 

[gacpac]

Okay, so here it happened again. My Cache drive filled again I'm downloading how I met your mother and after couple of minutes I started getting the same error. 

 

Please see the attached file 

tower-diagnostics-20181201-0243.zip

[gacpac]

I'm getting another error that is possible that my docker image is full by logs? 

 

I know if I restart it might be fixed, but that would be a shirt term solution 

tower-diagnostics-20181201-1009.zip

[John_M]

I see multiple issues.

 

There's a lot of this in your syslog. Problems with AFP service (are you using Time Machine?):

Nov 22 13:02:53 Tower afpd[32396]: afp_disconnect: primary reconnect failed
Nov 22 13:02:54 Tower afpd[32396]: dsi_stream_read: len:0, unexpected EOF

 

A huge amount of this. Related to the Dynamix File Integrity plugin, I believe, with strangely mangled file paths. Incorrect settings, perhaps? Maybe you need to exclude some locations that are used by temporary files:

Nov 22 13:06:01 Tower bunker: warning: lex Media Server/Logs/Plex Media Server.1.log is missing
Nov 22 13:06:01 Tower bunker: warning: gz is missing
Nov 22 13:06:01 Tower bunker: warning:  is missing

 

Quite a lot of this. I have no idea what "pulseway" is:

Nov 23 18:33:15 Tower pulseway: Error while registering online:SOAP 1.1 fault SOAP-ENV:Server[no subcode]#012"Timeout"#012Detail: connect failed in tcp_connect()
### [PREVIOUS LINE REPEATED 1 TIMES] ###

 

Yes, agreed, your cache disk is indeed full:

Nov 30 20:58:35 Tower shfs: cache disk full
### [PREVIOUS LINE REPEATED 18 TIMES] ###

 

And your docker.img is corrupt. Whether that's a cause or a symptom of the cache disk being full, I don't know:

Dec  1 02:42:38 Tower kernel: BTRFS error (device loop2): bad tree block start 0 110428160
### [PREVIOUS LINE REPEATED 9 TIMES] ###
Dec  1 02:42:42 Tower kernel: BTRFS info (device loop2): no csum found for inode 684 start 0

 

Perhaps you have a misconfigured download application that's writing to the cache and filling it up. So you'll want to check your docker container mappings. If all looks good, maybe run for a while with some containers disabled to find the culprit.

[JorgeB]

7 hours ago, gacpac said:

My Cache drive filled again

That is the problem, you can't get the cache drive completely filled or the docker won't have space for writes, there is a min cache free space setting and you also need to configure a a minimum free space for each share that uses cache, though if you have dockers for example downloading or unpacking to cache I'm not sure those minimum free space settings are enforced.

[Squid]

3 minutes ago, gacpac said:

I'm getting another error that is possible that my docker image is full by logs? 

 

Yeah, its not full

/dev/loop2       30G  6.7G   22G  24% /var/lib/docker

Once your cache drive fills up, BTRFS is having a conniption / hissy fit about everything

 

If you have zero plans about moving to a cache pool, I'd highly recommend to redo the cache drive as XFS instead of BTRFS.  Its far, far far more forgiving in these circumstances.

2 minutes ago, johnnie.black said:

though if you have dockers for example downloading or unpacking to cache I'm not sure those minimum free space settings are enforced.

depends upon how they are referenced in the template.

 

Specifying /mnt/cache/... as the host path, any minimum freespace settings or the useCache parameter is ignored.

Specifying /mnt/user/... as the host path, and the rules are enforced.

[gacpac]

So for the cache I have a 3GB limit and at some point it worked as expected saying. Your cache is full and we are writing to the array. The program I was using is transmission to download about 100gb. 

 

Time machine yes, I've been using it for backups, and file integrity plugin I've thinking of stop using it wince it doesn't really repair files. 

 

Pulseway I use to manage remotely my unraid server to know when it's down and to clean restart remotely just in case i get a power outage. 

[gacpac]

Right now I have btrfs because I can use the qcow format for my VMs. From the integrity i can exclude the app data I guess. And the docker image when that happened I recreate it and it happened again. However If I restart everything becomes normal. 

 

Now squid, is it worth using the integrity plugin? 

[John_M]

2 minutes ago, gacpac said:

So for the cache I have a 3GB limit and at some point it worked as expected saying. Your cache is full and we are writing to the array. The program I was using is transmission to download about 100gb.

That's what filled you cache, then. As long as you had more than 3 GB free it would start downloading to the cache, having no idea how big the file would eventually be. You need to set the minimum cache free to more than 100 GB - in fact, more than the largest file you expect to write there.

 

5 minutes ago, gacpac said:

file integrity plugin I've thinking of stop using it wince it doesn't really repair files. 

It is not intended to repair files.

 

[JorgeB]

3 minutes ago, gacpac said:

So for the cache I have a 3GB limit and at some point it worked as expected saying

It may have worked but something still completely filled the cache drive, the docker errors were write errors because of the lack of space, there was no filesytem problem with the cache drive itself:

 

Dec  1 02:40:15 Tower kernel: loop: Write error at byte offset 7097028608, length 4096.
Dec  1 02:40:15 Tower kernel: print_req_error: I/O error, dev loop2, sector 13861328
Dec  1 02:40:15 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

 

[JorgeB]

Just now, John_M said:

As long as you had more than 3 GB free it would start downloading to the cache, having no idea how big the file would eventually be. You need to set the minimum cache free to more than 100 GB - in fact, more than the largest file you expect to write there.

Good point.

[Squid]

 

12 minutes ago, gacpac said:

So for the cache I have a 3GB limit

 

When creating a file, the system looks at the amount of cache space free.  If its less than the minimum space then it'll write to the array instead of the cache drive (assuming use cache is yes or prefer)

12 minutes ago, gacpac said:

transmission to download about 100gb.

This is probably the key.

 

Assume that after the download (and all of the intermediate files) there is 50 GB free on the cache drive.  If the torrent (why people use them nowadays though I don't know, but to each their own) is a single 100GB file.  The system is going to look at the freespace and see that its more than the minimum and start writing to the cache.  But, since its a single zip file of 100GB, it won't fit, and the problems will happen.

Edited December 1, 2018 by Squid

[gacpac]

It's not a solid 100gb file. It's they ate about 600mb each file and in total fo to 100gb

[Squid]

@johnnie.black:  

 

A prior diagnostics (not the latest) showed this:

    Device size:		 119.24GiB
    Device allocated:		 119.24GiB
    Device unallocated:		   1.05MiB
    Device missing:		     0.00B
    Used:			  73.78GiB
    Free (estimated):		  43.67GiB	(min: 43.67GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		  23.00MiB	(used: 0.00B)

 

[gacpac]

Exactly and that's right. That was after the mover took place at 3 AM but now is empty. Everything should be fine now? 

[JorgeB]

The errors happened before the mover ran, and at that point the cache was full:

 

    Free (estimated):           3.05GiB    (min: 3.05GiB)

This was at the time the diags were created, but the errors happened before, and if there was a file e.g. unziping or downloading to cache it would be completely full.

[Squid]

12 minutes ago, gacpac said:

That was after the mover took place at 3 AM but now is empty.

On such a small cache drive, why not run mover more often?  Or fine tune the frequency of its runs via the Mover Tuning plugin?

[gacpac]

I'll check out the mover plugin if that's what it is causing the problem. But I guess it breaks the purpose of it's full save to the array. 

 

Then what about the logs. Do I need to worry with Time Machine or the file integrity plugin? 

 

I'll appreciate if there's any tip I should take for those. 

[itimpi]

What location are you downloading to?    

 

If it is a share have you made sure it has a Min Free Space setting under Global Share settings that is equal to (or larger) than the largest file you will download?   When the free space drops below that value new files will go directly to the array

 

If explicitly specifying the cache drive then that would explain why you cache disk keeps getting full with undesirable consequences.  When explicitly specifying the cache there is no way to control running out of space and you do not have a very large cache drive.

[gacpac]

The larger I would download is 3Gb at a time. And that's what I've set up. The files I'm downloading are about 600mb each. Which is what confused me. 

 

My download location is in the /mnt/user/share

 

Not /mnt/disk/cache

Edited December 1, 2018 by gacpac
Forgot to add text