Progeny42 Posted September 29, 2020 Share Posted September 29, 2020 I've been trialling Nextcloud as a cloud backup service for myself, and if successful, my family which live remotely. I'm using Duplicati to perform the backups, but that's not the point of this guide. The point is that when I backup files to Nextcloud, the Docker Image slowly fills up. I've never had it reach a point where the Image reaches 100%, but it's probably not a fun time.  After searching the internet for hours trying to find anything, I eventually figured out what was required.  For context, I'm running Nextcloud behind a Reverse Proxy, for which I'm using Swag (Let's Encrypt).  Through trial and error, the behaviour I observed is that when uploading files (via WebDAV in my case), they get put in the /tmp folder of Swag. Once they are fully uploaded, they are copied across to Nextcloud's /temp directory. Therefore, both paths need to be added as Bind mounts for this to work.  What To Do Head over to the Docker tab, and edit Nextcloud and add a new Path variable: Name: Temp Container Path: /tmp Host Path: /mnt/user/appdata/nextcloud/temp Next edit the Swag (or Let's Encrypt) container, and add a new Path variable: Name: Temp Container Path: /var/lib/nginx/tmp Host Path: /mnt/user/appdata/swag/temp  And that's it! Really simple fix, but no one seemed to have an answer. Now when I backup my files, the Docker image no longer fills up. 6 3 Quote Link to comment
greenflash24 Posted October 2, 2020 Share Posted October 2, 2020 I have also experienced this exact issue and could resolve it also through mounting the above mentioned Paths.  Only difference for me is that i have altered the Host Paths to the Unraid /tmp Dir so i am using the RAM and not the Cache drive for the Temp files. Therefore my writes on the cache drive are not as high.  Host Path: /tmp/nextcloud/ and Host Path: /tmp/swag/  Quote Link to comment
JonathanM Posted October 2, 2020 Share Posted October 2, 2020 4 hours ago, greenflash24 said: Only difference for me is that i have altered the Host Paths to the Unraid /tmp Dir so i am using the RAM and not the Cache drive for the Temp files. Be careful with this approach, it's easy to run out of RAM and cause bad results. Only do this if you have a very good handle on managing your server. Quote Link to comment
blaine07 Posted October 6, 2020 Share Posted October 6, 2020 On 10/2/2020 at 7:50 AM, jonathanm said: Be careful with this approach, it's easy to run out of RAM and cause bad results. Only do this if you have a very good handle on managing your server. So are their no other edits in any of the Nginx PHP Files or Nextcloud config files to point temp directories at the locations were defining here? Does "Nextcloud need told where temp directory" is in a config file or something anywhere? Swag/Letsencrypt need told same in any of the configs? Quote Link to comment
riddler0815 Posted January 6, 2021 Share Posted January 6, 2021 I have the same problem and I inserted the additional paths, but I still have the same issue, running my docker full everytime when I download data from my nextcloud. Â As @blaine07Â mentioned, is it really sufficient to add the variables in the docker or is there an adaption somewhere else needed? Quote Link to comment
boomam Posted January 25, 2021 Share Posted January 25, 2021 (edited) This doesn't sound right - you are pointing the temp files back at appdata? Defeats the idea surely? Would it not be better to store in perhaps a 'temp' share? Â To add for testing -Â on both the linuxserver/nextcloud image, and the NPM container, /tmp does not redirect inside the container to any host variable whatsoever. Very strange. Edited January 25, 2021 by boomam Quote Link to comment
TDA Posted January 25, 2021 Share Posted January 25, 2021 Same problem 😞 Is there any reliable solution? Quote Link to comment
mgutt Posted January 26, 2021 Share Posted January 26, 2021 11 hours ago, TDA said: Same problem 😞  Did you set "/mnt/user/appdata/nextcloud/temp"? If yes, are there any files in this folder when you upload files?  Alternatively upload your file and execute this command through the Webterminal: find /var/lib/docker -type f -printf "%T@ %p\n" | sort -n | cut -d' ' -f 2- | tail -n 50  It shows you the 50 most recent files in the docker.img.  Quote Link to comment
TDA Posted January 26, 2021 Share Posted January 26, 2021 37 minutes ago, mgutt said: Â Did you set "/mnt/user/appdata/nextcloud/temp"? If yes, are there any files in this folder when you upload files? Â Alternatively upload your file and execute this command through the Webterminal: find /var/lib/docker -type f -printf "%T@ %p\n" | sort -n | cut -d' ' -f 2- | tail -n 50 Â It shows you the 50 most recent files in the docker.img. Â Hi, I've set it so: Quote Link to comment
blaine07 Posted January 26, 2021 Share Posted January 26, 2021 I feel like this temp directory must be set in Nextcloud config.php or none of this directory stuff above matters? Quote Link to comment
mgutt Posted January 26, 2021 Share Posted January 26, 2021 17 minutes ago, TDA said: I've set it so:   I'm talking about Nextcloud. And what does my command return?  Changing tmp-paths of SWAG or NPM is not needed. Those are proxies which forward data and not saving it to disk (which would be extremely inefficient). If they would do that, they would be misconfigured. Quote Link to comment
mgutt Posted January 26, 2021 Share Posted January 26, 2021 28 minutes ago, blaine07 said: I feel like this temp directory must be set in Nextcloud config.php or none of this directory stuff above matters?    This could propably help. The problem is, that you have a tmp dir in the OS, a tmp dir in the webserver and a tmp dir in PHP. And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.  So give it a try. Set the tmp dir through the config.php to "/tmp" and follow the step in the first post to map it to a different location like "/mnt/user/appdata/nextcloud/temp". If you want to map it to Unraids "/tmp" dir which is located in your servers RAM, you must create a tmpfs dir first as mentioned in this guide: https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460  Otherwise it could totally fill up the RAM. So simply using "/mnt/user/appdata/nextcloud/temp" is easier.  Mapping the Proxy Containers tmp dir is not needed as I explained before.  1 Quote Link to comment
blaine07 Posted January 26, 2021 Share Posted January 26, 2021   This could propably help. The problem is, that you have a tmp dir in the OS, a tmp dir in the webserver and a tmp dir in PHP. And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.  So give it a try. Set the tmp dir through the config.php to "/tmp" and follow the step in the first post to map it to a different location like "/mnt/user/appdata/nextcloud/temp". If you want to map it to Unraids "/tmp" dir which is located in your servers RAM, you must create a tmpfs dir first as mentioned in this guide: https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460  Otherwise it could totallly fill up the RAM.  Mapping the Proxy Containers tmp dir is not needed as I explained before.  I’ll have to play with it later today. Makes sense I’m my head; what you’re saying. Quote Link to comment
TDA Posted January 26, 2021 Share Posted January 26, 2021 2 hours ago, mgutt said:   This could propably help. The problem is, that you have a tmp dir in the OS, a tmp dir in the webserver and a tmp dir in PHP. And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.  So give it a try. Set the tmp dir through the config.php to "/tmp" and follow the step in the first post to map it to a different location like "/mnt/user/appdata/nextcloud/temp". If you want to map it to Unraids "/tmp" dir which is located in your servers RAM, you must create a tmpfs dir first as mentioned in this guide: https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460  Otherwise it could totally fill up the RAM. So simply using "/mnt/user/appdata/nextcloud/temp" is easier.  Mapping the Proxy Containers tmp dir is not needed as I explained before.  I will try later - but I've set NEXTCLOUD to go to the TEMP because I want it in the ram, not on my slow disks. And I don't plan to upload a 100GB File - so I can't really fill up my ram with an upload. Quote Link to comment
mgutt Posted January 26, 2021 Share Posted January 26, 2021  2 minutes ago, TDA said: I will try later - but I've set NEXTCLOUD to go to the TEMP because I want it in the ram, not on my slow disks.   If you upload a file, it will be finally moved on the disk. And "slow" depends on your network connection. I mean, are you using 10G ethernet or why is your disk too slow?   Note: Linux already uses your RAM as write cache. The setting is called "vm.dirty_ratio" or "vm.dirty_bytes" and can be changed through the Tips and Tweaks plugin or manually through your Go-File. I added this to have 50% of my RAM as write cache: # ------------------------------------------------- # Raise write cache # ------------------------------------------------- sysctl vm.dirty_ratio=50   I mentioned this in my guide in #7. Quote Link to comment
boomam Posted January 28, 2021 Share Posted January 28, 2021 On 1/26/2021 at 7:36 AM, mgutt said: ....And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.  Otherwise it could totally fill up the RAM. So simply using "/mnt/user/appdata/nextcloud/temp" is easier.  Mapping the Proxy Containers tmp dir is not needed as I explained before.  As a note, NextCloud, if unspecified, uses /tmp for temp files. So unconfigured, it will use /tmp.  Re: using appdata for temp storage This to me seems somewhat inefficient, as the appdata should literally just be app data (configs), and not actual data storage for something like NextCloud.  I've set mine to a custom share called 'temp'.  However to note....it still doesn't work. Quote Link to comment
AnimusAstralis Posted January 28, 2021 Share Posted January 28, 2021 I'm so glad I've found this guide. My Docker image was filling up when users tried to download a large folder (>50gb), creating volumes solved this problem. Quote Link to comment
mgutt Posted January 28, 2021 Share Posted January 28, 2021 14 hours ago, boomam said: This to me seems somewhat inefficient Is it? I mean it becomes deleted through Nextcloud if the upload / download process is finished or not? Quote Link to comment
boomam Posted January 29, 2021 Share Posted January 29, 2021 (edited) 9 hours ago, mgutt said: Is it? I mean it becomes deleted through Nextcloud if the upload / download process is finished or not? Generally speaking, configuration storage shouldn't be used for other storage types, even if temporary. Whilst I doubt it would have a large effect for the average unraid user, in a production container environment, it would be a big no-no. Â For myself, I'd rather not have the extra writes on the SSD cache that has my appdata, but seeing as the redirection fails to work whatever its set to for me, its somewhat moot either way - I'm hoping its just a bug in 6.8.3, and will resolve when 6.9 gets released. Edited January 29, 2021 by boomam Quote Link to comment
mgutt Posted January 29, 2021 Share Posted January 29, 2021 7 hours ago, boomam said: Generally speaking, configuration storage shouldn't be used for other storage types, even if temporary. Â Your argument doesn't make sense as the docker.img is on the same disk (usually) and all containers write their temporary data to it. Quote Link to comment
boomam Posted January 29, 2021 Share Posted January 29, 2021 6 hours ago, mgutt said: Â Your argument doesn't make sense as the docker.img is on the same disk (usually) and all containers write their temporary data to it. You are thinking in Unraid terms. In an actual production system that uses containers, that kind of setup of file storage would be seen as bad practice. Â For Unraid specifically, yes you can get away with it, but the docker.img is only 'usually' on the same disk if the user has set it to be so. Having a separate share specifically for temp files in several apps, allows the user to pick where they want said data temporarily storing, allowing the potential for choice in speed, resiliency and IOP queues. Quote Link to comment
mgutt Posted January 29, 2021 Share Posted January 29, 2021 7 hours ago, boomam said: You are thinking in Unraid terms. Checked the URL of the forums... Yes we are using Unraid 😜 Quote Link to comment
boomam Posted January 31, 2021 Share Posted January 31, 2021 On 1/29/2021 at 6:33 PM, mgutt said: Checked the URL of the forums... Yes we are using Unraid 😜 True, but just because you are suggesting moving it to a place that the wider industry considers to be questionable, and the platform in use is Unraid, doesn't really mean anything in regards to the community forum's domain in use  Anyway, out of curiosity, are you running 6.8 or 6.9? I've a little time today and I want to see if I can work out why the redirects are getting ignored, starting with a difference in the docker package version/unraid version. Quote Link to comment
bpl294 Posted February 22, 2021 Share Posted February 22, 2021 (edited) I was able to get this resolved by setting a RAM /tmp mapping in the Nextcloud docker and then disabling proxy buffering by adding the following strings to the NPM proxy host settings:  proxy_request_buffering off; proxy_buffering off;   Edited February 22, 2021 by bpl294 1 Quote Link to comment
Fallwater Posted May 20, 2021 Share Posted May 20, 2021 Thanx @Progeny42 you saved my docker image! 🙂 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.