[Guide] Fixing Nextcloud Uploads Filling up Docker Image

[Progeny42]

I've been trialling Nextcloud as a cloud backup service for myself, and if successful, my family which live remotely. I'm using Duplicati to perform the backups, but that's not the point of this guide. The point is that when I backup files to Nextcloud, the Docker Image slowly fills up. I've never had it reach a point where the Image reaches 100%, but it's probably not a fun time.

 

After searching the internet for hours trying to find anything, I eventually figured out what was required.

 

For context, I'm running Nextcloud behind a Reverse Proxy, for which I'm using Swag (Let's Encrypt).

 

Through trial and error, the behaviour I observed is that when uploading files (via WebDAV in my case), they get put in the /tmp folder of Swag. Once they are fully uploaded, they are copied across to Nextcloud's /temp directory. Therefore, both paths need to be added as Bind mounts for this to work.

 

What To Do

Head over to the Docker tab, and edit Nextcloud and add a new Path variable:

Name: Temp 
Container Path: /tmp 
Host Path: /mnt/user/appdata/nextcloud/temp

Next edit the Swag (or Let's Encrypt) container, and add a new Path variable:

Name: Temp 
Container Path: /var/lib/nginx/tmp 
Host Path: /mnt/user/appdata/swag/temp

 

And that's it! Really simple fix, but no one seemed to have an answer. Now when I backup my files, the Docker image no longer fills up.

[greenflash24]

I have also experienced this exact issue and could resolve it also through mounting the above mentioned Paths.

 

Only difference for me is that i have altered the Host Paths to the Unraid /tmp Dir so i am using the RAM and not the Cache drive for the Temp files.

Therefore my writes on the cache drive are not as high.

 

Host Path: /tmp/nextcloud/

and

Host Path: /tmp/swag/

 

[JonathanM]

4 hours ago, greenflash24 said:

Only difference for me is that i have altered the Host Paths to the Unraid /tmp Dir so i am using the RAM and not the Cache drive for the Temp files.

Be careful with this approach, it's easy to run out of RAM and cause bad results. Only do this if you have a very good handle on managing your server.

[blaine07]

On 10/2/2020 at 7:50 AM, jonathanm said:

Be careful with this approach, it's easy to run out of RAM and cause bad results. Only do this if you have a very good handle on managing your server.

So are their no other edits in any of the Nginx PHP Files or Nextcloud config files to point temp directories at the locations were defining here? Does "Nextcloud need told where temp directory" is in a config file or something anywhere? Swag/Letsencrypt need told same in any of the configs?

[riddler0815]

I have the same problem and I inserted the additional paths, but I still have the same issue, running my docker full everytime when I download data from my nextcloud.

 

As @blaine07 mentioned, is it really sufficient to add the variables in the docker or is there an adaption somewhere else needed?

[boomam]

This doesn't sound right - you are pointing the temp files back at appdata? Defeats the idea surely?
Would it not be better to store in perhaps a 'temp' share?

 

To add for testing - on both the linuxserver/nextcloud image, and the NPM container, /tmp does not redirect inside the container to any host variable whatsoever. Very strange.

Edited January 25, 2021 by boomam

[TDA]

Same problem 😞
Is there any reliable solution?

[mgutt]

11 hours ago, TDA said:

Same problem 😞

 

Did you set "/mnt/user/appdata/nextcloud/temp"? If yes, are there any files in this folder when you upload files?

 

Alternatively upload your file and execute this command through the Webterminal:

find /var/lib/docker -type f -printf "%[email protected] %p\n" | sort -n | cut -d' ' -f 2- | tail -n 50

 

It shows you the 50 most recent files in the docker.img.

 

[TDA]

37 minutes ago, mgutt said:

 

Did you set "/mnt/user/appdata/nextcloud/temp"? If yes, are there any files in this folder when you upload files?

 

Alternatively upload your file and execute this command through the Webterminal:

find /var/lib/docker -type f -printf "%[email protected] %p\n" | sort -n | cut -d' ' -f 2- | tail -n 50

 

It shows you the 50 most recent files in the docker.img.

 

Hi,

I've set it so:

[blaine07]

I feel like this temp directory must be set in Nextcloud config.php or none of this directory stuff above matters?

[mgutt]

17 minutes ago, TDA said:

I've set it so:

 

 

I'm talking about Nextcloud. And what does my command return?

 

Changing tmp-paths of SWAG or NPM is not needed. Those are proxies which forward data and not saving it to disk (which would be extremely inefficient). If they would do that, they would be misconfigured.

[mgutt]

28 minutes ago, blaine07 said:

I feel like this temp directory must be set in Nextcloud config.php or none of this directory stuff above matters?

 

 

 

This could propably help. The problem is, that you have a tmp dir in the OS, a tmp dir in the webserver and a tmp dir in PHP. And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.

 

So give it a try. Set the tmp dir through the config.php to "/tmp" and follow the step in the first post to map it to a different location like "/mnt/user/appdata/nextcloud/temp". If you want to map it to Unraids "/tmp" dir which is located in your servers RAM, you must create a tmpfs dir first as mentioned in this guide:

https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460

 

Otherwise it could totally fill up the RAM. So simply using "/mnt/user/appdata/nextcloud/temp" is easier.

 

Mapping the Proxy Containers tmp dir is not needed as I explained before.

 

[blaine07]

 
 
This could propably help. The problem is, that you have a tmp dir in the OS, a tmp dir in the webserver and a tmp dir in PHP. And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.
 
So give it a try. Set the tmp dir through the config.php to "/tmp" and follow the step in the first post to map it to a different location like "/mnt/user/appdata/nextcloud/temp". If you want to map it to Unraids "/tmp" dir which is located in your servers RAM, you must create a tmpfs dir first as mentioned in this guide:
https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460
 
Otherwise it could totallly fill up the RAM.
 
Mapping the Proxy Containers tmp dir is not needed as I explained before.
 

I’ll have to play with it later today. Makes sense I’m my head; what you’re saying.

[TDA]

2 hours ago, mgutt said:

 

 

This could propably help. The problem is, that you have a tmp dir in the OS, a tmp dir in the webserver and a tmp dir in PHP. And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.

 

So give it a try. Set the tmp dir through the config.php to "/tmp" and follow the step in the first post to map it to a different location like "/mnt/user/appdata/nextcloud/temp". If you want to map it to Unraids "/tmp" dir which is located in your servers RAM, you must create a tmpfs dir first as mentioned in this guide:

https://forums.unraid.net/topic/35878-plex-guide-to-moving-transcoding-to-ram/page/12/?tab=comments#comment-894460

 

Otherwise it could totally fill up the RAM. So simply using "/mnt/user/appdata/nextcloud/temp" is easier.

 

Mapping the Proxy Containers tmp dir is not needed as I explained before.

 

I will try later - but I've set NEXTCLOUD to go to the TEMP because I want it in the ram, not on my slow disks.

And I don't plan to upload a 100GB File - so I can't really fill up my ram with an upload.

[mgutt]

 

2 minutes ago, TDA said:

I will try later - but I've set NEXTCLOUD to go to the TEMP because I want it in the ram, not on my slow disks.

 

 

If you upload a file, it will be finally moved on the disk. And "slow" depends on your network connection. I mean, are you using 10G ethernet or why is your disk too slow?

 

 

Note: Linux already uses your RAM as write cache. The setting is called "vm.dirty_ratio" or "vm.dirty_bytes" and can be changed through the Tips and Tweaks plugin or manually through your Go-File. I added this to have 50% of my RAM as write cache:

# -------------------------------------------------
# Raise write cache
# -------------------------------------------------
sysctl vm.dirty_ratio=50

 

 

I mentioned this in my guide in #7.

[boomam]

On 1/26/2021 at 7:36 AM, mgutt said:

....And finally as you mentioned, you could set a custom tmp dir for Nextcloud, too.

 

Otherwise it could totally fill up the RAM. So simply using "/mnt/user/appdata/nextcloud/temp" is easier.

 

Mapping the Proxy Containers tmp dir is not needed as I explained before.

 

As a note, NextCloud, if unspecified, uses /tmp for temp files.

So unconfigured, it will use /tmp.

 

Re: using appdata for temp storage

This to me seems somewhat inefficient, as the appdata should literally just be app data (configs), and not actual data storage for something like NextCloud.  

I've set mine to a custom share called 'temp'.

 

However to note....it still doesn't work.

[AnimusAstralis]

I'm so glad I've found this guide. My Docker image was filling up when users tried to download a large folder (>50gb), creating volumes solved this problem.

[mgutt]

14 hours ago, boomam said:

This to me seems somewhat inefficient

Is it? I mean it becomes deleted through Nextcloud if the upload / download process is finished or not?

[boomam]

9 hours ago, mgutt said:

Is it? I mean it becomes deleted through Nextcloud if the upload / download process is finished or not?

Generally speaking, configuration storage shouldn't be used for other storage types, even if temporary.

Whilst I doubt it would have a large effect for the average unraid user, in a production container environment, it would be a big no-no.

 

For myself, I'd rather not have the extra writes on the SSD cache that has my appdata, but seeing as the redirection fails to work whatever its set to for me, its somewhat moot either way - I'm hoping its just a bug in 6.8.3, and will resolve when 6.9 gets released.

Edited January 29, 2021 by boomam

[mgutt]

7 hours ago, boomam said:

Generally speaking, configuration storage shouldn't be used for other storage types, even if temporary.

 

Your argument doesn't make sense as the docker.img is on the same disk (usually) and all containers write their temporary data to it.

[boomam]

6 hours ago, mgutt said:

 

Your argument doesn't make sense as the docker.img is on the same disk (usually) and all containers write their temporary data to it.

You are thinking in Unraid terms.

In an actual production system that uses containers, that kind of setup of file storage would be seen as bad practice.

 

For Unraid specifically, yes you can get away with it, but the docker.img is only 'usually' on the same disk if the user has set it to be so.

Having a separate share specifically for temp files in several apps, allows the user to pick where they want said data temporarily storing, allowing the potential for choice in speed, resiliency and IOP queues.

[mgutt]

7 hours ago, boomam said:

You are thinking in Unraid terms.

Checked the URL of the forums... Yes we are using Unraid 😜

[boomam]

On 1/29/2021 at 6:33 PM, mgutt said:

Checked the URL of the forums... Yes we are using Unraid 😜

True, but just because you are suggesting moving it to a place that the wider industry considers to be questionable, and the platform in use is Unraid, doesn't really mean anything in regards to the community forum's domain in use

 

Anyway, out of curiosity, are you running 6.8 or 6.9?

I've a little time today and I want to see if I can work out why the redirects are getting ignored, starting with a difference in the docker package version/unraid version.

[bpl294]

I was able to get this resolved by setting a RAM /tmp mapping in the Nextcloud docker and then disabling proxy buffering by adding the following strings to the NPM proxy host settings:

 

proxy_request_buffering off;
proxy_buffering off;

 

 

Edited February 22, 2021 by bpl294

[Fallwater]

Thanx @Progeny42  you saved my docker image! 🙂