• [6.8.3] docker image huge amount of unnecessary writes on cache


    S1dney
    • Solved Urgent

    EDIT (March 9th 2021):

    Solved in 6.9 and up. Reformatting the cache to new partition alignment and hosting docker directly on a cache-only directory brought writes down to a bare minimum.

     

    ###

     

    Hey Guys,

     

    First of all, I know that you're all very busy on getting version 6.8 out there, something I'm very much waiting on as well. I'm seeing great progress, so thanks so much for that! Furthermore I won't be expecting this to be on top of the priority list, but I'm hoping someone of the developers team is willing to invest (perhaps after the release).

     

    Hardware and software involved:

    2 x 1TB Samsung EVO 860, setup with LUKS encryption in BTRFS RAID1 pool.

     

    ###

    TLDR (but I'd suggest to read on anyway 😀)

    The image file mounted as a loop device is causing massive writes on the cache, potentially wearing out SSD's quite rapidly.

    This appears to be only happening on encrypted caches formatted with BTRFS (maybe only in RAID1 setup, but not sure).

    Hosting the Docker files directory on /mnt/cache instead of using the loopdevice seems to fix this problem.

    Possible idea for implementation proposed on the bottom.

     

    Grateful for any help provided!

    ###

     

    I have written a topic in the general support section (see link below), but I have done a lot of research lately and think I have gathered enough evidence pointing to a bug, I also was able to build (kind of) a workaround for my situation. More details below.

     

    So to see what was actually hammering on the cache I started doing all the obvious, like using a lot of find commands to trace files that were written to every few minutes and also used the fileactivity plugin. Neither was able trace down any writes that would explain 400 GBs worth of writes a day for just a few containers that aren't even that active.

     

    Digging further I moved the docker.img to /mnt/cach/system/docker/docker.img, so directly on the BTRFS RAID1 mountpoint. I wanted to check whether the unRAID FS layer was causing the loop2 device to write this heavy. No luck either.

    This gave me a situation I was able to reproduce on a virtual machine though, so I started with a recent Debian install (I know, it's not Slackware, but I had to start somewhere ☺️). I create some vDisks, encrypted them with LUKS, bundled them in a BTRFS RAID1 setup, created the loopdevice on the BTRFS mountpoint (same of /dev/cache) en mounted it on /var/lib/docker. I made sure I had to NoCow flags set on the IMG file like unRAID does. Strangely this did not show any excessive writes, iotop shows really healthy values for the same workload (I migrated the docker content over to the VM).

     

    After my Debian troubleshooting I went back over to the unRAID server, wondering whether the loopdevice is created weirdly, so I took the exact same steps to create a new image and pointed the settings from the GUI there. Still same write issues. 

     

    Finally I decided to put the whole image out of the equation and took the following steps:

    - Stopped docker from the WebGUI so unRAID would properly unmount the loop device.

    - Modified /etc/rc.d/rc.docker to not check whether /var/lib/docker was a mountpoint

    - Created a share on the cache for the docker files

    - Created a softlink from /mnt/cache/docker to /var/lib/docker

    - Started docker using "/etc/rd.d/rc.docker start"

    - Started my BItwarden containers.

     

    Looking into the stats with "iotstat -ao" I did not see any excessive writing taking place anymore.

    I had the containers running for like 3 hours and maybe got 1GB of writes total (note that on the loopdevice this gave me 2.5GB every 10 minutes!)

     

    Now don't get me wrong, I understand why the loopdevice was implemented. Dockerd is started with options to make it run with the BTRFS driver, and since the image file is formatted with the BTRFS filesystem this works at every setup, it doesn't even matter whether it runs on XFS, EXT4 or BTRFS and it will just work. I my case I had to point the softlink to /mnt/cache because pointing it /mnt/user would not allow me to start using the BTRFS driver (obviously the unRAID filesystem isn't BTRFS). Also the WebGUI has commands to scrub to filesystem inside the container, all is based on the assumption everyone is using docker on BTRFS (which of course they are because of the container 😁)

    I must say that my approach also broke when I changed something in the shares, certain services get a restart causing docker to be turned off for some reason. No big issue since it wasn't meant to be a long term solution, just to see whether the loopdevice was causing the issue, which I think my tests did point out.

     

    Now I'm at the point where I would definitely need some developer help, I'm currently keeping nearly all docker container off all day because 300/400GB worth of writes a day is just a BIG waste of expensive flash storage. Especially since I've pointed out that it's not needed at all. It does defeat the purpose of my NAS and SSD cache though since it's main purpose was hosting docker containers while allowing the HD's to spin down.

     

    Again, I'm hoping someone in the dev team acknowledges this problem and is willing to invest. I did got quite a few hits on the forums and reddit without someone actually pointed out the root cause of issue.

     

    I missing the technical know-how to troubleshoot the loopdevice issues on a lower level, but have been thinking on possible ways to implement a workaround. Like adjusting the Docker Settings page to switch off the use of a vDisk and if all requirements are met (pointing to /mnt/cache and BTRFS formatted) start docker on a share on the /mnt/cache partition instead of using the vDisk.

    In this way you would still keep all advantages of the docker.img file (cross filesystem type) and users who don't care about writes could still use it, but you'd be massively helping out others that are concerned over these writes.

     

    I'm not attaching diagnostic files since they would probably not point out the needed.

    Also if this should have been in feature requests, I'm sorry. But I feel that, since the solution is misbehaving in terms of writes, this could also be placed in the bugreport section.

     

    Thanks though for this great product, have been using it so far with a lot of joy! 

    I'm just hoping we can solve this one so I can keep all my dockers running without the cache wearing out quick,

     

    Cheers!

     

    • Like 3
    • Thanks 17



    User Feedback

    Recommended Comments



    While this is fresh on my mind, here is a bit of a writeup on how to deal with docker writes. It is not that hard once you know what the writes are caused by and the commands to stop them.

     

    All the commands I will put below need to be entered into the

     

    Extra Parameters:

     

    section of the docker template in unraid.

     

    To match up a container ID with a container in unraid GUI, simply use crtl+f to search the docker page in unraid for the container ID you see in the activity logs. Generally the first 3 or 4 characters are enough to find the right container.

     

    There are a few basic places writes come from with docker and each has it's own fix.

     

    ------------------------------------------------------------------------------------------------------------------

     

    The first step is to run the inotifywait command from mgutt:

     

    Quote

    You could start with this, which returns the 100 most recent files of the docker directory:

     

    find /mnt/user/system/docker -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d: -f2- | head -n100

     

    Quote

    Another method would be to log all file changes:

     

    inotifywait -e create,modify,attrib,moved_from,moved_to --timefmt %c --format '%T %_e %w %f' -mr /var/lib/docker > /mnt/user/system/recent_modified_files_$(date +"%Y%m%d_%H%M%S").txt

     

    I chose to make a userscript with this command and then use the "run in background" option so I don't have to keep the terminal open.

     

    To kill the inotifywait run this:

    pkill -xc inotifywait

     

    ------------------------------------------------------------------------------------------------------------------

     

    For me the first and most common writes came from the internal logging mechanism in docker. It basically logs all the messages that would show up in the terminal if it was run directly among other stuff.

     

    These writes will be to:

    /var/lib/docker/containers/{containerid}/{containerid}-json.log

     

    These are stopped by the following command while leaving the unriad GUI logs in tact:

    --log-driver syslog --log-opt syslog-address=udp://127.0.0.1:541

     

    ------------------------------------------------------------------------------------------------------------------

     

    The next type of writes are from:

    /var/lib/docker/containers/{containerid}/container-cached.log

     

    These are the logs you see when you click the log option in the unraid gui, these require a stronger version of the above command:

    --log-driver none

     

    ------------------------------------------------------------------------------------------------------------------

     

    Next up are the helthcheck logs, these are seen as writes to these files:

    /var/lib/docker/containers/{containerID}/.tmp-hostconfig.json{randomnumbers}
    /var/lib/docker/containers/{containerID}/.tmp-config.v2.json{randomnumbers}

     

    These are solved by either extending the health checks or disabling them. I prefer extending them to ~1 hour.

    --health-interval=60m

     

    They can be disabled completely with:

    --no-healthcheck

     

    ------------------------------------------------------------------------------------------------------------------

     

    The next type of writes are internal logs from the program in the container to the /tmp directory of the container

    /var/lib/docker/containers/{containerid}/tmp/some type of log file

     

    This is from a program thinking it is writing to a ramdrive but by default docker does not map a ramdrive to the /tmp directory. You can do it yourself easily though with the following command (can be adapted to other dirs and use cases as well).

     

    This command creates a ramdrive in /tmp with full read/write permissions and a max size of 256gb (much larger then needed in most cases but it only uses ram as needed so should not hurt anything in most cases, you can make it smaller as well):

    --mount type=tmpfs,destination=/tmp,tmpfs-mode=1777,tmpfs-size=256000000

     

    ------------------------------------------------------------------------------------------------------------------

     

    After this things get a bit more complicated. Each container will behave differently and you will kinda have to wing it. I saw random writes to various files in containers, sometimes you could change the logging folder in the program to the /tmp folder.

     

    Others you need to map another ramdrive and still others need other workarounds unique to that specific program. The above commands should be pretty much all you need to figure it out on dockers end though combined with normal folder mappings and such.

     

    Once you know the writes and commands to fix them it is not that hard to log it and VASTLY reduce the writes.

     

    I am still testing and won't know the true effects for a few days but the activity logs are a small fraction of what they were when I started which gives me hope.

     

    I will update and possibly create a new thread with this information once I get results.

    Edited by TexasUnraid
    Link to comment
    5 minutes ago, mgutt said:
    
    docker ps --no-trunc

     

     

    Good command but sadly that only shows the full container ID. I have not found any way to connect the container ID to the btrfs/subvolume ID

    Link to comment

    Ok, so my latest write report came in and I am quite happy with the results.

     

    Before disabling docker logs / extending helthchecks to 1hour on a BTRFS formatted drive with a BTRFS docker image:

    75-85gb writes/day

    Disabling the "low hanging fruit" docker logs on the same btrfs/btrfs setup:

    50gb/day

    Disabling almost all the docker logs and extending healthchcks to 1 hour

    39gb/day

     

    Have a feeling that would come down a bit more if left for longer, so basically cut the writes in half, which is about right when you consider the next datapoint.

     

    When I first started logging the writes in docker the activity log was around ~100mb/day.

     

    Today I got a mere 2.5mb/day, most of that came when one container did some kind of update and flooded the log, excluding that it is like 0.5mb

     

    I also started monitoring writes in appdata and the activity log for it is 75mb/day, so all said, cutting writes in half is about right.

     

    If I go back to an XFS formatted drive, I should cut the writes by 3x, dropping me down to ~12gb/day, not bad at all.

     

    This is as far as I can reasonably take the docker side of the write issue, the remaining logs I do need on occasion so worth a few writes for them.

     

    Everything else is program dependent in appdata.

    Edited by TexasUnraid
    • Thanks 2
    Link to comment

    So looking into the appdata writes now, noticed that lancache was causing a fair amount of writes to the log files, I have never needed the logs so I disabled the appdata bind for those and instead left them in the containers and setup a ramdisk for the log folders to stop those writes but they are still accessible inside the container if needed.

     

    After that the worst offenders by far are the *arr's (aka, sonarr etc). The constant updates and logging that can not be disabled, adjusted or tweaked in any way causes quite a bit of writes unnecessarily. I don't need the logs and wish I could disable them until needed and also wish I could adjust the update timing, it is way more often then I need. Also causes my drives to spin up in the middle of the night for no reason.

     

    As a workaround I am testing a setup like this:

     

    I added a ramdisk folder to appdata

    In my array startup script I mount a ramdisk in that folder

    In the array start script I rsync all the *arr's appdata folders into the ramdisk

    I changed all those dockers to use the ramdisk

    I setup a new script that will run every 1-3 hours (TBD) that will rsync the ramdisk back to appdata so that I won't loose data in those dockers (at least more then a few hours worth).

     

    The only issue is that it has to copy the whole database back into appdata everytime it rsyncs, between them all that works out to around ~150mb per sync.

     

    So still around 2gb/day in writes doing it every 2 hours but it should have much less write amplification this way so only way to know the effects is to try it.

     

    It also requires 4gb of ram for the ramdisk and I have 8gb allocated just in case, fine for me but would not be so practical on a system with limited memory.

     

    Gonna leave this for a day or 2 and see the results.

     

    If only there was a way to selectively change the cache on a per folder or drive basis.

    Link to comment
    17 minutes ago, TexasUnraid said:

    So looking into the appdata writes now, noticed that lancache was causing a fair amount of writes to the log files, I have never needed the logs so I disabled the appdata bind for those and instead left them in the containers and setup a ramdisk for the log folders to stop those writes but they are still accessible inside the container if needed.

     

    After that the worst offenders by far are the *arr's (aka, sonarr etc). The constant updates and logging that can not be disabled, adjusted or tweaked in any way causes quite a bit of writes unnecessarily. I don't need the logs and wish I could disable them until needed and also wish I could adjust the update timing, it is way more often then I need. Also causes my drives to spin up in the middle of the night for no reason.

     

    As a workaround I am testing a setup like this:

     

    I added a ramdisk folder to appdata

    In my array startup script I mount a ramdisk in that folder

    In the array start script I rsync all the *arr's appdata folders into the ramdisk

    I changed all those dockers to use the ramdisk

    I setup a new script that will run every 1-3 hours (TBD) that will rsync the ramdisk back to appdata so that I won't loose data in those dockers (at least more then a few hours worth).

     

    The only issue is that it has to copy the whole database back into appdata everytime it rsyncs, between them all that works out to around ~150mb per sync.

     

    So still around 2gb/day in writes doing it every 2 hours but it should have much less write amplification this way so only way to know the effects is to try it.

     

    It also requires 4gb of ram for the ramdisk and I have 8gb allocated just in case, fine for me but would not be so practical on a system with limited memory.

     

    Gonna leave this for a day or 2 and see the results.

     

    If only there was a way to selectively change the cache on a per folder or drive basis.


    Hahahah you went down the rabbit hole on this one didn’t you.

    I’ve spent a fair amount of time of this as well, but at some point thought those few gigabytes weren’t worth my time (also became a father in the meantime so less time to play 🤣👊🏻).

     

    Having that said, aren’t you better off writing a guide to cut down on these writes and post them in a separate thread? This thread has become so big that it will scare off anyone accidentally coming here via Google or something. Would be a waste of knowledge that you have gained (that others could benefit from) if you ask me.

     

    Cheers 🤙🏻

    Link to comment
    24 minutes ago, TexasUnraid said:

    The constant updates and logging that can not be disabled, adjusted or tweaked in any way causes quite a bit of writes unnecessarily

    Are they writing to random folders or why isn't it possible to tweak?

    Link to comment
    1 minute ago, S1dney said:


    Hahahah you went down the rabbit hole on this one didn’t you.

    I’ve spent a fair amount of time of this as well, but at some point thought those few gigabytes weren’t worth my time (also became a father in the meantime so less time to play 🤣👊🏻).

     

    Having that said, aren’t you better off writing a guide to cut down on these writes and post them in a separate thread? This thread has become so big that it will scare off anyone accidentally coming here via Google or something. Would be a waste of knowledge that you have gained (that others could benefit from) if you ask me.

     

    Cheers 🤙🏻

     

    lol, yeah I tend to do that from time to time. Plus seeing the writes slowly add up and the remaining life slowly tick down on the SSD has been an itch I needed to scratch for some time.

     

    Figure I will see what is possible under the best case setup and then back off to comfortable compromise after that.

     

    For example the above docker side commands are well worth the effort IMHO as they "just work" once you know what containers need which command and have a big effect on writes.

     

    Some of the appdata writes are pretty simple fixes as well such as reducing/disabling logging or moving logs to an internal ramdrive instead of writing to appdata etc.

     

    My goal is to create another thread with a guide once I finish my testing. For now these posts are basically my journal on the matter for later reference.

    • Like 2
    Link to comment
    2 minutes ago, mgutt said:

    Are they writing to random folders or why isn't it possible to tweak?

     

    No, just writing to the log files and database and they offer no option to change the updates or logging in any way (well except to log even more). Others have complained about this as well but the devs have made it clear they have no intention of changing this for some reason.

     

    The writes are darn near constant as well, just flooding the activity logs.

    Link to comment
    54 minutes ago, TexasUnraid said:

    just writing to the log files

    And why isn't it possible to move them to the RAM with --mount type=tmpfs,destination=/logs or similar?

    Link to comment
    2 minutes ago, mgutt said:

    And why isn't it possible to move them to the RAM with --mount type=tmpfs,destination=/logs or similar?

     

    The log files themselves I possibly could but they are stored in the same folder with the database and the database folder is mapped to appdata. So can't really map it to a ramdrive inside the container as it will still be written to appdata.

     

    The logs are actually written to less then the database it seems as well. The database writes flood the activity logs.

     

    Only way to deal with that would be for the devs to allow changing the update timing or do what I am doing with a ramdisk in appdata.

     

    Speaking of a ramdisk on unraid, will it automatically unmount it during shutdown? I just rebooted and it hung during shutdown with a kernal panic? Could that be due to the ramdisk?

    Link to comment
    51 minutes ago, TexasUnraid said:

    database

    So it's a file based database like SQLite? Any support for external databases like MySQL?

    Link to comment
    5 minutes ago, mgutt said:

    So it's a file based database like SQLite? Any support for external databases like MySQL?

     

    Yeah I think so, not real sure on the technical side of it but pretty sure it is limited to the internal database only. I could not find any options to even change the path to the logs or database much less anything more.

    Link to comment

    Well, simply moving the *arr's and qbittorrent to the ramdisk has cut the appdata writes to a small fraction based on the activity logs. Have to wait until it syncs a few times to see what effects it has but the first sync happens and it was still a small fraction of the writes the log used to show.

     

    Looks like each sync will be around ~350mb x every 2 hours = ~4gb/day of writes.

     

    Real curious to see what the effects are in the real world LBA writes with write amplifcation factored in etc but have to wait until tomorrow night to get the report.

    Edited by TexasUnraid
    Link to comment

    So did some testing of running a script at array stop to sync the ramdisk and then unmount it but in testing it is run 1 second before unraid starts unmounting the drives and thus would not be able to sync.

     

    Is there another way to run a script during shutdown that would either start sooner or cause the rest of the shutdown to wait until it is finished?

     

    I saw in the logs a unmountscript.sh but could not find it on the USB drive.

    Link to comment

    Another idea. Are you using "powertop --auto-tune"? If yes, it reduces "vm.dirty_expire_centisecs" to "1500" = 15 seconds. You can check your settings as follows:

    sysctl -a | grep dirty

     

    The default value is "3000". You could raise it to "12000" (2 minutes). By that all writes to the disks are cached for 2 minutes in the RAM:

    sysctl vm.dirty_expire_centisecs=12000

     

    Of course you should have an UPS to avoid data loss through power outage.

     

    I'm using "6000" and raised vm.dirty_ratio to "50", so writes are using 50% of my free RAM.

    Link to comment
    4 hours ago, mgutt said:

    Another idea. Are you using "powertop --auto-tune"? If yes, it reduces "vm.dirty_expire_centisecs" to "1500" = 15 seconds. You can check your settings as follows:

    
    sysctl -a | grep dirty

     

    The default value is "3000". You could raise it to "12000" (2 minutes). By that all writes to the disks are cached for 2 minutes in the RAM:

    
    sysctl vm.dirty_expire_centisecs=12000

     

    Of course you should have an UPS to avoid data loss through power outage.

     

    I'm using "6000" and raised vm.dirty_ratio to "50", so writes are using 50% of my free RAM.

     

    I have no idea what powertop is?

     

    I have actually been manually setting my vm.dirty_expire_centisecs=12000 aleady for some time, it did help with writes for sure but when watching things on netdata noticed that it doesn't actually cache for a full 2 mins, there was something flushing the dirty writes every 30 seconds. Thus I didn't see reductions in writes past a certain point.

     

    So I ended up increasing everything having to do with dirty writes and it did finally increase the time (and reduce writes) but for some reason the system will lock up when I did that until writes were flushed at times so had to revert back. I don't understand all of those commands well enough right now.

     

    Same thing for the dirty ratio, if I increase it much (I have 256gb of ram) the system will completely lock up when flushing dirty writes for minutes at a time. So far the sweet spot seems to be around 2-5%.

     

    I do have a UPS and have no issues with caching things for 2-3 mins, in fact that would be ideal if it would not lock up the system when I try to do this.

     

    In other news, my activity log is only a few mb overnight vs ~50mb before. Most of the activity are the rsync's as well. For the first time ever my docker SSD is actually sitting idle for extended periods of time. Real curious what the writes will be tonight.

    Link to comment

     

    5 minutes ago, TexasUnraid said:

    but for some reason the system will lock up when I did that until writes were flushed at times so had to revert back.

    Did you flush it manually?! Linux is caching all the time, so it would be strange that it causes a lock up.

    Link to comment
    2 minutes ago, mgutt said:

     

    Did you flush it manually?! Linux is caching all the time, so it would be strange that it causes a lock up.

     

    Nope, not flushing everything, just the dirty writes. In netdata you can see the dirty writes and when they are flushed to disk and if it goes above a certain threshold it seems to lock up the entire system. Not sure if it is size or time based but think it is time based.

     

    Aka, if the flush takes too long, the system hangs waiting for it to finish. But this is not always the case, it would also lock up with only a few mb in the dirty writes until it was flushed.

     

    No idea what was going on but for whatever reason it is really sensitive to caching settings outside the defaults for some reason which is sad.

    Link to comment

    Um, wow.

     

    So with the latest changes with the appdata ramdisk, my writes for the last day was a mere 8gb?!?

     

    Now the rsync cron didn''t work for the first few hours for some reason so changed it to hourly for the last ~13 hours, so the real writes every 2 hours would most likely be a bit more but honestly might just stick with every hour if it is only ~16gb of writes.

     

    So to recap:

     

    BTRFS image > XFS SSD everything stock =

    ~25GB/day

     

    BTRFS image > BTRFS SSD everything stock =

    75-85GB/day

     

    Docker Folder > BTRFS SSD everything stock =

    ~60gb/day

     

    BTRFS image > BTRFS SSD > Disabled the low hanging fruit docker json logs =

    ~48gb/day

     

    BTRFS image > BTRFS SSD > Disabled all misbehaving docker json logs for running containers except those I want to see + added ramdrives to /tmp in containers that do internal logging =

    ~30gb/day

     

    BTRFS image > BTRFS SSD > Disabled all misbehaving docker json logs for running containers except those I want to see + added ramdrives to /tmp in containers that do internal logging + moved appdata for the *arr's and qbittorrent to a ramdisk with ~bi-hourly rsyncs to the ssd appdata =

    ~8-12gb/day

     

    Since most of the writes are large writes from the rsync, there is very little write amplification which vastly improves the total writes from the day even though that is possibly more raw data being written to the drive.

    Edited by TexasUnraid
    • Like 1
    Link to comment

    Overall disabling the docker logs and adding ramdisks inside the containers are recommended as it is a basically universal fix and pretty simple once you track down the misbehaving containers.

     

    Adding an appdata ramdisk and dealing with this on the other hand takes more legwork. That said someone that knows how the plugin system works should be able to create a plugin that would handle this automatically pretty easily.

     

    The actual setup and commands are pretty simple, I will post them once I have it figured out.

     

    The mounting and setting up of the ramdisk I have figured out, what I am having a harder time with is unmounting it during shutdown. I can't figure out a way to run a script during shutdown.

     

    I did some searching and saw a few people say that a plugin has the ability to do this or use the user script option but that doesn't run soon enough to work.

    Link to comment
    24 minutes ago, TexasUnraid said:

    I can't figure out a way to run a script during shutdown

    Just like there is a ‘go’ file that is run as part of the boot sequence, you can also have a ‘stop’ file that is run as part of the shutdown sequence.    Would that do what you want?

    Link to comment
    5 minutes ago, itimpi said:

    Just like there is a ‘go’ file that is run as part of the boot sequence, you can also have a ‘stop’ file that is run as part of the shutdown sequence.    Would that do what you want?

     

    Yes, that should work just fine if it can complete before unraid proceeds with the normal shutdown.

     

    I was actually just digging through google results and didn't see a single mention of a stop file but did find the rc.local_shutdown file and noticed that it called a stop script but could not figure out what that was.

     

    So basically the stop script is called before the "main shutdown"? So if I was to rsync and then unmount/remove the ramdisk, this would complete before unraid starting shutting things down?

     

    The rc.local_shutdown doesn't seem to actually be the shutdown script for unraid, just the fallback it uses if it has to force a shutdown? So not sure how everything interacts.

     

    Basically it just needs to run a rsync and then umount the ramdisk and after that is finished proceed as normal.

    Link to comment
    9 minutes ago, TexasUnraid said:

    Yes, that should work just fine if it can complete before unraid proceeds with the normal shutdown.

    The stop file running is part of the normal UnRaid shutdown sequence.   I think it is run quite near the end after all the array and services have been stopped but I could check that if you need to know exactly when it runs.
     

    it is also quite easy to run scripts on any of the UnRaid internal events if this would be more desirable.   I think I have some documentation on how to achieve this.

    Link to comment
    2 hours ago, itimpi said:

    The stop file running is part of the normal UnRaid shutdown sequence.   I think it is run quite near the end after all the array and services have been stopped but I could check that if you need to know exactly when it runs.
     

    it is also quite easy to run scripts on any of the UnRaid internal events if this would be more desirable.   I think I have some documentation on how to achieve this.

     

    I just ran a test to try to see when it was called and it looks like it will work perfect!

     

    I just ran a simple loop to print the date into the syslog and it started basically first thing when I clicked shutdown in the GUI and it waited until it was finished to continue the shutdown procedure!

     

    Exactly what I was looking for!

     

    Found I can runthis command to shutdown all the docker containers before doing the final rsync so they can close the files properly.

    docker stop $(docker ps -q)

     

     

    The more I look at this, the more I think it could be turned into a plugin pretty easily from someone that knows how the plugin system works. The commands are very simple and the risks minimal for most use cases (you loose an hours worth of data from sonarr/plex, big deal).

     

    Create ramdisk of user selected size (it could calculate the size of the appdata folders to let the user know the minimum size as well)

    select docker containers / appdata folders to move to ramdisk

    Rsync them to the ramdisk

    update docker files for said containers to point to ramdisk.

    Rsync the ramdisk data back to disk at user selected interval

    rsync and umount ramdisk at shutdown

     

    It could also have the ability to disable docker logging on selected containers / add ramdrive to /tmp in them using the above commands I posted as well. A 1 stop shop to prevent excessive writing since this issues does not appear to be going away.

    Edited by TexasUnraid
    • Thanks 1
    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.