Shonky

Ok well at risk of bursting your bubble, that's how mine is setup anyway and I still had the problem. I have a /24 LAN. 1-99 are static which I just assign manually. Some are dockers some are things like routers/printers. DHCP is 100+. That's the way it's always been and kind of has to be really. If you have static IPs in the middle of a DHCP server's range you're going to have a bad time (tm) at some point. Putting dockers above or below the DHCP range makes no difference.

I have recently implemented hashbackup to replace CrashplanPRO but believe that hashbackup has just made the problem more visible. I had seen Crashplan backing up files more than once but never investigated why mainly because the size wasn't affecting me. Anyway, I'm seeing files that have not been changed in years having their modified/changed times altered. e.g. I do a hashbackup, then run it again a few hours later and something's modified some files e.g. a particularly random one just now: ~# stat "/mnt/user/store/manuals and drivers/A8N-SLI Premium Drivers/15.23_nforce_winxp32_international_whql.exe" File: /mnt/user/store/manuals and drivers/A8N-SLI Premium Drivers/15.23_nforce_winxp32_international_whql.exe Size: 53370976 Blocks: 104248 IO Block: 4096 regular file Device: 2fh/47d Inode: 648799821402998927 Links: 1 Access: (0666/-rw-rw-rw-) Uid: ( 99/ nobody) Gid: ( 100/ users) Access: 2017-04-04 06:21:53.417766486 +1000 Modify: 2008-10-05 09:03:49.000000000 +1000 Change: 2021-10-10 17:53:26.528479637 +1000 Birth: - There's no way I touched it just now. That's about 20 minutes ago. Whole chunks (not every file in a folder though) randomly have their change time altered and the backup program picks it up. Could the file integrity plugin do anything like this? I've set it up recently but pretty sure the repeated backups were happening long before that. If a file is not being actually changed how else could this be happening? All disk file systems are xfs. Not entirely sure what else would be relevant. Nothing in syslog at the change time. Checking the direct /mnt/diskx/... file shows the same info but with the birth field entered (seems just before the access time)

How's your fix holding up? I get this sporadically. certainly not every 3-4 days. 1-3 months perhaps. I do not have the mentioned switch. I have a static IP but don't use VLANs within unRAID nor on my network. I don't really follow the fix though. I do have a br0 setup and some of my dockers connect that way to get a specific LAN IP of their own. My docker br0 is IPv4 custom network on interface br0: Subnet: 192.168.99.0/24 Gateway: 192.168.99.1 DHCP pool: not set That's my LAN too. Is the suggested fix to have docker restricted to a sub range of the full /24 and then for each docker that needs it, only use IPs within that range? My DHCP range for my LAN is 192.168.99.100-192.168.99.255. Below .100 I reserve for static assigned IPs and that's where my dockers that have their own IP run from. @corgan if you're still getting the problem does that not mean you didn't really fix it though?

Jinxed it. Caught it happening again tonight so none of the above things fixed anything. I only touched two dockers to bring it back around. Gitea and Photoprism. The latter did seem to be scanning. I'm only playing with it so killed it pretty quickly. Previously it did seem like CrashPlanPRO was a cause, but not always. Seems like dockers that generate a lot of IO (Plex scanning, Photoprism scanning, CrashPlanPRO indexing/scanning etc ) tend to trigger it off. Stop one or two of them or even others sometimes and it comes back around. Almost like it's disk thrashing on a mechanical disk.

@DingHo Now that you say that, I do also run plex and had some issues with the scheduled scan failing/timing out/something on a couple of random files. I think they were video files and I ended up removing them. If it was those files specifically, it wasn't 100% repeatable because it would have happened every night. I don't remember if if I was looking at the Plex issue because of the loop2 reads or independently. Sorry that's not really helpful. If you turn on debug in plex it's pretty easy to find the specific files causing the problem. And you can trigger the full scan manually. The scan runs but is killed after an hour from memory if it's taking too long.

I've not made any definite change that fixed it but I haven't had a re-occurrence for a while now. Running 6.9.2 but I'm pretty certain that I was still getting it with 6.9.2 also. Not that 6.9.2 had changed expected to fix this. I think the most significant thing I did change was I moved my cache drive SSD SATA connection from the HP Microserver ( Gen 8 ) motherboard and used the PCIe SATA card I had installed. The main reason I did was that it was limited to SATA II speeds. I can't say 100% but at least since then I've not had an issue. The other thing I had done was that I changed the SSD itself which meant recreating the whole docker.img filesystem. No improvement from that change.

buff/cache is RAM allocated to buffers by the OS. It is not allocated RAM and is available if something tries to allocate RAM. I doubt there is any significant difference in the buff/cache handling in unRAID. It's still pretty close to a regular Linux kernel. If unRAID was filling its ram disk (and you didn't have enough), you'll start seeing OOM errors. My system shows / as 4.8GB total with about 900MB used. I don't have a netdata docker and haven't tried it either.

I don't think it's your unhealthy container, but have nothing to back that. If I catch mine in the early stages, things like docker ps, docker stop/start etc work ok. Usually by the time I catch it though, it's because something like Plex or Pi-Hole dockers are being blocked from doing anything. It's too late then and docker commands just get blocked until it comes back. So I can't properly shut down a docker and have to kill some processes that docker is running. I've toyed with just as a work around, something monitoring system load. Any time it gets over say 20 or 30 even on 1 minute and restart a docker perhaps after capturing some logs. Whilst CrashPlanPRO seems to be involved I have had occasions where restarting other very lightweight dockers was enough. Unfortunately mine does it even less than it used to now.

I am fairly certain it wasn't RAM related for me. I have 10GB and whilst I run quite a few dockers I don't think I'm hitting any memory issues. Looking at my screenshot of top here, shows ~2GB cached so not running out of memory. If that were an issue you should start seeing oom logs. In the general process of things I have replaced my cache drive (so new filesystem on it and then re-created the docker.img which is loop2 so new filesystem there too). I also moved if from the SATA II only internal port to a spare SATA III port I had on an add in card. It does appear to be happening less, but I had another instance the other day so that certainly hasn't resolved it. My CPUs (4C/8T Xeon E3) weren't getting pegged, but there was a lot of iowaiting happening.

Upgraded to 6.9.1 from 6.8.3. Some messages now seem to report as "devX" for the disks rather than "sdX". Why was this changed? e.g. old warning Event: unRAID device sdi SMART health [187] Subject: Warning [MARS] - reported uncorrect is 117 Description: ST2000DL003-9VT166_5YD2SZRE (sdi) Importance: warning New warning Event: Unraid device dev1 SMART health [187] Subject: Warning [MARS] - reported uncorrect is 117 Description: ST2000DL003-9VT166_5YD2SZRE (dev1) Importance: warning Yes I can tell from the serial number but what is dev1, dev2 etc? Some messages still use sdX Event: Unraid Disk 5 message Subject: Notice [MARS] - Disk 5 returned to normal utilization level Description: WDC_WD40EZRX-00SPEB0_WD-WCC4E0262246 (sdg) Importance: normal Ah I see "Dev 1" and "Dev 2" as the the first column of unassigned devices on the Main page. Then I guess the question becomes, why doesn't the space message use "Disk 1" etc. Not consistent.

I am still running 6.8.3 and have this issue, so I don't think this is related to 6.9.x. I do run Plex but it's not a transcoding issue. Are you running Crashplan? I don't believe Crashplan is specifically the problem, but perhaps somehow causes it more often. At the moment my system is doing it within a few hours of starting the Crashplan docker.

I'm getting a wierd double up Crashplan Docker. It names it "0" too. I definitely don't have two dockers and there's no old containers. Also strangely it's supplying the gotify icon for the image. I do have the gotify docker running too. Any ideas?

I'm pretty certain Plex isn't the cause in my issue linked above. I re-created my docker.img btrfs with no effect. /dev/loop2 is the docker image that lives on the cache drive (usually). So it's not just used for Plex - but for any/all dockers you may have running. Note that the issue we are seeing is lots of *reads* (this thread and mine linked above), not writes on the /dev/loop2 device.

Ah hadn't found much on this last I searched. I only just got around to posting my issue but I've been seeing it for quite a while now (>6 months). Sounds exactly like your issue:

Yeah I think I'd found that, but /tmp would have been fine. The log (and the workaround) shows that actually it's creating it in the wrong place. That would appear to be a Crashplan bug. It shouldn't be creating temporary files where its binaries etc are stored. That's precisely what /tmp is for. Code42 probably won't care about a docker installation, but I would expect this to potentially cause problems in other systems too as unprivileged users may also not be able to write there. Perhaps there's an unset configuration or environment variable it looks for that's missing as a docker?

Apologies if this is covered, there seems to be no way to search in an individual thread. My install has been working fine and I did a restore back in January, but it's now giving me problems. When I start the restore it gets stuck on "Preparing files..." in the GUI. The service.log.0 gives a somewhat helpful error message about being unable to create a pipe [08.17.20 14:21:41.294 WARN background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Failed to create pipe for restore tool connection. userName=app, error=com.code42.jna.PlatformException: Problem in mkfifo() for path=/usr/local/crashplan/./restore-pipe-967861654629759799-request, mode=384, errno=13 [08.17.20 14:21:41.295 INFO background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Not sending request; no pipe available [08.17.20 14:24:09.070 INFO DefaultGroup .code42.messaging.peer.PeerGroup] PG::DefaultGroup DONE Managing connected remote peers. numConnected=4, numFailedConnectedCheck=0, duration(ms)=1 [08.17.20 14:24:11.296 WARN background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Failed to create pipe for restore tool connection. userName=app, error=com.code42.jna.PlatformException: Problem in mkfifo() for path=/usr/local/crashplan/./restore-pipe-967861906287999799-request, mode=384, errno=13 [08.17.20 14:24:11.297 INFO background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Not sending request; no pipe available [08.17.20 14:26:41.298 WARN background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Failed to create pipe for restore tool connection. userName=app, error=com.code42.jna.PlatformException: Problem in mkfifo() for path=/usr/local/crashplan/./restore-pipe-967862157946239799-request, mode=384, errno=13 [08.17.20 14:26:41.298 INFO background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Not sending request; no pipe available [08.17.20 14:29:09.072 INFO DefaultGroup .code42.messaging.peer.PeerGroup] PG::DefaultGroup DONE Managing connected remote peers. numConnected=4, numFailedConnectedCheck=0, duration(ms)=1 [08.17.20 14:29:11.300 WARN background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Failed to create pipe for restore tool connection. userName=app, error=com.code42.jna.PlatformException: Problem in mkfifo() for path=/usr/local/crashplan/./restore-pipe-967862409621257015-request, mode=384, errno=13 [08.17.20 14:29:11.300 INFO background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Not sending request; no pipe available [08.17.20 14:31:41.301 WARN background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Failed to create pipe for restore tool connection. userName=app, error=com.code42.jna.PlatformException: Problem in mkfifo() for path=/usr/local/crashplan/./restore-pipe-967862661279497015-request, mode=384, errno=13 [08.17.20 14:31:41.302 INFO background-0 e.restoretool.RestoreToolService] RESTORE TOOL:: Not sending request; no pipe available I did recently recreate my docker.img so set up the dockers all over (using the same application data folder) but think it may be more likely related to the 8.2.2 version? Is it trying to create this pipe file in the docker image and that's the cause of the problem? I did find a reference to perhaps running the service as root but of course that has some security issues. Is anyone else able to restore with the latest version just to rule that out? I see /usr/local/crashplan in the docker is 755 permissions owned by root:root so can see a non root user won't be able to write. I did a quick chmod 777 on /usr/local/crashplan and sure enough the queued up restore which had been failing, then successfully went through. So the question is, what is the correct way to fix this? Thanks

Must have been thinking compression. Still worth trying. Is CPU usage high?

Backup tab and then Advanced Settings. And yeah nah on backing up terabytes and terabytes of TV

Have you tried just outright disabling deduplication (temporarily) ? At least then you'll know if that's the problem or not. And I have to ask, is it really necessary to back up TV shows? Seems somewhat extreme.

Well upgrading it, broke it initially. It seems the Crashplan Desktop needed a restart after Crashplan(?). Initially the IP was still 0.0.0.0 in .ui_info. A second restart of Crashplan Desktop and/or Crashplan seemed to update that automatically to 172.17.42.1 and now it's working. Wasn't seamless but fixed itself.

Thanks for fixing gfjardim and ljm42. Desktop worked straight away for me on my existing Crashplan container that had self upgraded. Regarding the main application, given that it has automatically updated itself anyway, what does updating docker actually do? I'm a bit wary of upgrading it given it's already running the latest version and working fine. I had some issues when moving over from v5 with a plugin to v6 and docker that I don't wish to revisit

I have found that it just takes longer (sometimes significantly), but does eventually sync both ways. Have you tried just using very small text files to test? The delay for up-sync means that I wouldn't do any siginificant changes on UNRAID - don't know if this is the docker, or dropbox client related. Mine is doing this too. I have two machines (ignoring others but they should not make a difference), my Windows 7 laptop and my unRAID (and the Dropbox cloud I guess). Anything I put on the Windows 7 machine syncs fine up to Dropbox and back down to the NAS. No problem there. Anything I put on the unRAID machine, only syncs if it's at the top level or one down i.e. Dropbox/* or Dropbox/somefolder/* are OK but Dropbox/somefolder/anotherfolder/* does not sync. If Dropbox restarts, the deeper files *might* sync from unRAID (it takes a long time to restart). I seem to get a lot of conflicts possibly from this running portable apps from my Dropbox I have about 80000 files synced for about 25GB. The docker container reports /proc/sys/fs/inotify/max_user_watches as 524288 which should be plenty. I had similar issues back on unRAID v5 with a plugin. Does everyone else have Dropbox syncing up from unRAID within a few folders?

Not quite following that last post but I think saying the same thing as me. I found that the automatic updates seemed to get "stuck" sometimes, so manually restarting the Docker and just monitoring the log files for the CPVER line was enough to push it up to 4.4.1. If it seemed to stop, just restart it. .ui_info seems to the be only file that needs to be change on a Windows frontend I found. .ui_properties seems to be ignored as does ui_<user name>.properties. Just set the local port to whatever you are tunneling through to the unRAID machine along with the GUID and you should be good to go. One thing I did find was that if I changed any of the mapped paths, it seemed to reset the container's version of CrashPlan back to 4.3.0 and it had to go through the stepped upgrade process again. Not sure if that's expected or not. Otherwise thanks. It's now working after mulitple false starts trying to get it to adopt my old install on unRAID v5. Also, should the default path be updated? Currently it's /mnt/sdd/appdata/crashplan whereas other dockers generally use /mnt/user/appdata/xxxxx. Minor annoyance but something I had to do multiple times since I was removing and re-installing a few times whilst I attempted to get it to work.