Shares disappear daily

[tateburns]

Greetings,

 

I really could use some help diagnoising what my problem might be. I'm running Duplicacy on my Mac to backup data to my unRAID over SFTP. I work in video production so these datasets I'm backing up are fairly large in size, or a least one of them. 3 volumes ony my Mac, 41TB (projects archive), 2.5TB (current editing project), and 500GB (Macintosh HD). The backup jobs run hourly. Duplicacy performs separate check and prune jobs nightly.

 

It used to take about 7-10 days for unRAID shares to disppear from underneath /mnt/user. At a certian point during the week (hard to identify when) they'd vanish in both the GUI and when logged in via the terminal. See output below for ls -l on /mnt/

d????????? ? ?      ?       ?            ? user/
 

A reboot would bring the shares back and I'd be good for another 7-10 days. Now the shares are disappearing nightly. I'm unable to make it through a full day of backup schedules. Based on the failure logs from Duplicacy, I'm guessing it is either the check or prune jobs that are killing them. My guess is that since the dataset has grown, the length of time it'd take for these jobs to kill off my shares has decreased. I had seen folks complaining that certain large rm commands could kill their shares, so maybe the prune is doing it?

 

Here's what I've done to attempt to troubleshoot:

 

Checkdisk on all disks = Clean

Memtest (5 passes) = Clean

NFS = Disabled

Docker = Disabled

Cache disk = Not in use for any shares hosting active datasets (I still have appdata on it from when I had a docker running)

Upgraded unRAID = 6.11.5

Plugins = All upgraded, very few installed anyways

 

I've done a lot of research on this issue and those items above seem to represent the most popular things to check. At this point I'm at a complete loss how to proceed. I've been an unRAID user for about 12+ years, have some other non-critical path servers for hosting various media. I'm thinking about moving to TrueNAS for this system, but really don't have the time to learn a new platform right now. I just need something stable and really don't need any features whatsoever. Just a dumping ground for data.

 

Could someone take a look at my diagnostic log and let me know if there's something obvious I'm missing? I just output the logs right now while shares are gone. You'll probably see consistent errors about my UPS losing connection. I just haven't hooked up the cable to my UPS since I lost it when we moved recently. Thank you!

 

dagobah-diagnostics-20230225-0847.zip

Edited February 25, 2023 by tateburns

[Gragorg]

Its a long shot but when I had this happen is was corruption on my USB.  Put your USB in a windows PC and see if it detects any errors.  Let windows repair it finds any.

[tateburns]

Wow that's a path I hadn't considered. I really appreciate the creative idea. I'll give it a shot and report back!

 

Some other things if anyone takes a peek at my diagnostic logs. I'm logging into my unRAID from Duplicacy for the SFTP backups using the unRAID root account. Don't SecOps shame me too hard 😞

[Squid]

Feb 24 21:59:41 Dagobah kernel: Out of memory: Killed process 6278 (shfs) 

Any clue what was happening in and around that time?

[tateburns]

Mounted boot thumb drive on Windows. No errors. Was able to ready it fine.

 

Regarding out of memory warnings, you've got me on the trail of something. Like said, I didn't have many plugins installed beyond the Dynamix ones. However, I did have Dynamix Cache Dirs installed. I'm wondering if it's been running out of memory when I'm running my Duplicacy check/prune jobs from my Mac nightly? It seems improbable that I could fill up 8GB of memory with lists of files, but who knows? 

 

I've taken a very unscientific and scorched earth approach here and removed every single plugin, including the Dynamix Cache Dirs one. At this point I am only running the bare unRAID OS. I've fired up the jobs that I think might be killing the shares to see if it makes a difference. Thanks everyone for the suggestions so far.

[tateburns]

I just wanted to updated I've been running my Duplicacy backups from my Mac -> to unRAID over SFTP for the past week with no issues since I removed all plugins. Again, an unscientific approach so I don't know which was causing it. If I had to guess though it was the Dynamic Cache Dirs one