February 21, 20179 yr Quick background: I have two identical (with relation to hardware) UnRAID servers that mirror each other. I access them both over pooled NFS shares (MergerFS) to my Linux servers running docker. My Linux servers/dockers use the MergerFS mounts using the FF (first found) policy so basically that means my UNRAID01 shares are used all the time unless they're down for whatever reason, at which point the UNRAID02 shares (identical) take over. Issue: Starting this past Friday, my NZBGet docker started showing all downloads failing. I would get one of two errors. Either "Unrar error code: 10" (unpack failed) or "failure to read files in /downloads/queue/..." The "failure to read files /downloads/queue..." error messages (tons of them) would subsequently cause all my user shares to disappear and be inaccessible until a full server reboot. Though as we stand right now, the only error I seem to be getting is the Unrar unpack failure. I investigated over the weekend thinking this was an NZBGet issue but I was skeptical of that from the beginning since I've made no changes to that config in over a year. What helped me isolate the issue to UNRAID01 was that if I forced my NZBGet docker to use the UNRAID02 Downloads share instead of UNRAID01, it works without issue. Below you will find diagnostics taken shortly after getting the "failure to read files in /downloads/queue/..." error which caused all my user shares to disappear. For reference, both my servers' cache pools are configured as follows: 2 x Intel 730 480GB SSDs in RAID0 (so 960GB usable) Downloads share is cache only and looks as follows: spe-unraid-diagnostics-20170220-0918.zip If anyone has suggestions for how to further troubleshoot/isolate this issue I'm all ears. The only thing I've tried is NewPerms to no avail. Edited February 21, 20179 yr by IamSpartacus
February 21, 20179 yr I'd like to help but I don't use MergerFS or NZBGet. If you can eliminate those two parts of the problem and it still happens it's possible I or someone might be able to make a suggestion. In other words, you need to break the problem into smaller ones.
February 21, 20179 yr Author 16 minutes ago, John_M said: I'd like to help but I don't use MergerFS or NZBGet. If you can eliminate those two parts of the problem and it still happens it's possible I or someone might be able to make a suggestion. In other words, you need to break the problem into smaller ones. I have eliminated MergerFS and currently just have the single shares mounted. If I use the UnRAID01 shares I have this issue, UnRAID02 shares it works fine. I can't eliminate NZBGet because thats where the problem is seen and is reproducible right now.
February 21, 20179 yr "Failure to read files" suggests either a permissions problem or some other utility has moved/deleted/renamed the files in question.
February 21, 20179 yr 11 minutes ago, John_M said: "Failure to read files" suggests either a permissions problem or some other utility has moved/deleted/renamed the files in question. I second that
February 21, 20179 yr Author 33 minutes ago, John_M said: "Failure to read files" suggests either a permissions problem or some other utility has moved/deleted/renamed the files in question. 22 minutes ago, ijuarez said: I second that I thought so myself which is why I did NewPerms as well as watched the files being created in the location that was trying to be accessed. Basically as NZBGet is trying to unpack a download it creates all these queue files and for whatever reason it was spitting errors trying to read the files it had just created. However, that's not my focus right now because currently the only error I'm getting is the Unrar Error Code: 10. I know this is not an NZBGet error because if I change my docker volume mount from /UNRAID01/mnt/user/downloads:/downloads to /UNRAID02/mnt/user/downloads:/downloads it works fine with no changes/edits on the NZBGet config file. I'm going to have to examine my shares/permissions more closely between the two servers so I can try to find any other possible difference. I just don't get what could have caused these differences.
February 21, 20179 yr I Googled "unrar error code 10" and the first hit was this. I don't know if it helps.
February 21, 20179 yr Author 42 minutes ago, John_M said: I Googled "unrar error code 10" and the first hit was this. I don't know if it helps. Yup, saw that and as I said in my OP I investigated this as being an NZBGet issue for 2 days as that was certainly my first thought. However that issue seems to have been patched in later versions of NZBGet which I'm running the latest of (18.0). Remember, it works fine when using the Downloads shared off UNRAID02 with the EXACT same NZBGet container and config. Nothing changes on the NZBGet side of things, only the volume mapping from UNRAID01 to UNRAID02 changes. Edited February 21, 20179 yr by IamSpartacus
February 21, 20179 yr I'm afraid there was so much in your OP that didn't mean anything to me I must have missed that. You say that the two servers are configured exactly the same but logically that can't be true if they're behaving differently. There will be some subtle difference that you need to find. Good luck.
February 21, 20179 yr Author It's definitely permissions related. I compared the permissions on the Downloads shares which looked like this: I then did a NewePerms on the Downloads share on UnRAID02 and now I'm getting the same issue when using both shares.
February 21, 20179 yr Author More strangeness. So I ran the following commands on UNRAID02: chown -R 1000:1000 /mnt/user/Downloads/tmp chown -R 1000:1000 /mnt/user/Downloads/queue chown -R 1000:1000 /mnt/user/Downloads/nzb This changed the permissions to 1000:1000 as you'd expect and now the Downloads again work. However, when I ran those same commands on UNRAID01, it changes the permissions to spe 1000 instead of 1000 1000. SPE is the user account on my other Linux boxes that is running all the dockers. I'm totally perplexed why I'm getting such different behavior.
February 21, 20179 yr Author Through all the strangeness, simply deleting the Downloads share and re-creating it fixed the issue. Wish I had tried that a lot earlier. At least my issue is resolved.
February 22, 20179 yr It looks as though you've got something running as UID 1000 GID 1000 when it ought to be UID 99 GID 100 (i.e. nobody:users) instead.
February 22, 20179 yr Author 3 hours ago, John_M said: It looks as though you've got something running as UID 1000 GID 1000 when it ought to be UID 99 GID 100 (i.e. nobody:users) instead. Yea UID/GID 1000 is linked to the user "spe" on my other Linux servers that run my dockers. It turns out I also had created a user "spe" on UNRAID01 and it was causing some kind of conflict. I removed that user account from UNRAID01 and recreated the Downloads share and I'm up and running. Thanks for helping me bounce ideas back and forth. 2 hours ago, ijuarez said: Awesome, never heard of mergefs, going to check it out. MergerFS is great and very easy to setup. Let me know if you have any questions about it.
Archived
This topic is now archived and is closed to further replies.