I have two HDDs formatted as ZFS in my array. Reason: to take advantage of file compression, because I have a large collection of files which do benefit from 1.25:1 to 1.3:1 compression. There is one share using those two disks only, on which this collection of data resides

While cleaning up this large collection of files and folders, I move a lot of things around from the server to my main PC, clean them up (extract files from archives, rename directories and so on), via a 10g LAN connection. Nothing new, I have done that before successfully.

 

The share works, everything is fine, except for one annoying issue.

 

More often than not, moving files does NOT delete the original files from the share. For example, I have a folder with, say, 50 files. I access it remotely, via Total Commander, I select them, press F6 to move them all to my PC.
Once the process ends, the source still contains, say, 41 files. I select them, try to delete them, 26 files remain. Repeat, now there's 12 files left. Maybe a final pass will sometimes get rid of them all, sometimes there's still a couple files remaining.

The destination folder contains all moved files, but the source does not empty. Rarely, it does, but that's more of an exception than a rule.

 

Troubleshooting:

 

1. I checked whether this behavior can be replicated for other disks (XFS, btrfs😞 it cannot. This is specific to ZFS only.
2. I have bought more RAM, thinking maybe the ARC cache was too small. I am now up from 32 GB to 128 GB RAM, and an ARC cache of 64 GB RAM. Issue persists.
3. I tried using other file managers: TeraCopy (issue persists), Krusader docker on Unraid (issue persists, even when moving files using two steps: copy, then delete)
4. Interestingly, I tried a couple small batches of file moving using Unraid's Terminal, and the issue did not appear there, but it might have been a fluke.
5. This issue has always occurred, since day 1 of using ZFS.

 

This is particularly annoying, because everything else works perfectly, and I have started to second-guess any file operation I perform on ZFS drives: "Are the files truly gone? Do I have duplicates? Have I moved all archives to my main PC?"

 

Diagnostics are attached.

Thank you!

tower-diagnostics-20241226-2051.zip

There shouldn't be any issues moving files from zfs, I do it every day, the log is being spammed with:

 

Dec 25 19:43:17 Tower nginx: 2024/12/25 19:43:17 [error] 8829#8829: *1485164 limiting requests, excess: 20.900 by zone "authlimit", client: 172.17.0.2, server: , request: "GET /login HTTP/1.1", host: "192.168.2.10"

 

Do you know what 172.17.0.2 is?

4 minutes ago, JorgeB said:

There shouldn't be any issues moving files from zfs, I do it every day, the log is being spammed with:

 

Dec 25 19:43:17 Tower nginx: 2024/12/25 19:43:17 [error] 8829#8829: *1485164 limiting requests, excess: 20.900 by zone "authlimit", client: 172.17.0.2, server: , request: "GET /login HTTP/1.1", host: "192.168.2.10"

 

Do you know what 172.17.0.2 is?

Tautulli docker.

LE: I stopped it, I am not using it much anyway. I believe it might be spamming the log because it tries to read changes to FS, and I am currently in the process of moving 300+ GB from my main PC to the server. But now it's off.

Edited December 26, 20241 yr by war4peace

That wasn't the cause.

I just moved 29 files from one directory, and 21 remained untouched, despite the expectation they would have been deleted.

Reboot to clear the logs, then post new diags after it happens again.

Done.

Diagnostics attached, generated right after the first occurrence of failed file deletions during a move, after system reboot.

tower-diagnostics-20241227-1926.zip

Nothing relevant logged, you try mapping the Windows share with UD and then using midnight command on Unraid to do the move and see if there's any difference.

 

I move files every day from  zfs pool to my Windows desktop using explorer, never had issues, but they are mostly larger media files.

It's happening using Krusader as well, which is docker local to Unraid. I managed to narrow it down to file deletions only, which means it's not necessarily a "file move" issue, it's a "file deletion" issue.

I wasn't expecting anything to be logged because there are no errors. The file move process finishes as expected, basically the file system, for lack of better wording, "lies" to the copy/move/delete solution (whichever that is) that the file deletion was successful, where, in fact, it was not. This is actually the weird part of it.

The files are both larger (up to 4 GB) and smaller (as small as several MB), and the issue does not consistently favor any size range.

 

But I will try mc locally, check whether the issue persists there as well.

I think I know what has been causing this, but I am not 100% sure.

Back when I added those HDDs, I used Unbalanced plugin several times to consolidate data, and I think there were some errors when gathering data from other HDDs to the ZFS ones, which resulted in unbalanced not deleting the source properly, which, in turn, left many files and folders „duplicated” across HDDs.

Naturally, in the share itself, those appear only once, but when deleting them, they „pop back in” once more, since only one of the „mirrored” files was deleted.

 

I'll now have to figure out a way to find all duplicates across disks, rather than shares, and delete the ones outside Disk3 and Disk6.

 

Once that's done, I'll update here.