(SOLVED) Mover moving files but not deleting them from cache

[StudiesTheBlade]

Unraid w/ NVIDIA Plugin: v6.8.3

Mobo: Asrock X470 Taichi w/ BIOS P2.0

CPU: AMD Ryzen 2700X

RAM: 2x16GB Gskill Ripjaws V @ 2400Mhz

 

Hello,

 

Today I noticed that my cache drive was filling up and I started poking around the terminal and noticed that there were files in /mnt/cache that should have been moved (hourly mover schedule). I turned on mover logging and noticed that upon the first move attempt for a given file it reports back with this:

Jul 28 12:31:30 unraid-tower move: move: file /mnt/cache/isos/Dynmap-3.0-beta-10-spigot.jar
Jul 28 12:31:30 unraid-tower move: move_object: /mnt/cache/isos/Dynmap-3.0-beta-10-spigot.jar Operation not supported

On the second run of the mover I get this:

Jul 28 12:32:53 unraid-tower move: move: file /mnt/cache/isos/Dynmap-3.0-beta-10-spigot.jar
Jul 28 12:32:53 unraid-tower move: move_object: /mnt/cache/isos/Dynmap-3.0-beta-10-spigot.jar File exists

Sure enough, the file has been moved from the cache to the array, but it never gets removed from the cache drive.

 

This seems to have just recently started (the past day or so) and I cannot recall any setting that I may have set that would cause this. I did recently experiment with enabling zstd compression on the btrfs filesystem used by the cache, but that seemed to be working with no issues for at least a week.

 

If someone could offer any insight into why this might be happening, It would be greatly appreciated.

unraid-tower-diagnostics-20200728-1218.zip

 

[StudiesTheBlade]

Bump, has anyone ever seen anything like this?

[JorgeB]

Not that I recall, can you move the file manually, if not maybe there will be a more detailed error.

[StudiesTheBlade]

I can move the files to that specific disk manually with no issue. I can also delete the file from the disk and then invoke the mover without issue. It's almost as if the file is being written to the disk and the cache at the same time during the initial write.

[StudiesTheBlade]

I ran 'mount -l' and was curious about a few results:

proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /var/log type tmpfs (rw,size=128m,mode=0755)
/dev/sde1 on /boot type vfat (rw,noatime,nodiratime,flush,dmask=77,fmask=177,shortname=mixed) [UNRAID]
/boot/bzmodules on /lib/modules type squashfs (ro)
/boot/bzfirmware on /lib/firmware type squashfs (ro)
hugetlbfs on /hugetlbfs type hugetlbfs (rw)
/mnt on /mnt type none (rw,bind)
tmpfs on /mnt/disks type tmpfs (rw,size=1M)
nfsd on /proc/fs/nfs type nfsd (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/dev/md1 on /mnt/disk1 type xfs (rw,noatime)
/dev/md2 on /mnt/disk2 type xfs (rw,noatime)
/dev/md3 on /mnt/disk3 type xfs (rw,noatime)
/dev/md4 on /mnt/disk4 type xfs (rw,noatime)
/dev/nvme0n1p1 on /mnt/cache type btrfs (rw,space_cache=v2)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
secure: on /mnt/disks/backblaze type fuse.rclone (rw,nosuid,nodev,allow_other)
/mnt/cache/system/docker/docker.img on /var/lib/docker type btrfs (rw,noatime,space_cache=v2)
/mnt/cache/system/libvirt/libvirt.img on /etc/libvirt type btrfs (rw,noatime,space_cache=v2)

Is it normal to have that /mnt mount and both a /mnt/user and a /mnt/user0 ?

[JorgeB]

22 minutes ago, StudiesTheBlade said:

Is it normal to have that /mnt mount and both a /mnt/user and a /mnt/user0 ?

Yes, /mnt/user0 is the same content as /mnt/user except cache.

[StudiesTheBlade]

Not sure what exactly was causing this, but it was resolved after using the mover to move everything off the cache drive and running:

blkdiscard /dev/sdx

After moving everything back, the mover appears to be moving everything properly again.