- Solved
- Urgent
Setup:
Data distribution method: High-Water
Share Split level: Split top 3
Disk Fullness:
parity: 12TB
parity 2: empty
disk1: 50%
disk2: 50%
disk3: 50%
disk4: 50%
disk5: 15% (highwater mark for new files)
Last night I was attempting to use the new mover command on disk5 to empty it to the remaining disks 1-4 as I wanted to pull disk5 from the array and repurpose it as a parity disk. The plan was to keep hardlinks intact instead of using alternative methods to spread the data.
I ran
mover start -e disk5 |& logger &
and to my surprise it appears as if data was moving to disk5 from itself. I stopped mover, disk5 was missing a few dozen GBs of data.
I changed every share to "exclude disk5" to ensure the data would be spread to the disks 1-4. Ran the above mover command once more, this time I confirmed data was in fact still moving to disk5 and then upon the move being completed, mover would delete the original file which just happened to be the source file as they were named the same. This time I was missing about 40GB of data and confirmed it was in fact NOT moved to disks 1-4 as their used/free space remained the same before I started all of this.
Here is an example from the "file-activity" plugin which monitors when a file is accessed/modified, you can see the .partial moves from disk5 to disk5 is then read opened and deleted. I suspect this was mover moving it and deleting it upon successful move... which is not good considering I wanted data moved to disks 1-4.
May 07 15:57:58 MOVED_FROM => /mnt/disk5/data/t----s/r----s/B----OG/s-----31.bin.partial
May 07 15:57:58 MOVED_TO => /mnt/disk5/data/t----s/r----s/B----OG/s-----31.bin
May 07 15:57:58 ATTRIB => /mnt/disk5/data/t----s/r----s/B----OG/s-----31.bin
May 07 15:57:58 OPEN => /mnt/disk5/data/t----s/r----s/B----OG/s-----31.bin
May 07 15:57:58 DELETE => /mnt/disk5/data/t----s/r----s/B----OG/s-----31.bin
I do not have mover tuning installed or any other plugin that modifies the behavior of mover.
See also:
-
1
