Mover not working

[matus]

Hi all,

I have all sorts of problems since I replaced my cache SSD for 1 cache & 1 NVME and changed FS to ZFS.
Currently mover is not working at all. It just blinks and log is showing: 

Quote

Aug 23 22:51:32 XXXXXX move: mover: started
Aug 23 22:51:32 XXXXXX move: mover: finished

Changed share settings multiple times, downloaded mover plugin, uninstalled mover plugin. Tried with VMs & Docker off and on and still nothing. 

 

My settings:

I don't think I can go back to previous Unraid version as I've set NVME & Cache to ZFS and my previous version was 6.11.5.

[trurl]

Attach diagnostics to your NEXT post in this thread.

[matus]

I'd love to, but I think Diagnostics froze after 3 hours of generating Download. Is this normal time to generate Diagnostics report? Think it took seconds last time I needed it.

[JorgeB]

Reboot in safe mode, enable mover logging, run the mover, post new diags

[matus]

43 minutes ago, JorgeB said:

Reboot in safe mode, enable mover logging, run the mover, post new diags

Thank you! Here it is, hope it helps.

diagnostics-20230824-1235.zip

[JorgeB]

Apparently there's nothing to be moved, which shares do you expect to be moving?

[itimpi]

Looking at where the files for shares are located the only anomaly I see is where a share has parts on two different pools:

appdata                           shareUseCache="prefer"  # Share exists on cache, nvme

However mover does not move files between pools - this needs to be done manually.

[matus]

It's weird, cause I can't initialize mover in any case (e.g. I've tried appdata: cache; NVME; array - mover just blinks and that's it). As per the screenshot in my first post, files are not in locations set and mover is still not doing anything. Multiple shares are physically in NVME even though I don't have NVME set for any share.

Just a little background: yesterday I found out my server is down. I could access gui, but couldn't do anything. I found this in log:
 

Quote

This is the log from yesterday:
Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=380350464 size=12288 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=379961344 size=12288 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=380313600 size=8192 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=1234878464 size=69632 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=47233413120 size=12288 flags=180880

Aug 22 13:00:05 XXXXXX kernel: WARNING: Pool 'nvme' has encountered an uncorrectable I/O failure and has been suspended.

I was freaking out and thought my setup is gone, because I didn't have appdata & system backed-up (they were in NVME pool). I turned server off and on and everything seemed to be fine. Until I find out what was the failure with NVME I wanted to move everything from NVME to cache, but with no success.

I was able to initialize mover once after I installed Mover Tuning and set "Only move at this threshold of used cache space:" to 5%. It went for some time, but physically mover didn't do anything. I've uninstalled & installed Mover Tuning, but can't replicate it anymore.
 

Quote

 

itimpi wrote:

Looking at where the files for shares are located the only anomaly I see is where a share has parts on two different pools:

appdata                           shareUseCache="prefer"  # Share exists on cache, nvme

However mover does not move files between pools - this needs to be done manually.

 

 

But I can't move files from nvme to array as well? 

Edited August 24, 2023 by matus

[JorgeB]

29 minutes ago, matus said:

But I can't move files from nvme to array as well? 

Appdata is set to cache prefer, so it would move to cache, set the mover correctly for that share, run the mover and post new diags.

[matus]

Don't know what's happened now, but mover is running currently. I don't think it will actually move anything, so I can try again once it's finished and will post new diags.

How should I set it to test the mover (I've tried all possible combinations for appdata and mover only blinked so I'm paranoid and doubting myself)?

 

 

[itimpi]

For mover to do anything then you need to have a pool set as primary storage, the array set as secondary storage and the mover direction set appropriately.

[matus]

I think it's working.

I've re-enabled docker and stopped all individual dockers. After that, I've disabled docker again. Mover started immediately and now I have everything on my Array. So this is the solution - together with correct shares setting.

Any pointers regarding issue that started it all?

Quote

This is the log from yesterday:
Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=380350464 size=12288 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=379961344 size=12288 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=380313600 size=8192 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=1234878464 size=69632 flags=180880

Aug 22 12:57:14 XXXXXX kernel: zio pool=nvme vdev=/dev/nvme0n1p1 error=5 type=2 offset=47233413120 size=12288 flags=180880

Aug 22 13:00:05 XXXXXX kernel: WARNING: Pool 'nvme' has encountered an uncorrectable I/O failure and has been suspended.

I don't think I can trust this disk until I know what caused it.

 

Oh, and thank you both itimpi & JorgeB for your patience and support.

Edited August 24, 2023 by matus
forgot to say thank you

[JorgeB]

1 hour ago, matus said:

Any pointers regarding issue that started it all?

Do you have the full diags to see the errors in context?

[trurl]

2 hours ago, matus said:

now I have everything on my Array

Assuming that is true, it is a good start since mover won't move between pools. So now you can work on getting appdata, domains, system shares on the pool you want them on.

 

Post new diagnostics