Cache drive BTRFS errors

[ldog88]

Hi, 

My unraid server has suddenly got BTRFS errors on the cache drive that are causing docker to not run.

Please can someone offer some assistance as to how to fix this? Whilst the server has been up and running well for some time, I am definitely a newbie so layman's terms would be appreciated.

I have attached the diagnostics.

Thanks

diagnostics-20220901-1850.zip

[JorgeB]

Since btrfs was detecting some data corruption before this it's a good idea to run memtest first, then backup and re-format the pool.

[ldog88]

Thanks for this. 

Please can you send a step by step guide to doing what you said? 

Thanks

 

[JorgeB]

Start with memtest, it's a boot option in the Unraid boot menu 

[ldog88]

So I rad the mem test over the weekend for 3 days straight with no issues by the looks of it. 

Would the next stage be to backup and reformat the pool? If so, please can you outline how I do this? 

Thanks 

[JorgeB]

Assuming the pool is still mounting just copy anything important elsewhere, then re-format and restore the data, you can use for example the Dynamix File Manager plugin to easily copy the files.

[ldog88]

I can't access krusader as the docker won't work with the current error. 

How do I go about copying all of the data to a USB hard drive that I have mounted using unasiged devices? Is it possible to do this through the unraid GUI? 

Thanks

[ChatNoir]

9 minutes ago, ldog88 said:

I can't access krusader as the docker won't work with the current error. 

Since you are on 6.10, you could use the great File Manager plugin to copy data.

[ldog88]

Just to double check, I need to go into Main then select Cache and then select all files and move to "remotes"? 

Sorry I'm very much a noob

[ChatNoir]

If I am not mistaking 'remotes' would be a network path to another system.

 

An external USB drive attached to your server would be at /mnt/disks/xxx I think (or simply 'disks' in the plugin).

[ldog88]

thanks for this. Managed to work it out but unfortunately the copy didn't work.

I have attached the diagnostics so if you could see any issues that would be much appreciated.

Thanks

diagnostics-20220906-2245.zip

[JorgeB]

Sep  6 21:44:17 Lawson-NAS kernel: nvme nvme1: I/O 908 QID 8 timeout, aborting
Sep  6 21:44:17 Lawson-NAS kernel: nvme nvme1: I/O 916 QID 8 timeout, aborting
Sep  6 21:44:17 Lawson-NAS kernel: nvme nvme1: I/O 917 QID 8 timeout, aborting
Sep  6 21:44:17 Lawson-NAS kernel: nvme nvme1: I/O 918 QID 8 timeout, aborting
Sep  6 21:44:17 Lawson-NAS kernel: nvme nvme1: I/O 919 QID 8 timeout, aborting
Sep  6 21:44:47 Lawson-NAS kernel: nvme nvme1: I/O 908 QID 8 timeout, reset controller
Sep  6 21:45:17 Lawson-NAS kernel: nvme nvme1: I/O 8 QID 0 timeout, reset controller
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 1068352 op 0x0:(READ) flags 0x80700 phys_seg 27 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 1836344 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 1837776 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 2003208 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 1838768 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 146702808 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 2263264 op 0x0:(READ) flags 0x80700 phys_seg 13 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 1068960 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 84863328 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 0
Sep  6 21:45:50 Lawson-NAS kernel: blk_update_request: I/O error, dev nvme1n1, sector 84862880 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0

 

Log shows issues with one of the NVMe devices, power cycle the server and try again.

[ldog88]

So I think the nvme device is the issue here and that is what is set up as the cache drive. It appears that most if not all of the files have copied over into my USB hdd. 

How do I now go about formatting the cache drive? 

I tried removing it from the array but was unable to then format it. 

Thanks 

[JorgeB]

You can wipe it by stopping the array and typing

blkdiscard -f /dev/nvme#n1

then start the array to format

 

[ldog88]

So I managed to clear it using preclear disks and then formatted. 

Having now added it back in and copied all of the data back to it, I am missing all of my docker containers. 

Is there a way to point docker to the cache so that it picks them up? 

Thanks

[JorgeB]

If you restored the appdata you can recreate the docker image.

[ldog88]

So I followed the recreate docker image steps however it doenst appear to have worked. 

I have attached the diagnostics. 

Thanks

diagnostics-20220912-2013.zip

[trurl]

https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file

 

https://wiki.unraid.net/Manual/Docker_Management#Re-Installing_Docker_Applications

[ldog88]

Crisis averted. Looks like I managed to move some files to the wrong folders.
Thank you very much for your assistance!