[SOLVED] Disk disabled during transfer

[jensrobot]

Was copying files inside my array and i got errors and now it's disabled.

The disk was just added to the array. It's old but passed smart and the preclear operations. Smart is still not reporting anything wrong but drive is disabled and array is unprotected. Can someone advise me what to do? i attach logs

Error started at Aug 31 03:43:05 "SKABET kernel: md: disk6 write error, sector=885344016"

I have these disks attached to a LSI sas controller I flashed to IT mode. I haven't used this before. I'm also using an old Icydock for this disk which is also a new addition to the machine. Might be relevant?

Havent had array failures before so this is new territory for me...

skabet-diagnostics-20210901-0316.zip

Edited September 15, 2021 by jensrobot
solved

[trurl]

Looks like you were having problems communicating with both disks 6,7 even while you were preclearing them then after assigning them to the array disk6 just happened to be the one that got disabled first and with single parity it can only disable one.

 

Seems like you have already identified some suspects:

34 minutes ago, jensrobot said:

I have these disks attached to a LSI sas controller I flashed to IT mode. I haven't used this before. I'm also using an old Icydock for this disk which is also a new addition to the machine. Might be relevant?

 

[jensrobot]

thx for the reply, and yeah i'm beginning to suspect the 2 new variables. I've had some trouble with the sas controller hanging at the pre-bios screen, but that seemed to "fix itself" with a reboot or just waiting a couple of minutes. My case is a mess as it is now, and i use a lot of splitters for sata power (molex for icydock) and what not. I'm not too confident with this... I'm also using a 350W PSU, and i think im pushing its limits but i dont rly know for sure.

I still have a spare sata port on mobo. I'm thinking if isolating this issue by connecting disk 6 directly into that. Disk 7 was only added to array but doesnt have data on it, so ill remove that.

Do i need to rebuild/repair the array? i mean the disabled disk needs to be reformatet and re-added or?

[jensrobot]

I cant find the model of the icydock im using but its very similar to this one (mine seems to be and older version - no model number anywhere on product): https://www.icydock.com/goods.php?id=48

This one is advertised as 3 molex connectors power 5 drives. Mine is also a 5 drive dock, but only have 2 molex which i connected power to (only 2 drives in my bay atm)

[trurl]

12 minutes ago, jensrobot said:

i use a lot of splitters

That might be something to examine, maybe the icydock would be OK if it was getting good power.

 

Technically you should rebuild the disabled disk, but you must be able to reliably read every bit of all other disks to rebuild.

 

If you don't want disk7 involved you might New Config without disk7 and rebuild parity instead of rebuilding disk6 and hope for the best. Since it looks like the disk is OK maybe it will be OK without rebuilding it or its filesystem could be repaired if that turned out to be necessary.

 

Another possibility would be to attempt rebuild of disk6, but to a spare disk and so keep the original as is in case there are problems rebuilding.

 

Emulated disk6 is mounted so you should be able to copy files from it to somewhere off the array if any are very important.

 

[jensrobot]

Currently transferring off the emulated disk. Will take quite a while, but after that I will feel more secure dealing with this.

 

45 minutes ago, trurl said:

If you don't want disk7 involved you might New Config without disk7 and rebuild parity instead of rebuilding disk6 and hope for the best. Since it looks like the disk is OK maybe it will be OK without rebuilding it or its filesystem could be repaired if that turned out to be necessary.

 

can you elaborate what you mean with "new config" i'm a bit confused. I'm guessing you mean i stop array, remove disk7 from dropdownlist and start array?

[trurl]

8 hours ago, jensrobot said:

Currently transferring off the emulated disk. Will take quite a while, but after that I will feel more secure dealing with this.

Best if you are transferring that data off the array, since writing to other disks in the array would be a lot slower due to emulation needing to read all disks. And you currently have no protection since there is already one disabled disk with single parity.

 

Even better would be if you already had backups of everything important and irreplaceable. Lots of more common ways to lose data that parity can't help with, including user error.

 

8 hours ago, jensrobot said:

can you elaborate what you mean with "new config" i'm a bit confused. I'm guessing you mean i stop array, remove disk7 from dropdownlist and start array?

It wouldn't even let you start the array since you would have too many missing or disabled disks. Even if you didn't already have a disabled disk, simply unassigning a disk and starting the array without it would only make it emulate the missing disk so it could rebuild it when you replaced it.

 

Tools - New Config lets you assign disks however you want, including adding, removing, or changing the order of disks. Then (optionally and by default) rebuilds parity on any disks assigned to any parity slots. If you remove (or add) a data disk with New Config, then parity rebuild is required since it isn't valid without exactly the same disks it was using before.

 

Note if you go that route, there would no longer be any option to rebuild disk6. Whatever is on disk6 is what you would get. Maybe it would be OK since the disk is OK, but it would have had some incomplete writes (which is why it was disabled) and you couldn't recover those. Some possibility that the incomplete writes would have caused some filesystem corruption that would have to be repaired.

[jensrobot]

 

3 hours ago, trurl said:

Best if you are transferring that data off the array, since writing to other disks in the array would be a lot slower due to emulation needing to read all disks. And you currently have no protection since there is already one disabled disk with single parity.

 

Even better would be if you already had backups of everything important and irreplaceable. Lots of more common ways to lose data that parity can't help with, including user error.

I transferred all contents of disk 6 of to another pc. It's not super important data, just video files that can be replaced. I checked some of the files and the play fine. I guess there might be a defective file(s) somewhere, or did the emulated drive take over when the error occurred? (i was doing the transfer between parity disks in mc when error occurred, and mc didnt give me error and seemed to complete the transfer).

I do have backups of irreplaceable data on other machines

3 hours ago, trurl said:

It wouldn't even let you start the array since you would have too many missing or disabled disks. Even if you didn't already have a disabled disk, simply unassigning a disk and starting the array without it would only make it emulate the missing disk so it could rebuild it when you replaced it.

Aah yes makes sense since I only have the one parity drive. Is it ill advised to only have 1 parity drive? I expanded quite a bit since i first got unraid, so maybe it would be smart to add a 2nd parity drive?

 

3 hours ago, trurl said:

Tools - New Config lets you assign disks however you want, including adding, removing, or changing the order of disks. Then (optionally and by default) rebuilds parity on any disks assigned to any parity slots. If you remove (or add) a data disk with New Config, then parity rebuild is required since it isn't valid without exactly the same disks it was using before.

 

Note if you go that route, there would no longer be any option to rebuild disk6. Whatever is on disk6 is what you would get. Maybe it would be OK since the disk is OK, but it would have had some incomplete writes (which is why it was disabled) and you couldn't recover those. Some possibility that the incomplete writes would have caused some filesystem corruption that would have to be repaired.

So just to be clear, if i remove disk7 from here it will re-activate disk6 and try to rebuild parity?

Wouldn't the files from disk6 be ok since emulated drive took over? and if file corruption took place, would it not only be the one file it was transferring when error occurred thats corrupted?

[trurl]

When a write fails to a data disk in the array, Unraid updates parity anyway and disables the disk because it is now out-of-sync. After the disk is disabled, it isn't used again until rebuilt (or New Config). After the disk is disabled, any reads of the disabled disk are emulated from the parity calculation by reading all other disks plus parity.

 

That initial failed write, and any subsequent writes to the disk, are emulated by updating parity. At a higher level (like mc) everything still works. So more than just the file being written at the time the disk was disabled is involved.

 

All those writes can only be recovered by rebuilding the disk. New Config would mean those writes couldn't be recovered. The contents of parity plus all other disks are required to emulate the disk. New Config means the disk wouldn't be emulated anymore and parity would be rebuilt based on the contents of whatever disks were assigned in the new config.

 

Also, some of the things written are not just file contents, but filesystem metadata that allows the raw data to be organized into files and folders. That is why I said there might be some corruption, though that would typically only happen if metadata update were incomplete at the exact time the disk was disabled since all subsequent metadata updates for later file operations would be using the emulated disk.

 

Since you know you were transferring multiple files at the time instead of just writing a single file, seems like rebuild is the best idea.

[jensrobot]

I really appreciate your help and time, and you explain it well i just think im way in over my head here and right now im honestly super confused.

I have a 2 port sata controller i can use to connect disk 6 and 7. This way I avoid the sas controller and icydock variables. Do i rebuild disk 6 like stated here?: https://wiki.unraid.net/Manual/Storage_Management#Rebuilding_a_drive_onto_itself

I dont have a spare 6TB disk, and in hindsight i should maybe have kept disk 7 as a spare disk instead of adding it to array right away

[trurl]

19 minutes ago, jensrobot said:

I have a 2 port sata controller i can use to connect disk 6 and 7.

What controller is it?

[jensrobot]

5 minutes ago, trurl said:

What controller is it?

Model:SSU-SATA3-T2 - Chipset: ASM1061

I used it in my old unraid system for years with no trouble

[trurl]

30 minutes ago, jensrobot said:

2 port sata controller i can use to connect disk 6 and 7

Go ahead and set that up then post new diagnostics.

 

 

[jensrobot]

53 minutes ago, trurl said:

Go ahead and set that up then post new diagnostics.

Here is the diag. i did 2 boots because the first time i didnt remove the sas card, but then it didnt register the sata card, so i removed sas completely and now i see disks

skabet-diagnostics-20210901-2313.zip

Edited September 1, 2021 by jensrobot

[trurl]

That looks good so far. Since you have already copied the data should be pretty safe to rebuild.

2 hours ago, jensrobot said:

Do i rebuild disk 6 like stated here?: https://wiki.unraid.net/Manual/Storage_Management#Rebuilding_a_drive_onto_itself

yes

[jensrobot]

thank you. I have now started the rebuild process, and will report back when it's done

[trurl]

I usually estimate 2-3 hours per TB.

[jensrobot]

Rebuild done with no errors yay  I'll probably remove disk 7 next so i have a spare disk.

I attach log in case you wanna have a look.

I learned some new things, and i will have to troubleshoot the bigger issue carefully. Since pre-clear and smart went fine on icydock+sas controller, i have to be smart about it. The errors happened after i already transferred multiple terabytes without error. I'm thinking about getting a new PSU (to get out of adapter/splitter hell), add some disks to the icydock (but don't add to array) and start some large test file transfers.

If i transfer from array directly to an unassigned disk, and if an error like previous occurs, it will break the transfer since i wont have an emulated drive taking over right?

 

If you have any suggestions on how to proceed i'm all ears. You've been a great help so far, so thanks again.

skabet-diagnostics-20210902-1923.zip