Help unraid keeps disabling disks

[buster84]

My unraid server worked great until i added two more disks. I added two 14tb drives to get a second parity and add osme space. Since then Disk 10 and disk 3 wont mount and keep saying un mountable. Parity one has done the same. I rebuilt the parity, then rebuilt disk 3 and 10 and then it said complete, only for the parity to go back to being disabled and disk 3 and 10 are back to unmountable. 

 

I really don't know how to fix this, ive tried all the basics. Can anyone help.  Ive attached the diagnostic for the server. 

nas-diagnostics-20230825-2125.zip

Edited August 26, 2023 by buster84

[buster84]

Wow, whats worse is that i just noticed that i lost my pictures folder. ITs completely missing from shares. I knew i should have backed that up. dammit.

 

I have a feeling parity drive 1 had the data still but that is showing as disabled. Parity two is working but it missnig the photos. Is there anyway to get this data back or am i totally screwed? How can i disable parity 2 and use parity 1 to re-fix drive 3? I have a feeling the two paritys dont match which might explain why it became disabled after parity 2 synced. I really hope i didnt loose this data i only added a second parity drive to keep the data safe, not knowing that im risking it further. 

 

I truly hope someone can help me. My pet died and i kept all my photos on there. Im completly mad at myself for adding a second parity now. Something got totally screwed up and now i feel like ive lost it all forvever.

Edited August 26, 2023 by buster84

[JorgeB]

Replace the SATA cable for disk10, then check filesystem on both unmountbale disks (run it without -n) and post new diags after array start.

[itimpi]

Are you sure your power supply is up to handling the additional drives?

[buster84]

4 hours ago, JorgeB said:

Replace the SATA cable for disk10, then check filesystem on both unmountbale disks (run it without -n) and post new diags after array start.

 

I replaced the cable and then performed this using the xfs_repair button on drive 3, but it just stops after saying this 

 

"

Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair."

 

I just re-ran it in terminal ands searching for a secondary superblock. I'll let it run and see what it does. Odd that it didnt say that when runing from the webui. i uploaded a new diagnostic and ill upload another once it finished (if it does and its not actually frozen).

 

 

1 hour ago, itimpi said:

Are you sure your power supply is up to handling the additional drives?

 

its a 600w power supply, it had enough plugs to fit the drives so i would say yes, but it is almost 10 years old. I didnt think about this being an issue, but its possible. 

nas-diagnostics-20230826-1003.zip

Edited August 26, 2023 by buster84

[itimpi]

25 minutes ago, buster84 said:

I just re-ran it in terminal ands searching for a secondary superblock.

 

This almost certainly means you got the command wrong.   What was the actual command you used?

 

[buster84]

4 minutes ago, itimpi said:

 

This almost certainly means you got the command wrong.   What was the actual command you used?

 

all I wrote is 
 

xfs_repair /dev/(drive name)

 

all its doing now is showing dots and adding more to it as it goes to there next line. I tried using the webui without -n and it only went as for as what I posted above. 

[itimpi]

5 minutes ago, buster84 said:

all I wrote is 
 

xfs_repair /dev/(drive name)

 

all its doing now is showing dots and adding more to it as it goes to there next line. I tried using the webui without -n and it only went as for as what I posted above. 

The question is what did you use for ‘drive name’.   It is easy to get it wrong when not using the GUI.

[buster84]

19 minutes ago, itimpi said:

The question is what did you use for ‘drive name’.   It is easy to get it wrong when not using the GUI.

I choose SDK which is disk 3. Drive 3 is showing as green while the rest are grey and asleep now (I set drives to sleep at 15mins)

 

its also showing a read speed of 250mbps and is moving up and down so it’s doing something

Edited August 26, 2023 by buster84

[buster84]

Is it possible that the second parity was created without drive 3 contents when I did a parity rebuild or would parity 1 prevent that from happening? After the rebuild of parity two I let it rebuild drive 3 and 10 then turned off and on the array and that is when parity one got disabled. 
 

it feels like parity two is missing drive 3 contents while parity 1 still has them and that might explain why parity 1 became disabled as it doesn’t match parity two? Im just speculating as I don’t know how it all works exact but if you could explain it to me that would really help.

 

thabks for your help. I really hope I can regain access to my photos again. 

[itimpi]

9 minutes ago, buster84 said:

I choose SDK which is disk 3.

But did you use sdk1 ?   You need to include that partition number, not just the device part.   Note that using the raw device names will invalidate parity.  Ideally you should have the array started in Maintenance mode and then use /dev/md3p1 (Unraid 6.12.x) or /dev/md1 (Unraid 6.11.x and earlier) which maintains parity.

[buster84]

11 minutes ago, itimpi said:

But did you use sdk1 ?   You need to include that partition number, not just the device part.   Note that using the raw device names will invalidate parity.  Ideally you should have the array started in Maintenance mode and then use /dev/md3p1 (Unraid 6.12.x) or /dev/md1 (Unraid 6.11.x and earlier) which maintains parity.

I just used SDK,

 

So i should re-do it with this command? xfs_repair /dev/sdk1

 

I also had it in Maintenance mode and im on 6.11.5

 

 

When i do it from the UI it says this "Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... Log inconsistent or not a log (last==0, first!=1) empty log check failed zero_log: cannot find log head/tail (xlog_find_tail=22) ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair."

 

should i just use this instead and do -L?

Edited August 26, 2023 by buster84

[itimpi]

29 minutes ago, buster84 said:

should i just use this instead and do -L?

Yes - do it from the GUI and use -L (and not -n)

[buster84]

20 minutes ago, itimpi said:

Yes - do it from the GUI and use -L (and not -n)

ok it finished. Ive attached the new log. I seen it read the second parity and not the first. Is there no way that the data is missing from one parity vs the other? I see in the log there were files sent to lost and found. Can i start the array now or anything else i should do? 

 

 

nas-diagnostics-20230826-1208.zip

[itimpi]

There is no reason to use both parity drives if repairing a single failed drive as parity has no concept of data - it just holds the information that in conjunction with the good data drives allows the bit pattern of the failed drive to be reconstructed.    If you have a lost+found folder that will contain any files/folders for which the repair process could not find the directory entry to give them their correct name.

 

at this point is the drive being emulated?    If so then the repair happened against the emulated drive and it is possible it might be more successful against the physical drive.

[buster84]

25 minutes ago, itimpi said:

There is no reason to use both parity drives if repairing a single failed drive as parity has no concept of data - it just holds the information that in conjunction with the good data drives allows the bit pattern of the failed drive to be reconstructed.    If you have a lost+found folder that will contain any files/folders for which the repair process could not find the directory entry to give them their correct name.

 

at this point is the drive being emulated?    If so then the repair happened against the emulated drive and it is possible it might be more successful against the physical drive.

ok i turned the array on and it appears to have found some data, but not my entire pictures share. That is missing entirly and there as no photos in the lost+found. How do i get this back? It sounds like the missing stuff is on the other parity? 

nas-diagnostics-20230826-1229.zip

Edited August 26, 2023 by buster84

[itimpi]

23 minutes ago, buster84 said:

It sounds like the missing stuff is on the other parity? 

There is no concept of data being on one parity disk and not the other - they are equal in terms of content.

 

are you sure that none of the files/folders in lost+found are actually photos?   They will have cryptic names but you can use the Linux ‘files’ command to find out what the file types (and thus their extension) should be.

 

the only think I can think of is whether a disk recovery utility such as UFS Explorer on Windows might be more successful.

 

are you actually sure the photos were on the drive you are currently working on?

 

 

 

 

[buster84]

9 minutes ago, itimpi said:

There is no concept of data being on one parity disk and not the other - they are equal in terms of content.

 

are you sure that none of the files/folders in lost+found are actually photos?   They will have cryptic names but you can use the Linux ‘files’ command to find out what the file types (and thus their extension) should be.

 

the only think I can think of is whether a disk recovery utility such as UFS Explorer on Windows might be more successful.

 

are you actually sure the photos were on the drive you are currently working on? 

im 100% sure. I created a share called pictures when i copied it over from my other nas when migrating. It copied it all to drive 3. I had almost 7tb of data on that drive, it only found 205gb. 

 

When the second parity was added I immeditly lost drive 3 and 10 to become un mountable. Then i did a partiy recovery on that drive and when it completed the drive became unmountable again as if the parity its self was corrupt or incomplete. 

 

Is there no way to force the system to rebuild drive 3 from the disabled parity 1? I already copied the lost and found folder so at this point i have nothing else to loose. The loss of all my photos is worse than loosing anything else on my system. Thankfully i had some duplicated in a shared pictures folder to share with family, but my original pictures share completly dissapeared. 

[buster84]

What would happen if i did a new configuration and click parity is valid? if both parities dont match will it just redo both parities? At this point i dont have much to loose so im just thinking of something i can try. 

 

I will say that i am happy thoughh, i was able to recover some files in the lost and found. I found more pictures by searching for them and pulling them out of the folders. Still missing alot, but a little it better than nothing. Thanks for all your help and continued help!

Edited August 26, 2023 by buster84

[buster84]

ok its happening again. I actually never did a new config left it alone as i was annoyed. Now im super annoyed. More data moved to lost and found. Im ready to go back to readynas at this point. Ive never experienced so much crap in my life and Its merging all my data messing it up. I dont understand what is going on.  Now its disabled disk 1.

nas-diagnostics-20230901-1225.zip

Edited September 1, 2023 by buster84

[JorgeB]

Multiple disks getting disable without an apparent reason point to a hardware problem, first thing to do would be to update the LSI firmware:

LSISAS2308: FWVersion(20.00.06.00)

 

All 20.xx releases except 20.00.07.00 have known issues.

[buster84]

23 minutes ago, JorgeB said:

Multiple disks getting disable without an apparent reason point to a hardware problem, first thing to do would be to update the LSI firmware:

LSISAS2308: FWVersion(20.00.06.00)

 

All 20.xx releases except 20.00.07.00 have known issues.

I thought about this too; however, the first time they got disabled it was disk 10 and then the parity which are run of the sas of the motherboard not the LSI. I'll try to update it anyways, but something else must be going on. 

[JorgeB]

Could also be a PSU problem, or just SATA cables and/or power connector/splitter issues.

[buster84]

1 hour ago, JorgeB said:

Could also be a PSU problem, or just SATA cables and/or power connector/splitter issues.

 

PSU is my next replacement if this didnt fix it. 

 

I upgraded the firmware, wasnt fun at all since i had to learn from scratch. Got it done with dos. I then tried to turn on my unraid server learning that my flash drive wouldnt get recognized anymore for boot. Not sure why, but it would load up in my other PC. I then re-installed unraid and transfered the key over to another usb drive. 

 

I have 2 questions.

 

1) Could a bad usb drive cause the issues im having? 

2) Whats the easiest way of removing a drive? I'd like to remove 2 drives with no data on them to help see if maybe it is my powersupply and removing the drives would revert me back to how it was before i added the drives and ran into this issue. 

[JonathanM]

1 minute ago, buster84 said:

Whats the easiest way of removing a drive? I'd like to remove 2 drives with no data on them

tools, new config, save all, main gui page, unassign drives you want to remove, start array to rebuild parity without them. Do NOT select parity is already valid, it is not, it must be rebuilt.