On 11/21/2024 at 5:47 AM, JorgeB said:

Type powerdown in the CLI, if it doesn't shutdown after 5 minutes force it, no point in trying to continue the rebuild

 

 

 

so an update after 3 day re-sync...i assumed the worst since the entirety of both drives were dumped into lost+found with none of the original share remaining.  however in my spot checking, the overwhelming majority of the data seems to be intact and correct.  the worst part seems to just be tons of folders with incorrect folder names, but the contents are still sorted and seem to be intact and named correctly.  

 

some questions:

 

1.  if there is an empty folder, does it necessarily mean there was something in there that was lost?  i see both numbered folders i didn't have before as well as correctly named folders with no files in them.  does that mean there were files in them that could not be recovered?  i did notice the overall TB size of lost+found seems to be the same as what was before the event.  

 

2.  what is an efficient way to move files around in lost+found share to my primary share, can i do it instantly?  moving from lost+found seems to want to initiate a copy to my cache drive instead.  i have 26 TB or so worth of data to move across, it would be very slow having to copy it to cache and move to array. 

 

im aware of hard links but since the shares themselves are different, can i still use them?  or maybe rename the root folder?

 

for example, my two repaired disks are as follow:

 

/mnt/disk9/lost+found/

/mnt/disk12/lost+found/

 

to allow for instant file moves, can i move the files into a folder called:

/mnt/disk9/stuff/lost+found/

/mnt/disk12/stuff/lost+found/

 

my main share i want to move these to is called 'stuff', which is not present in these disks.   can i just rename the root lost+found in each disk to /stuff/lost+found?  that would make it from a root share into a separate folder inside of the 'stuff' share, is that right?  then i can do atomic movies as i move these files back to where they belong?  im not sure though if this type of manual disk manipulation will break parity.  

Edited November 25, 20241 yr by oliver

14 hours ago, oliver said:

to allow for instant file moves, can i move the files into a folder called:

You should be able to move them from one share to another, as long as it's in the same disk, you can also export the lost+found folder as a user share.

2 hours ago, JorgeB said:

You should be able to move them from one share to another, as long as it's in the same disk, you can also export the lost+found folder as a user share.

 

i've currently enabled user share for lost+found but it is making a copy of files when i want to move them into their proper directories so that approach for 30TB would take ages to fix. 

 

so the follow commands:

 

mv /mnt/disk9/lost+found/ /mnt/disk9/stuff/lost+found/
mv /mnt/disk12/lost+found/ /mnt/disk12/stuff/lost+found/

 

should instantly transfer all files into the other share, while preserving parity and not overwriting anything? 

 

and once they are all in the same share, i can move folders around at the user share level and not mess anything up?

 

my goal is to leave them on the same disk, just transfer to a different share.  
 

Edited November 25, 20241 yr by oliver

1 hour ago, oliver said:

should instantly transfer all files into the other share, while preserving parity and not overwriting anything?

I think so.

Had some time to look at the mv command, that was my only doubt, parity will always be maintained, you can do it but need to create the "stuff" share/folder on those disks first.

 

On 11/25/2024 at 12:46 PM, JorgeB said:

Had some time to look at the mv command, that was my only doubt, parity will always be maintained, you can do it but need to create the "stuff" share/folder on those disks first.

 

thank you.  final update a week later after looking at lost and found.  out of 60k files, all but 3 were intact other than the root filenames of the folder (subfolders/files were fine).  i was able to repair the 3 bad video files so this is basically 100% recovery from FS corruption.  thanks again, the data wasn't critical but it's alway nice to save 30tb!

 

On 12/3/2024 at 7:27 AM, oliver said:

thank you.  final update a week later after looking at lost and found.  out of 60k files, all but 3 were intact other than the root filenames of the folder (subfolders/files were fine).  i was able to repair the 3 bad video files so this is basically 100% recovery from FS corruption.  thanks again, the data wasn't critical but it's alway nice to save 30tb!

 

sorry to bother you again but today i stopped the server and started it to see the same behavior pop up again.  i am able to emulate those two disks successfully  but i've decided to just replace them, they are nothing but trouble.  i did hear a loud strange beep come from the disks as well.  

 

this time, i want to be careful not to ruin the filesystem as i can see both disks correctly emlulated.

 

the problem is...i cannot stop the array, clicking the stop button isn't doing anything at all.  there's no containers running or vm's.  clicking the stop button just has no effects

 

EDIT: i see this error when clicking STOP array button, something wrong with http server?

 

Dec  8 10:17:56 nas nginx: 2024/12/08 10:17:56 [error] 14354#14354: *17335 connect() to unix:/var/run/emhttpd.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: 172.16.1.240, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "172.16.1.3:82", referrer: "http://172.16.1.3:82/Main"

 

restarting nginx but i still get same error.  can i safely stop array via CLI?  

 

nas-diagnostics-20241208-1011.zip

Edited December 8, 20241 yr by oliver

Type reboot on the CLI, and if it doesn't reboot after 5 minutes you will need to force it.

37 minutes ago, JorgeB said:

Type reboot on the CLI, and if it doesn't reboot after 5 minutes you will need to force it.

 

thanks i got a clean reboot.  im mulling over leaving it all shutdown vs emulating until new drives arrive.  

 

i sat next to the sever while booting and i notice right when the UI loads, the area with the 2 drives, it sounds like the hard drives basically restart.  it sounds like they are abruptly stopped and restarted.  im not starting the array in any way, just when the UI is initially loading.   still doing more disassembly to narrow down individual drives

 

im curious, from these diags attached are you able to see anything to that?  

 

from a hardware perspective, i am actually running my array on two power supplies since the last incident.  all of the hard drives are on one entirely dedicated 750W PSU and the rest of the hardware is on it's own.  at this point i've also replaced all the data cables so at this point i don't know what else it could be other than the drives although very interesting they seem to fail always together.  there's a set of 4 drives connected to the same power/data and the other two seem fine.  

nas-diagnostics-20241208-1104.zip

I do see what look like power/connections issues for disks 9 and 12 during boot.

48 minutes ago, JorgeB said:

I do see what look like power/connections issues for disks 9 and 12 during boot.

 

ok so i took the machine apart some more and something i realized.

 

There is a 4 way split power adapter going into my drives, two of which are connected to disk9/12.  While I replaced the source of the power to this splitter, I don't think I replaced the power splitter itself.  I can't remember for sure but given how dusty it was, I doubt I switched it up.  The other two disks on the same splitter never showed issues though.  That being said, I did replace it and didn't notice the restart noise coming.  This would also pretty much line up with my experience being having these errors two times after I opened the chasis and physically made changes to the array.

 

I can think of 3 options at this point: 

 

1.  Do a new array config and preserve parity.  - When this error first happened, it was a freshly started array (the prior shutdown was normal).   I think there were a few hundred bytes written according to the UI but no files.  It was only up for maybe 15 minutes.  

2  Try to rebuild these two disks from parity, emulation looked good when the error first occurred, so I don't think there's FS damage this time.

3.  Wait for RMA drives to arrive and rebuild.  Use dual emulation in the interim.  While large, the array isn't very heavily used, I can at the very least limit array usage to reads and minimal writes until RMA is done.  Even if option 1 or 2 is successful, I might prefer this just for peace of mind.  

 

I'd love to hear your input on which of these 3 approaches you think is best.  I also did attach a diag if you don't mind confirming this was a 'clean' start (I noticed notice strange noises).  

 

nas-diagnostics-20241208-1254.zip

Edited December 8, 20241 yr by oliver

20 minutes ago, oliver said:

There is a 4 way split power adapter going into my drives

Was this a SATA->SATA or a Molex->SATA splitter?    You should not split the SATA->SATA type more than 2 way although you can normally get away with 4 way on the Molex type.

4 minutes ago, itimpi said:

Was this a SATA->SATA or a Molex->SATA splitter?    You should not split the SATA->SATA type more than 2 way although you can normally get away with 4 way on the Molex type.

 

it's SATA- > SATA.  but to be clear, it's only 4 drives on the entire cable going to my PSU, the only reason i need to use the splitter is to remove the 3.3v and i only have one cable to remove it.  

42 minutes ago, oliver said:

it's SATA- > SATA.  but to be clear, it's only 4 drives on the entire cable going to my PSU

Note it is the number of drives on the splitter that can be the issue.   That is because a single SATA connector cannot carry enough current to power 4 drives reliably.    The cable to the PSU is rarely the issue as that is normally beefy enough to carry whatever current the PSU can deliver.

38 minutes ago, itimpi said:

Note it is the number of drives on the splitter that can be the issue.   That is because a single SATA connector cannot carry enough current to power 4 drives reliably.    The cable to the PSU is rarely the issue as that is normally beefy enough to carry whatever current the PSU can deliver.

thanks, i think i have some molex connectors spare, i can try to reduce the number of drives per connector.  should take care of the 3.3v issue as well.  it's just odd how intermittent the issue is

1 hour ago, itimpi said:

Note it is the number of drives on the splitter that can be the issue.   That is because a single SATA connector cannot carry enough current to power 4 drives reliably.    The cable to the PSU is rarely the issue as that is normally beefy enough to carry whatever current the PSU can deliver.

so my PSU has 4 molex outputs.  can each molex handle only a single drive or a cluster of 4?

 

this is a spec sheet to my exact PSU - https://coolermaster.egnyte.com/dl/GqmpIy1k3U

 

My HDD are all mostly HC550 - SATA model - i found this spec sheet - https://www.e4company.com/wp-content/uploads/data-sheet-ultrastar-dc-hc550.pdf

 

with the adaptors i have on hand, i can put a total of  8 drives on two molex connectors, assuming it's safe to split 4 x 2.  the remaining 8 drives, i can put on standard SATA connectors, one per conenctor.  

Edited December 8, 20241 yr by oliver

41 minutes ago, oliver said:

can each molex handle only a single drive or a cluster of 4

In my experience a Molex connector can carry enough current to handle 4 drives.  

1 hour ago, itimpi said:

In my experience a Molex connector can carry enough current to handle 4 drives.  

 

 

after shuffling around a bunch of parts, i finally have it in a much better state with no splitters.  

 

16 drives. 

 

1 molex -> 4x sata

1 molex-> 4x sata

8x sata-> sata no splitters/adapters

 

i wish there was a safe way to test power draw on all the drives at once.  i suppose a parity check in maintenance mode without corrections would normally be the way to go but since i have 2 drives in a bad state, i can't do that.  

 

 

im wondering which of these options below is the way to go.

Quote

 

1.  Do a new array config and preserve parity.  - When this error first happened, it was a freshly started array (the prior shutdown was normal).   I think there were a few hundred bytes written according to the UI but no files.  It was only up for maybe 15 minutes.  

 

2  Try to rebuild these two disks from parity, emulation looked good when the error first occurred, so I don't think there's FS damage this time.

 

3.  Wait for RMA drives to arrive and rebuild.  Use dual emulation in the interim.  While large, the array isn't very heavily used, I can at the very least limit array usage to reads and minimal writes until RMA is done.  Even if option 1 or 2 is successful, I might prefer this just for peace of mind.  

 

 

 

 

 

nas-diagnostics-20241208-1727.zip

Edited December 8, 20241 yr by oliver

If both disabled disks are emulating correctly, it's usually recommended to just rebuild them.

1 hour ago, JorgeB said:

If both disabled disks are emulating correctly, it's usually recommended to just rebuild them.

 

i was thinking overnight, is there any kind of safe way to spin all drives in order to 'test' that my power supply can handle the current?  without any array operations.  if another drive drops out during rebuild, i end up in the same mess as when i first started.

 

the closest i can think of would be to run an extended smart test on all drives at the same time, which i should would be max output.  thoughts?

You can start in maintenance mode and spin the disks down and up.

13 minutes ago, JorgeB said:

You can start in maintenance mode and spin the disks down and up.

 

is maintenance mode generally risk free of further damage since things aren't mounted (thinking filesystem mainly)?  if another data drive for example was to lose power in maintenance mode, would it also become marked as 'bad'?  

1 hour ago, oliver said:

if another data drive for example was to lose power in maintenance mode, would it also become marked as 'bad'?  

It shouldn't since there won't be any writes, and also you already have two disabled drives, there can't be three with dual parity.

On 12/9/2024 at 8:45 AM, JorgeB said:

It shouldn't since there won't be any writes, and also you already have two disabled drives, there can't be three with dual parity.

another successful rebuild on first try and all files intact.  i feel i should play the lottery at this point.  thanks for the millionth time with your assistance. 

On 12/9/2024 at 8:45 AM, JorgeB said:

It shouldn't since there won't be any writes, and also you already have two disabled drives, there can't be three with dual parity.

 

woke up today to see a parity drive dropped out.  looks like another random power/data failure?  sigh, it ran for a good week before just resetting.  i did notice this happened when mover was running, which i guess makes sense since there was writing happening

 

is there any way to see from logs if it's data vs power issue?

 

i suppose now a parity rebuild is required

nas-diagnostics-20241221-0546.zip

Edited December 21, 20241 yr by oliver