Installed New MB and had to go back to Old MB (will explain) V6.1.3

[ajgoyt]

Hi Guys got an issue i bought a new MB and cpu and ram per i need an upgrade per my old duo 2 mb is getting old, I performed a recent parity chk before the switch out last night.

 

Here is my issue i shut down the V6.1.3  server as normal and installed the new mb upon 1st start new mb found my flash drive and booted unraid and i had a couple of unmountable drives. after playing stopping and restarting the computer trying different cables etc, nothing seemed to help the unmountable status of the drives, Oh and there was a few hard boots in there

 

So a little panic started so i shutdown (Normally) and installed the OLD mb and restarted the server - Now i have 5 yes 5 drives that are unmountable..... 

 

I am following the WIki for checking the FS through the WEBUI in maintennace mode  and now performing a reiserfsck on the first of 5 disks that is unmountable and it just finished and said all was ok.

I then stopped maintenance mode and started the array and the first disk i just checked is now reading normal.

 

SO i have the following questions

1.  if all disks come back mountable and no errors should i run a new parity check on the system?

2. if the answer to #1 is perform a parity check can i try it with the new MB? or should i wait for the parity check to finish with the old mb?

3. is there a way to reiserfsck on (all) the unmountable disks in the system through the webui? 

 

 

 

 

[ajgoyt]

Well after an upgrade Hell - I am rebuiding data with the NEW MB/CPU,RAM on disk 1 per no matter what i tried cables - different ports etc i finally switched out drive 1 with another and again its almost done rebuilding the data on it, Ok now for my bigger issue i still have 5 yes...... 5 Drives that show UNMOUNTABLE.

 

See the picture of whats happening from the webui , the unmountable drives show green but it's like their file system is not there!

 

I have and will not format (I will loose my data on those drives) or even check parity until i hear from an expert on the next steps.

 

I have stopped my docker - Plex and also turned it off from auto starting...  Please help i am really frustrated and I have searched the site and there seems to be articles on this subject but i am skeptical on which path to take....

 

thanks AJ

 

 

 

 

 

 

[trurl]

Tools - Diagnostics. Post complete diagnostics zip

[ajgoyt]

Well after taking from advice from Tom on an email - I gutted the whole server - drives and addon cards and the just added the (unmountable) drives 6 of them --- yes 6 unmountable drives -  back into the new mb, I hooked them up with NEW sata cables with locking tabs, then powered up the computer.

 

Per Toms advice then - went to tools - New Config and assigned the drives and data drives, Keep in mind per Tom there is no parity drive installed just the 6 unmountable drives.  Then i assigned all 6 unmountable drives into the original slots per unraid and then started the array, well guess what all 6 drives are still unmountable, I am really getting wooried that all my data pictures etc are Gone.

 

Again the only hardware in the system right now is the following - the new MB,PS, 6 drives hooked directly to the mb 6 sata slots.  it appears that the reiserfs file system is non exsistant if you click on a drive from the main menu.  I did get a syslog and its attached.

 

all 6 of these drives were basically full and they add up to about 10tb .... does anyone know how to make these drives become mountable in Unraid?

 

AJ

 

tower-syslog-20151130-2157.zip

[garycase]

...  I am really getting wooried that all my data pictures etc are Gone.

 

Do you not have backups of your data??  [You're not alone, but it always amazes me how many folks don't bother to backup and then wonder why they lose data.  My thoughts are detailed here:  http://lime-technology.com/forum/index.php?topic=31020.0 ]

 

Notwithstanding that, if these were all Reiser-formatted drives, the Reiserfsck utility is VERY good at recovering data that would likely be toast in most other file systems.    I would, one-at-a-time, run Reiserfsck against the problematic drives ... and copy all recovered data to another drive.    Search the forum for Reiserfsck and you'll find excellent instructions on using this from some of the Linux experts.    I've never personally used it [No need as I have superb backups] ... but it often amazes me how well it can recover data from corrupted Reiser disks.

 

The basic initial command you want to use is:

 

reiserfsck --check /dev/mdX  (where X is the disk # you want to check)

 

Then, depending on what it finds, you'll need to use more advanced commands ... possibly rebuilding the file system tree structure.    Ask here for more detailed help ... or via e-mail to Limetech.

 

In reviewing the thread I see you ran this on at least one of the disks earlier in the process -- which options did you use for it?    And are you sure you ran it on the correct device?  I believe for some actions you have to also specific the partition # ... but as I noted, I am NOT an expert on using this.

 

 

 

 

[garycase]

Looked back at a recent thread where a user used reiserfsck to recover some badly corrupted disks.

 

The commands used were:

 

reiserfsck --check /dev/sdX1  (where X is the disk ID ... e.g. sdg1)

 

which resolved most of the disks ... and in a couple cases the full rebuild command was necessary:

 

reiserfsck --rebuild-tree --scan-whole-partition  /dev/sdX1

 

 

There are a couple other options that may be necessary (rebuild the superblock for example) ... but as I've noted a couple of times, first do the basic check (be sure you specify the partition #) and see if that resolves your issue.

 

[ajgoyt]

...  I am really getting wooried that all my data pictures etc are Gone.

 

Do you not have backups of your data??  [You're not alone, but it always amazes me how many folks don't bother to backup and then wonder why they lose data.  My thoughts are detailed here:  http://lime-technology.com/forum/index.php?topic=31020.0 ]

 

Notwithstanding that, if these were all Reiser-formatted drives, the Reiserfsck utility is VERY good at recovering data that would likely be toast in most other file systems.    I would, one-at-a-time, run Reiserfsck against the problematic drives ... and copy all recovered data to another drive.    Search the forum for Reiserfsck and you'll find excellent instructions on using this from some of the Linux experts.    I've never personally used it [No need as I have superb backups] ... but it often amazes me how well it can recover data from corrupted Reiser disks.

 

The basic initial command you want to use is:

 

reiserfsck --check /dev/mdX  (where X is the disk # you want to check)

 

Then, depending on what it finds, you'll need to use more advanced commands ... possibly rebuilding the file system tree structure.    Ask here for more detailed help ... or via e-mail to Limetech.

 

In reviewing the thread I see you ran this on at least one of the disks earlier in the process -- which options did you use for it?    And are you sure you ran it on the correct device?  I believe for some actions you have to also specific the partition # ... but as I noted, I am NOT an expert on using this.

 

Gary - not sure where i would back up almost 24 tb of data - run another unraid machine?  - I bought into unraid for redundancy but i appreciate the comments i really do and the help. (reading your page about backups now) So being moderate Linux command user - i noticed on some threads you can perform this check in maintenance mode, but then it switches you over to commands from the keyboard. I have reviewed the wiki and the search forum and i am not seeing a clear cut step by step way to recover....  can you point me to the specific topic or topics you were seeing? 

 

AJ 

[RobJ]

Usually (almost always) when there are multiple drives that are suddenly inaccessible, there's nothing at all wrong with the drives, but a problem with a controller.  So please don't do anything drastic with the drives until you know the full situation.  Check to see what controller the affected drives are connected to.

[ajgoyt]

Usually (almost always) when there are multiple drives that are suddenly inaccessible, there's nothing at all wrong with the drives, but a problem with a controller.  So please don't do anything drastic with the drives until you know the full situation.  Check to see what controller the affected drives are connected to.

  I am in total agreement Rob , before i switched back for a 1/2 day back to the old mb and all my pci addon cards i had the issue with the old board and all those addon cards, Now i am going for it with the new MB with NO addons just all the suspect drives hooked up directly to the new MB. Now to prove my point that the new MB is fine or the PSU i could as a test just take one or more of the Good drives and install it into the mb as a test to prove that the new mb controller is fine...  I just want to get my data back and ill go buy bigger drives to transfer to if i have to. I did do a serach of the wiki etc and it's somewhat clear of the approach of rebuilding / repairing the drives, So again being a moderate linux command user (lightly) do you have a clear process to follow?   

 

thanks

aj

[garycase]

Usually (almost always) when there are multiple drives that are suddenly inaccessible, there's nothing at all wrong with the drives, but a problem with a controller.  So please don't do anything drastic with the drives until you know the full situation.  Check to see what controller the affected drives are connected to.

 

I had the same thought ... but if I've read this correctly he's moved all of the "unmountable" drives to the motherboard controller and has no add-in controllers  (per Tom's suggestion via e-mail).    And he had the same issue using the motherboard ports on his old system as well.

 

But it certainly wouldn't hurt to connect these drives, one-at-a-time, to a spare SATA port or USB bridge device on a Windows PC and see if you can "see" the data okay using a Linux IFS, such as the free LinuxReader [  http://www.diskinternals.com/linux-reader/ ]

 

[ajgoyt]

Usually (almost always) when there are multiple drives that are suddenly inaccessible, there's nothing at all wrong with the drives, but a problem with a controller.  So please don't do anything drastic with the drives until you know the full situation.  Check to see what controller the affected drives are connected to.

 

I had the same thought ... but if I've read this correctly he's moved all of the "unmountable" drives to the motherboard controller and has no add-in controllers  (per Tom's suggestion via e-mail).    And he had the same issue using the motherboard ports on his old system as well.

 

But it certainly wouldn't hurt to connect these drives, one-at-a-time, to a spare SATA port or USB bridge device on a Windows PC and see if you can "see" the data okay using a Linux IFS, such as the free LinuxReader [  http://www.diskinternals.com/linux-reader/ ]

  I am right there with you i took the smallest unmountable drive a 300gb and hooked it up to my win 10 pc and it made my pc hang on boot, another issue ...  ok so remembering an old trick i downloaded knoppix live DVD burned it and booted the win 10 machine and that didnt work out per i am not to familar with knoppix and it was in German and couldnt get it switched over to english....  I still have the drive hooked up and windows 10 - doesnt even see it from disk management, so not sure if the free linux reader will see the drive? 

[garycase]

... Gary - not sure where i would back up almost 24 tb of data - run another unraid machine?  - I bought into unraid for redundancy ...

 

Redundancy is NOT a backup ... it just provides a higher uptime %, since a single disk failure will allow the system to continue to run.    But system errors can still cause problems ... and if something gets corrupted, you simply have a fault-tolerant corrupted disk.

 

As for backing up 24TB ... I have over twice that amount and it's all VERY well backed up  [see my notes in the backup thread I wrote].      A 2nd server for backups is a reasonable way to do it => it's not even mandatory (although I'd recommend it) that your backup server be fault tolerant (i.e. you could save the cost of a parity drive).  If your data is relatively static, you can simply keep a "current backup" disk attached to your main PC, and copy everything to it that you copy to the server ... replacing the disks as they get full and storing them in a safe place (mine are all in a data-rated, waterproof, fireproof safe).    I use mostly older disks that I've either replaced with larger disks or replaced due to pending sectors or too many reallocated sectors for my backups, so that added cost is really fairly nominal.

 

 

... I have reviewed the wiki and the search forum and i am not seeing a clear cut step by step way to recover....  can you point me to the specific topic or topics you were seeing? 

 

By far the best way to run reiserfsck is from the command line on an attached console (keyboard and monitor).    That's definitely how I would do it.    The thread I quoted above that shows the specific Reiserfsck commands is here:

http://lime-technology.com/forum/index.php?topic=43674.msg418455#msg418455

 

[That link is to the post where he details the commands he ran -- you may want to read back a couple pages to see some of the other steps that were tried.]

 

 

 

[ajgoyt]

Usually (almost always) when there are multiple drives that are suddenly inaccessible, there's nothing at all wrong with the drives, but a problem with a controller.  So please don't do anything drastic with the drives until you know the full situation.  Check to see what controller the affected drives are connected to.

 

I had the same thought ... but if I've read this correctly he's moved all of the "unmountable" drives to the motherboard controller and has no add-in controllers  (per Tom's suggestion via e-mail).    And he had the same issue using the motherboard ports on his old system as well.

 

But it certainly wouldn't hurt to connect these drives, one-at-a-time, to a spare SATA port or USB bridge device on a Windows PC and see if you can "see" the data okay using a Linux IFS, such as the free LinuxReader [  http://www.diskinternals.com/linux-reader/ ]

  I am right there with you i took the smallest unmountable drive a 300gb and hooked it up to my win 10 pc and it made my pc hang on boot, another issue ...  ok so remembering an old trick i downloaded knoppix live DVD burned it and booted the win 10 machine and that didnt work out per i am not to familar with knoppix and it was in German and couldnt get it switched over to english....  I still have the drive hooked up and windows 10 - doesnt even see it from disk management, so not sure if the free linux reader will see the drive?

  Installing Linux reader now! will advise...

[garycase]

If connecting the drive to a SATA port is causing boot issues, try using a USB bridge device (assuming you have one) ... you can then plug it in after the system is booted.

 

[ajgoyt]

If connecting the drive to a SATA port is causing boot issues, try using a USB bridge device (assuming you have one) ... you can then plug it in after the system is booted.

its booted now and disk management sees it but my computer doesn't show it, probably per theres no assigned number or name, ok got linux reader installed and it shows the 300g drive but i get errors trying just to open it - so i tried to mount it and its looking for something i am not sure? see the pix... i guess i better read up..

[RobJ]

I can't give advice without knowing what the true situation is, and right now it's really confusing.  Earlier, your diagnostics were requested, could you provide that now, and also (if you have it) the diagnostics or syslog from the previous system before this motherboard upgrade.  What you provided above is only a syslog, from a new motherboard with only 6 drives attached (Disks 1, 2, 3, 9, 12, and 13), the same ones that are now showing as unmountable, but your other disk controllers were not installed (or the other 8 drives).

 

That syslog shows all 6 drives with severely corrupted file systems, possibly ReiserFS.  Any chance you had converted drives to XFS?  Did you boot with the same unRAID flash drive as before?  And do you have a backup of your flash drive, one that's before the upgrade?  I don't know of any process or mistake that could have corrupted so many drives simultaneously, which is very confusing.  I have to wonder if this is a bad motherboard or a bad power supply or bad memory.

 

When you built this system, did you check it out first, run several passes of a Memtest, make sure the power supply is good, and sufficient for the new motherboard and added cards (graphic and disk controllers)?

[ajgoyt]

I can't give advice without knowing what the true situation is, and right now it's really confusing.  Earlier, your diagnostics were requested, could you provide that now, and also (if you have it) the diagnostics or syslog from the previous system before this motherboard upgrade.  What you provided above is only a syslog, from a new motherboard with only 6 drives attached (Disks 1, 2, 3, 9, 12, and 13), the same ones that are now showing as unmountable, but your other disk controllers were not installed (or the other 8 drives).

 

That syslog shows all 6 drives with severely corrupted file systems, possibly ReiserFS.  Any chance you had converted drives to XFS?  Did you boot with the same unRAID flash drive as before?  And do you have a backup of your flash drive, one that's before the upgrade?  I don't know of any process or mistake that could have corrupted so many drives simultaneously, which is very confusing.  I have to wonder if this is a bad motherboard or a bad power supply or bad memory.

 

When you built this system, did you check it out first, run several passes of a Memtest, make sure the power supply is good, and sufficient for the new motherboard and added cards (graphic and disk controllers)?

Rob your spot on and following me and my situation well, Yes i did back up my flash drive before i tried anything and have it saved, earlier way earlier i was trying to save drives with linux commands and it just was taking to long and it looked dismal so i stopped the process , geez for all i know that might have corrupted the 6 drives. 

 

Now for the MB i built it up installed it (parity check was only a week away and nothing was saved recently that i wouldnt care about after the parity drive check, No i didn't run a mem test i just went for it.. The power supply is lepa Bon BM1000 and its been running great for 6 months, I really think my issue all started with the 4 port  sig PCI express , Not pci express 1 type, again This model of  controller SIIG - sata 6gb/s 3i + 1 ssd hybrid PCI E  i installed into the 3rd PCI express slot of my Gigabyte  MB could never see any of the drives, either because this addon card doesn't like this board or the drives were already either being corrupted or ?? maybe the drives weren't the 6gbs type but either way you would think that the card/ drives would be backwards compatible.

 

After a lot of confusion why i couldn't see the drives i started experimenting moving other drives to that addon card and it all started to fall apart. I am going to take the addon card back to Frys.  On a good note for grins i installed easeus data recovery and it's seeing the drive now in Windows 10 through that program and its scanning and finding files   So I believe i could be on the recovery road (fingers crossed)  see the picture and thanks to you and Gary for replying and caring about my situation. I will advise later on the recovery efforts, As a side note can you or Gary recommend a good 4 or 8 port PCI express addon card , it has to be the old smaller PCI express slot? 

[garycase]

As Rob noted, it's not real clear exactly what you've done overall.

 

Your first post indicated that when you went back to your old motherboard a parity check was started ... this may have corrupted parity if you had some disks not being read correctly.

 

A later post showed a rebuild in progress on disk #1 -- not clear if that finished and whether it resulted in a readable disk or not.

 

It's fairly clear that you then created an UnRAID system with ONLY the problem disks in it (per Tom's advice via e-mail) ... but that the disks were still being shown as unmountable [=> it can't find a good file system on the disks].

 

What's not at all clear is how the original file system became corrupted ... this could indeed have been due to bad memory; poor connections; or inadequate power on the new system; a missing power connection (e.g. the 4 or 8 pin CPU Aux power connector); etc. when you first started it.

 

At this point it seems likely that the damage has indeed been done.  If the disks were still okay you should easily see the file structure with LinuxReader on your PC.   

 

It's not going to help fix the corruption, but one thing I would definitely do before anything else at this point is boot to MemTest on the UnRAID box and let it run for a couple hours.    You simply choose MemTest from the initial boot screen when the flash drive starts to boot.    This is a reasonable test of both memory and the CPU (since it's running the test) ... and if it's error-free after 2 hours you should be fine.    [i run it on new systems for 10-12 hours overnight]

 

Assuming the system is otherwise okay, it's then time to, one-at-a-time, start doing a Reiserfsck to see if you can recover the data from the disks.

 

 

[garycase]

I presume you're looking for a PCIe x1 card -- NOT a PCI card -- is that correct?

 

Does your motherboard not have any PCIe x4 or larger (x16) slots that aren't used?  Perhaps a video card slot that you're not using?

 

BTW, when I see files you're recovering titled "Kennys 6th Birthday" and a lot of pictures from years ago, I find it amazing that you didn't at least backup your family memories !!    I hope you are able to recover all of these -- and that you learn the lesson about backing up data you don't want to lose.

 

[ajgoyt]

I presume you're looking for a PCIe x1 card -- NOT a PCI card -- is that correct?

 

Does your motherboard not have any PCIe x4 or larger (x16) slots that aren't used?  Perhaps a video card slot that you're not using?

 

BTW, when I see files you're recovering titled "Kennys 6th Birthday" and a lot of pictures from years ago, I find it amazing that you didn't at least backup your family memories !!    I hope you are able to recover all of these -- and that you learn the lesson about backing up data you don't want to lose.

I understand and advice taken on the prevoius post and will run a memtest for tonight etc ....  Yes i do have all the pictures backup i just remembered on dropbox and another external drive outside the array....   

 

I have attached a picture of the MB installed yes it does have 3 pci express 16 slots free and - 1 PCI slot  - and 3 pci express slots , and then 6 internal slots on the board itself, after this disaster i don't feel comfortable using another PCI express 16 slot would you? sorry just a little paranoid per that's how i believe all of this BS started. So with 6 built in slots on the MB and 3 open PCI express 1.0 slots ( the older style) if i could find 4 port cards between the MB internals and the 3 older pci express slots i could have 18 drives if need be,Right now i have 15 drives but some of those drives are probably getting ready to fail and are smaller drives. As any Unraid person does they typically upgrade in size as they fail, so in theory if i started to buy 6tb drives i could cut down on the amount of drives.... but would have room to expand if need be.

 

couldn't crop the picture of the board to fit on the forum so here is a link https://www.dropbox.com/s/u91kx7j2wubbvyy/2015-12-01%2022.38.01.jpg?dl=0

 

aj

[garycase]

I would buy an 8-port PCIe x4 or x8 card and install it in one of the x16 slots.

 

The Dell Perc H310's are well thought of by many users on the forum:

http://www.amazon.com/3P0R3-Controller-PCI-E-mini-SAS-PowerEdge/dp/B00ZSXK1YO/ref=sr_1_1?ie=UTF8&qid=1449040794&sr=8-1&keywords=perc+h310

 

[Available used for less on e-bay]

 

[ajgoyt]

I would buy an 8-port PCIe x4 or x8 card and install it in one of the x16 slots.

 

The Dell Perc H310's are well thought of by many users on the forum:

http://www.amazon.com/3P0R3-Controller-PCI-E-mini-SAS-PowerEdge/dp/B00ZSXK1YO/ref=sr_1_1?ie=UTF8&qid=1449040794&sr=8-1&keywords=perc+h310

 

[Available used for less on e-bay]

 

Wow ok,  i can try it and it should have better transfer rate then a 1.0 slotted card and other choices ?  Also see the picture it found 13k files on the 300g drive i first scanned...  also i am running the server on memtest and its at 40% of test 9 with no errors...

[ajgoyt]

Still Backing up ...    - Just bought this - http://www.ebay.com/itm/261753320440  and these cables - http://www.amazon.com/gp/product/B008VLHTA4?psc=1&redirect=true&ref_=oh_aui_detailpage_o00_s00  for the rebuild, Which the way its going with the backup will be finishing up next week.....