OK, I've been having several problems with my unRAID system and am working one step at a time to get it back and stable.

 

Background: I've been running unRAID for about two years now. The first 18 months went by without any problems, but then the server filled up. Upon a suggestion from this forum, I added a 4 drive Sans Digital external enclosure and dropped four 4TB HDDs in it. It ran alright for a month or so, at least long enough to get 7TB of data on the drives. From that point on, I continuously received data errors off of the drives in the enclosure. I ran SMART reports on the drives, and the drives were always fine. I've tried to fix whatever is wrong with the bay, to no avail. I've replaced the drives, I've replaced the controller card, I've replaced cables, I've examined internal components; but I always get the errors after several hours of having the server up and running. Sometimes if I'm lucky it will go for a few days before reporting errors.

 

In order to solve this, I am now replacing some 3TB drives in the main server with new 6TB drives (where were these 6 months ago?). I pre cleared the first of the 6TB drives and replaced the parity disk with it. I booted up the machine, started the parity sync, and BOOM, a drive failed from the external enclosure. The parity sync stopped, and now I have two invalid disks. I still have the original parity disk, but am unable to replace it as that would be two missing disks coupled with the other failed drive. So my question is, where do I go from here?

 

How can I reset the configuration to read the old parity disk long enough to re-enable the failed drive?

 

Once this is complete, is there any way to sync the new parity drive without being dependent on the drives in the external enclosure to not give any errors?

 

Once the new parity drive is synced, and a 3TB drive is replaced with a 6TB drive, what is the best way to move data over and decrease the number of drives in the array? My plan was as follows:

 

1) Replace one 3TB drive with 6TB.

2) Move data from 4TB external enclosure drive to new drive.

3) Replace one 3TB drive with newly cleared 4TB drive.

4) Move data from 4TB external drive to newly cleared 4TB drive.

5) Repeat steps 3 & 4 and all the data will be on the main rig and I can completely remove the Sans Digital enclosure (and go all Office Space on it).

 

Thanks in advance.

Is this a port-multiplier case?  [i.e. does it have a single SATA connection to the PC with an internal port multiplier to allow 4 drives internally?]

 

If so, and if you have ample SATA ports in your UnRAID box, I'd simply hook up the drives independently, since the case clearly isn't working well.    Since your plan is to get rid of it, it's not worth the hassle to try and troubleshoot that particular issue.

 

Just run SATA cables to your drives -- sit them on a shelf behind the PC if necessary -- and either run power from the PC to them, or use a separate power unit for provide the drive's power (a spare power supply would work).

 

As for attempting to reconstruct your failed drive .... the first thing I'd do is see if the drive is actually bad, or if this is just another issue with the external box.    Remove the drive;  attach it to a PC's SATA port; and install the free LinuxReader.  See if you can "see" and read all of your data.  [ http://www.diskinternals.com/linux-reader/ ]

 

If the drive is actually good, I'd be inclined to simply do a New Config with only internal drives (6TB parity; 6TB data; 4TB data drive; etc)  and then copy the data from the unassigned drives to the new array from your PC (using LinuxReader to read the data).

 

If the drive has actually failed;  then you'll need to proceed more cautiously.  As I noted above, I'd attach the drives from the external case directly to SATA ports in the PC if possible.    Then do a New Config using your OLD parity drive ... and choosing the "Trust Parity" option.  DO NOT do this if there have been ANY writes to the system since you removed the drive.    If that's the case, you simply can't do a drive rebuild, so you'll need to re-copy the data from that drive to the array from your backups (or if you don't have backups, just copy what you can access from LinuxReader and the rest will be lost).

 

 

 

Yes, port multiplier case.

 

Hooking up the drives independently is an interesting idea, and something I hadn't considered. I have the ports and power connections for it, so that could definitely work.

 

I'm confident the drive is good. The errors were very similar to what has happened every other time the drives have failed (started the array, and it immediately showed 100+ billion reads on 3 drives and each drive had the exact same number of errors).

 

Even if I hook the drives up independently, the array will still have two drives with errors. It shows one of the drives from the external enclosures needs to be rebuilt, and the array is looking for the 6TB drive for parity, but the parity sync was never completed. How do I get past this? Can that be done through New Config?

 

EDIT: Am reading up on New Config now as I have not used this utility previously. If I'm reading this right, and using the suggestion above, what I need to do is the following:

 

Unassign the drives in the external enclosure and disconnect from array

Place new 6tb drive in the parity slot

Apply New Configuration to rebuild parity

After Parity is synced, replace one drive in the main case with a new 6tb drive

Connect old drives from external enclosure to another computer and move the files from PC to unRAID using LinuxReader

Repeat until all data is moved over?

 

Please verify if this is correct.

 

 

The key thing is confirming that your drive is good ... and it sounds like you're confident of that.

 

Given that, then I'd Stop the array, click the New Config button; and then just shut down.

 

Now configure the drives the way you want for the new configuration (move them larger drives into your UnRAID box, etc.).    Then boot and assign the drives for the new configuration -- 6TB parity, 6TB data, and any of the old drives you want to keep (i.e. the 4TB units).    Start the array and let it do the parity sync.  When that's done, do a parity check to confirm all went well.

 

... at this point, you've got your new configuration; a 6TB parity drive; 6TB data drive; and whatever else you included in the configuration.    Note there was no need to use a drive in the new configuration that you planned to replace -- just use the 6TB drive instead, and then (after the system's configured and fault-tolerant) just copy the data from the old drive you wanted to replace.

 

Now connect all drives that aren't in the new configuration, one-at-a-time, to a PC and copy the files from PC to UnRAID using LinuxReader.

 

Basically that's what you just outlined, except I'd just put the new 6TB data drive in the new configuration and copy the data from the drive it's replacing later instead of including a drive in the config that you know you want to replace.

 

I think I understand what you're saying, I'm just a little paranoid about completely resetting everything, but I'm giving it a shot.

 

One more question. Rather than hooking up each drive externally, do you see any problems with putting the data drives in the external enclosure and hooking that up through esata to a laptop in order to move data back to the array? Doing this, I'm obviously betting on the enclosure working correctly, and all my errors simply being compatibility issues. I'm just thinking of doing this as a plan A, and then if that doesn't work, resort back to the original plan of hooking the drives up independently.

Connecting the enclosure to a laptop with all of the non-array drives in it should be fine, assuming the laptop's eSATA connection works okay with the port multiplier (you'll know that easily enough when you try it).

 

Certainly easier than doing one drive at-a-time.

 

Simply starting over with a new config is nothing to be paranoid about PROVIDED you're confident the drives are good.    If you had a known-bad drive, and no backups for it, then I'd suggest a different approach that would at least let you try to reconstruct the bad drive.

 

But that doesn't appear to be the case, so I'd just jump in head first and do the new config with your 6TB drives in place;  then copy those drives not in the new array across your network.

 

ok, both 6tb drives have been pre cleared. I shut down the server, replaced three of the 3tb drives with the 4tb ones that were in the external case. Now the system won't boot. It will run through the SATA drive checks, but then I just get the flashing curser where I would normally have the option to start unraid.

 

A couple observations:

Normally on startup, the indicator lights for each drive would all be on (both green and blue). Now, they all show a blue light, but the drives that are attached to the mobo do not show a green light, while all the others do.

mobo: ASRock 880GM-LE FX

Norco SS-500 hot swap bays

SATA Cards: SUPERMICRO AOC-SASLP-MV8, SYBA SY-PCI40010

 

I've pulled up the boot menu on the mobo and think I have it set correctly to boot from the USB. It's been two years since I last had to do this, but I am a little skeptical that it is setting it to boot correctly. It pulls up a list of bootable drives and the USB drive is in the middle of the list. I highlight the USB, hit enter to select, but then it just goes to the blinking curser. When I pull up the boot list again on the next restart, the USB drive is still in the middle of the list. I thought this list would be indicative of the boot order, and the USB should be on top, But I'm not 100% sure on this.

Delete all other entries in the list. Leave only the USB drive..

Well I got it up and running, but through a workaround. I'd like to get some other input as to whether this should be a permanent solution.

 

My motherboard has 6 SATA ports on it, and I had 3 open ones on the controller card. I moved 3 cables from the motherboard to the controller, and it booted right up that time. Is this normal, or is something screwy with the motherboard that I should be looking into?

 

Right now it is running parity on the new 6tb drive. There are two unformatted drives (old parity, and new 6tb). Should I format those now, or wait till the parity sync completes?

 

I'm also getting a lot of Reiserfs errors from another drive (syslog below). Should I just let parity sync complete before messing with anything further? We're looking at about 3 days before that finishes.

 

I have access to the media share through windows explorer, but it's only showing a fraction of the data at the moment. Will that be complete once parity sync finishes?

 

/usr/bin/tail -f /var/log/syslog

Oct 12 03:25:53 Tower shfs/user: shfs_readdir: fstatat: S01E03.iso (13) Permission denied

Oct 12 03:25:53 Tower shfs/user: shfs_readdir: readdir_r: /mnt/disk12/Media/TV Shows/Orange is the New Black (13) Permission denied

Oct 12 03:25:53 Tower shfs/user: shfs_readdir: fstatat: S01E03.iso (13) Permission denied

Oct 12 03:25:53 Tower shfs/user: shfs_readdir: readdir_r: /mnt/disk12/Media/TV Shows/Orange is the New Black (13) Permission denied

Oct 12 03:25:53 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 3

Oct 12 03:25:53 Tower kernel: REISERFS error (device md12): vs-5150 search_by_key: invalid format found in block 628051208. Fsck?

Oct 12 03:25:53 Tower kernel: REISERFS error (device md12): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [6 8 0x0 SD]

Oct 12 03:25:53 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one 3

Oct 12 03:25:53 Tower kernel: REISERFS error (device md12): vs-5150 search_by_key: invalid format found in block 628051208. Fsck?

Oct 12 03:25:53 Tower kernel: REISERFS error (device md12): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [6 8 0x0 SD]

My motherboard has 6 SATA ports on it, and I had 3 open ones on the controller card. I moved 3 cables from the motherboard to the controller, and it booted right up that time. Is this normal, or is something screwy with the motherboard that I should be looking into?

 

It's almost certain the reason it wouldn't boot with the drives connected to the motherboard ports;  but would with them connected to the add in card; is that the boot order was incorrect.  It was trying to boot from the hard drives instead of your UnRAID USB flash drive.

 

You didn't need to move the drive connections -- you just needed to change the boot order in the BIOS 

 

... if you don't have a keyboard and monitor connected to the UnRAID box, you'll need to do that to fix that issue.

 

 

Right now it is running parity on the new 6tb drive. There are two unformatted drives (old parity, and new 6tb). Should I format those now, or wait till the parity sync completes?

 

Wait for the sync to finish.    Either way will work, but it's quicker to do them sequentially.

 

 

I'm also getting a lot of Reiserfs errors from another drive (syslog below). Should I just let parity sync complete before messing with anything further? We're looking at about 3 days before that finishes.

 

Yes, wait until the sync's done before doing anything else.    Something's not working right, however, if it's showing 3 days for the sync.  Even with a 6TB drive, it shouldn't take nearly that long.    UNLESS your add-in SATA card is PCI -- and if that's the case things will be much quicker once you move the drives back to motherboard ports.

 

 

I have access to the media share through windows explorer, but it's only showing a fraction of the data at the moment. Will that be complete once parity sync finishes?

 

You should see all of the data on your share UNLESS some of the data is on a drive you don't currently have included in the configuration.    But, as with the other items, you really don't want to "mess" with this until the parity sync completes.

 

 

I have the monitor and keyboard setup at the terminal and was finally able to get the boot order fixed to boot from USB, but it still gave the blinking curser. Moving the cables were what got it to finally boot. Will troubleshoot some more once parity sync is complete.

 

The SATA card is PCI. At the time, that was the only option and it was only being used for one drive. Will look into reconfiguring shortly after getting this back and running.

 

I'll let the parity sync complete before attempting anything else. It's showing 4200 minutes to go. Is it possible I just need to run new permissions once this is complete in order to be able to view all the files? Right now there are only a fraction of the files showing.

When the parity sync finishes, connect the drives back to the motherboard SATA ports, and then look carefully at the boot order settings.    It's almost certain that's the reason it's not booting from the USB flash drive.    If your BIOS supports a hot-key for boot selection, press that and force a boot from the USB flash drive -- not a permanent solution, but it will let you get back to UnRAID.

 

A PCI card explains why this is running so slow.  Three (or possibly 4) modern drives on a PCI card are a very notable bottleneck, and will slow everything down in a parity check, since all of the drives are being accessed at the same time.

 

I really have no idea why you're not seeing all your files.  Hopefully it's just a permissions issue; but it could be related to the Reiser issues you're seeing.    Did you by any chance write to this array while running v6 Betas?    There were 2 or 3 Betas that had a MAJOR Reiser corruption issue ... I forget the exact numbers (I think it around 7 & .

 

In any event, just be patient and wait for the parity sync to finish.  Then I'd change the SATA ports and get the boot order fixed so it boots okay.    And THEN I'd look into the other issues.

 

I tried the hot key selection previously and it just did the blinking curser. I'm wondering if something on the motherboard could be defective. I had the boot order set so that it would boot directly from USB, but that did not work until I unplugged those three connections and plugged them in on the PCI card. I've also had an issue recently where the keyboard would just quit responding. I can still get this exact same motherboard for $50, so I might order that just to have it ready when the parity sync is done.

 

And last night, I lost access to the WebGUI. I can still look at the Tower share through windows explorer, but the WebGUI is not available and MargaritaToGo gives a connection error. I guess I'll just need to monitor the lights on the hard drives to know when parity sync is complete?

 

The only version of unraid I have used is 5.0-r16c

The only version of unraid I have used is 5.0-r16c

 

This doesn't have any impact on the booting issue, but I'd certainly upgrade to the v5 final release before troubleshooting the other issues.

 

As for the boot problem -- replace the motherboard battery.  It's only a couple bucks, and a bad battery can result in a variety of strange issues.    Not sure that has anything to do with your problem; but it's simple to eliminate as a potential cause.    I'd certainly try that before buying another board ... although $50 for a spare is certainly not a bad price.

 

Battery replacement was the first thing I tried and didn't help. I think I might look at a new board though, to see if there's anything that would be a better fit than one that's been out for 2+ years. Will have to research this evening to see what's compatible with the hardware I'm already using. Also looking at the SYBA SI-PEX40064 SATA card to replace the PCI card and go into the now open PCI-E x1 slot (http://www.newegg.com/Product/Product.aspx?Item=N82E16816124064).

 

So, I'm still operating blind on the parity sync. No webGUI access and MargaritaToGo app still doesn't connect. This morning though, the green access lights were only on on 6 of the 15 drives. Should I be concerned at all? The terminal monitor only shows a bunch of errors for drive 12. No prompt to enter any commands.

 

Appreciate all the help

... This morning though, the green access lights were only on on 6 of the 15 drives. Should I be concerned at all?

 

Depends ... but most likely this is fine.    If you have 6 drives that are larger than all the rest of the drives; then it's normal that once they're the only ones still "involved" in the parity check, all of the other drives will spin down after they're not involved anymore and the appropriate spindown time has passed.

 

If that's not the case, then something's wrong -- but it's almost certain that's what's happening.  It does, however, give you a clue as to where in the parity check the system is => you know it's past the point of the largest drive not counting the 6 still active.

 

Parity Check is completed, so now it's time to troubleshoot the other issues.

 

1. I can only view a small number of the files through the user share. If I go into each individual disc, I can view all the files on that disc but I can't see them from windows explorer, or through the "Shares" section of the webGUI. ex. My "ISO" folder is completely empty. I have over 1200 ISO files though. If I click on each disc, either through windows explorer or the webGUI, I can view the files, but there is nothing in the main "ISO" folder.

 

I ran the new permissions utility, reset the router, and restarted the server as well as the PC, but nothing changed.

Just to be sure ... on the Settings page, under Share settings -- are user shares enabled?

 

User Shares are enabled

 

Could I just go through each drive individually in windows explorer and move all the files into their appropriate server folder? Seems a bit laborious, but might be the easiest thing if that would work.

 

I'm still getting a lot of reiserfs "read locked....i/o failure" on drive 12. What would be best here? Drop the drive from the array, reconfig, then add a new drive?

With User Shares enabled, then all top-level folders on each drive should automatically be combined into a share.

 

e.g. if disk1, disk2, disk3, etc. all have folders named "Movies", then there should be a Movies share that combines all their contents.

 

If this isn't working, you need to let Tom know.

 

BTW, what does the "Shares" tab list for your shares?

 

Shares tab lists "Media" and "lost+found"

 

Each drive has a "Media" folder as the top level folder. I can view "Media" through windows explorer. The problem is with the subfolders. I can view all of them, but they haven't combined all the files within the subfolders. I can click on the individual folders from the main page and view everything, but they aren't included in the shared folder.

 

I ran Reiserfsck on drive 12, which was giving the read only errors. It responded with the rebuild-sb command. Could this have anything to do with the reason the folders aren't working. That drive doesn't have a lot of data on it. Can I just transfer the and run a pre clear on it again?

Subfolders aren't shares.    If your share folders have somehow been moved to the next level, you need to move them back to the root of the drives.

 

e.g. if you have disk1\Movies,  disk2\Movies, disk3\Movies, etc. then "Movies" will be seen as a share.

 

But if you have disk1\Media\Movies, disk2\Media\Movies, disk3\Media\Movies, then "Movies" will NOT be a share.  If you move it to the root on each drive, then it will be seen as a share.

 

Is this what's happening?

 

I think all the folders are correct. My file structure is as follows:

 

\Media

\Media\ISO

\Media\MKV

\Media\TV Shows

\Media\TS

\Media\Dune 10Muse

\Media\DVD

 

Every drive has the "Media" folder, and then some of the subfolders, depending on what files are on the drive.

 

Yes, that looks okay.  If you browse the Media share you should see all of the content in the subfolders as well -- I presume that's the issue you're referring to.    I though you were attempting to browse one of the subfolders (i.e. "TV Shows") and expecting it to act like a share (combining content from multiple disks with that same subfolder) -- which would NOT be the case.

 

Yes, I can view the files through the individual drives. Can't view them through the share, either by windows explorer or webgui.