I've not really touched my server since I installed it several years ago, it's worked flawlessly aside from 2 failed HDDs at different times, and parity rebuilt fine.  My array is 34.5TB made up of 20 x 2TB hard drives, and now that I'm about to start migrating over to 6TB drives, I thought I should upgrade the O/S.  I was on unraid 4.7, and I followed the upgrade instructions.

 

Everything went fine, and I copied my Pro key and the .cfg files (disk.cfg, ident.cfg, network.cfg, share.cfg) as well as my shares folder, into the new config folder.

 

The server rebooted fine, but none of the drives were assigned.  I went through and re-assigned them to the original order, and it came up with a warning that the parity drive would be erased when the array is started, and that parity needed to be rebuilt.  I clicked to start the array before realising I should have just clicked "parity is valid"...but too late.

 

Anyhow...all of a sudden a load of errors are appearing for several disks, and there is no status listed (i.e. temps) for these drives, and I'm wondering if I've screwed my whole array...?  Any ideas?

 

For now I've cancelled the parity rebuild but it might be a moot point  :'(

Feels like things are getting worse...just went to obtain my syslog and I see this error:

 

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 132423680 bytes) in /usr/local/emhttp/plugins/dynamix/include/DefaultPageLayout.php(300) : eval()'d code on line 73

 

When I click log in the main gui the wall of text is this:

 

Oct 17 23:01:10 Tower kernel: md: disk12 read error, sector=2401296
Oct 17 23:01:10 Tower kernel: md: disk14 read error, sector=2401296
Oct 17 23:01:10 Tower kernel: md: disk15 read error, sector=2401296
Oct 17 23:01:10 Tower kernel: md: disk16 read error, sector=2401296
Oct 17 23:01:10 Tower kernel: md: disk17 read error, sector=2401296
Oct 17 23:01:10 Tower kernel: md: disk18 read error, sector=2401296
Oct 17 23:01:10 Tower kernel: md: disk10 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk12 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk14 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk15 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk16 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk17 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk18 read error, sector=2401304
Oct 17 23:01:10 Tower kernel: md: disk10 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk12 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk14 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk15 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk16 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk17 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk18 read error, sector=2401312
Oct 17 23:01:10 Tower kernel: md: disk10 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk12 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk14 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk15 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk16 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk17 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk18 read error, sector=2401320
Oct 17 23:01:10 Tower kernel: md: disk10 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk12 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk14 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk15 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk16 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk17 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk18 read error, sector=2401328
Oct 17 23:01:10 Tower kernel: md: disk10 read error, sector=2401336
Oct 17 23:01:10 Tower kernel: md: disk12 read error, sector=2401336
Oct 17 23:01:10 Tower kernel: md: disk14 read error, sector=2401336
Oct 17 23:01:10 Tower kernel: md: disk15 read error, sector=2401336
Oct 17 23:01:10 Tower kernel: md: disk16 read error, sector=2401336

Download diagnostics before you reboot, then post them here.

 

I'm betting a controller that is common to all the error-ed drives went offline, and the diagnostics should tell exactly what happened.

 

Also, a brief description of your hardware, MB, CPU, RAM, HBA, PSU, etc.

 

BTW, as soon as you start migrating to 6TB drives, I'd convert to XFS from ReiserFS. It would be fairly easy to do at that point, and may save you some headaches. 10K ft overview, install new parity drive and install new data drive at the same time. Build parity with all your existing 2TB drives and one new 6TB, formatted to XFS. Copy 3 old drives to new drive, verify copies, reformat now duplicated old drives to XFS, copy 3 more drives, verify, repeat until done.

Hi, I downloaded diagnostics, but not sure what to attach here?  syslog is too big as an attachment.

Hi, I downloaded diagnostics, but not sure what to attach here?  syslog is too big as an attachment.

The diagnostics zip file as a whole may fit. We want the entire thing anyway.

D'oh!  Didn't think about the zipped file...sorry my brain is a bit scrambled as I'm in panic mode!

 

I've attached it now, thanks for the quick response, appreciated!

tower-diagnostics-20161017-2324.zip

Probably a stupid question, but should I reboot?  Should I rebuild parity with the array as it is?  Should I reboot then rebuild?  Or should I wait to rebuild until my 6TB drives arrive (which should be tomorrow).

 

Thanks

Before you reboot, post a detailed description of your hardware ....

 

motherboard; CPU; memory; add-in controllers you're using; etc.

 

You have 7 drives that appear to have problems -- BUT they were all read fine when the array was started, as it shows the details just fine (drive name, serial #, file system type, size, and amount of used space.    Strange ... but if they are all on the same controller that may at least be a clue as to what's happening.

 

How many 6TB drives do you have coming?    ... and do you have a Windows machine with a spare SATA port available?  You MAY want to do this an entirely different way => but first post the details of your hardware.

 

 

This is the spec of the server:

 

Asus M4A785TD-V-Evo motherboard

AMD Athlon II X2 245 Regor Core AM3 2.9ghz processor

4GB of Corsair Dominator DDR RAM

Corsair 650w PSU

Supermicro AOC-SASLP-MV8 SATA expansion card x2

2 x 3Ware forward SATA breakout cables

San Disk Cruzer Micro 2GB USB drive

USB motherboard adapter

4 x X-Case 5-in-3 hotswap HDD cages

I have 2 x 6TB WD Reds coming tomorrow - I was planning on changing parity to one of them, and then using the other to replace one of the 2TB data drives.

 

I have one of these to make plugging in internal HDDs into my main PC less of a chore:  https://www.amazon.co.uk/gp/product/B00IS7Y96I/ref=oh_aui_detailpage_o00_s00?ie=UTF8&psc=1

Interesting, just stopped the array and it looks like a controller has failed...or didn't start properly?

 

Okay, your motherboard has 5 on-board SATA ports; so I suspect you've got 5 drives plugged in to the motherboard; 8 into one of the MV8's, and 7 in the other MV8 ... and I'd guess the 7 in the 2nd MV8 are the drives that aren't being seen.

 

So ... with power off, reseat both of your MV8's (or at least the one with 7 drives attached; and double-check the cabling between that controller and the drives, as well as the power connections for those drives.    As I noted above, they WERE seen at boot time, since it shows all of the details of the drives ... but for some reason they are all having read issues -- which very strongly points to the controller, assuming they are indeed all connected to the same controller (which seems VERY likely).

 

After you've reseated the controller and double-checked all the cabling, power on and boot and see what it shows.  Do NOT do anything except Start the array and see if it is now working okay (just let the parity check run as long as everything is okay).  If it still doesn't show the temps for those drives (which indicates they're not spun up) or any other error message then DO NOT do anything else.

 

Thanks Gary

 

Just rebooted and they are all showing fine.  Just started the array and they all mounted fine too. 

Looks like you've already confirmed what was pretty apparent -- the controller is not working => but it DID work fine initially, or you wouldn't have seen the details of the drives in your initial display.

 

BTW, your inital display also confirmed that you had assigned all of the disks correctly, since all of the data drives showed Reiser formatting and had reasonable used/free space values.  So your data is all intact -- assuming two things:  (1) that you don't do ANYTHING that might cause writes to the drives until you get this sorted out; and (2) that all was well before you started this process (i.e. did you run a parity check BEFORE you did the upgrade to confirm everything was okay? ... if not, then it's too late to do that, since you no longer have good parity; so any errors on the disks will still be there in the new version -- unless you have backups or checksums that let you validate your files, you'll just have to assume everything's okay.

 

Have you double-checked that the controller is seated correctly and the cables are okay?

 

Thanks Gary

 

Just rebooted and they are all showing fine.  Just started the array and they all mounted fine too.

 

STRANGE ... especially if you didn't reseat anything and just did a reboot.

 

FIRST thing I'd do is let the parity check run to completion => you definitely want to get everything parity protected before doing anything else.    Then you'll need to run the New Permissions utility ... and then everything should be fine.

 

The next question is WHY you had the issue with the controller => it could have been (a) a fluke (one of those never-understood problems that simply go away);  (b)  a failing controller;  or ©  a failing power supply.    I'd be inclined to lean towards © ... but you may want to simply leave everything alone to see if the issue repeats itself.    But FIRST get good parity !!

 

To be safe I've just shut down the server, reseated all of the cables and rebooted again.  In my tower the usb drive is mounted just below the offending card...so I must have knocked it. 

 

Parity was good before I did the upgrade.  The drives were showing fine to begin with, but once I started the parity rebuild the errors appeared.  The rebuild showed at 0.2% before I stopped it and posted on here.

 

Will that have affected any data on those drives, do you think? 

No, it would have only messed up the parity => but a new parity check will resolve that.

 

So let a new parity check complete; then do NOT use the server at all (so nothing changes).

 

... and then replace the parity drive with a 6TB drive when it arrives and do let it rebuild parity.  After that completes, do a parity check to confirm all went well.    The 6TB rebuild followed by a parity check will take a LOT of hours ... but it's best to be absolutely certain all is well before you start doing further array changes.

 

THEN you'll be ready to start the process of upgrading your data drives and (if you want to) converting the file system.  Conversion is not really necessary -- especially on full drives you rarely write to -- but doesn't hurt anything, and as long as you're VERY careful about how you proceed you shouldn't lose anything in the process.  Be CERTAIN you understand the "user share copy bug" ... which can cause complete data loss ... BEFORE you start the conversion process.

 

Note that if you want to upgrade the file system, you should ADD the 1st 6TB drive to the system, using XFS as the file system for it, rather than doing an upgrade of an existing drive, which would maintain the Reiser file system.  Then (being CERTAIN you avoid the user share copy bug) you could copy data from a Reiser disk to the XFS disk; then reformat the Reiser disk;  and proceed to slowly and carefully proceed with the conversion process.

 

 

Thanks so much for your help!  I've left it running overnight and the rebuild is at 33% with no errors so far.

 

Sorry, it's so long since I've done anything with unraid - you tend to forget things when you don't use things you've learnt!  From memory I used to preclear all of my drives, as I was expanding my array.  Now that my array is full, any drives I add will be replacing others, which means as soon as I plug in the new drive my server will be degraded.  Is there a way I can pre-clear my new 6TB drives offline, i.e. plugged into my main PC?

 

Just so I'm clear, the process now is:

 

Let parity rebuild with my existing parity drive.

Run new permissions from Tools.

Preclear 6TB new parity drive and data drive.

Shut down server, swap 2TB parity with new 6TB drive.

Let parity rebuild.

Run a parity check.

 

What's the best way of replacing one of my data drives?  I'm assuming I just swap out a 2TB data drive with the other new 6TB drive, and let it rebuild (although naturally that's risky).  My current server is at capacity though, so it's not easy to move data around, although I could copy one drives entire contents onto my main PC, swap the drive and then I have a proper backup.

 

If you have the space consider adding a 2nd parity, this way array will remain protected during the disk upgrades.

What's the best way of replacing one of my data drives?  I'm assuming I just swap out a 2TB data drive with the other new 6TB drive, and let it rebuild (although naturally that's risky).  My current server is at capacity though, so it's not easy to move data around, although I could copy one drives entire contents onto my main PC, swap the drive and then I have a proper backup.

you can reduce the risk by keeping the replaced disk around until the rebuild has successfully completed.    However as has been noted that unless you have dual parity you are not protected against a different drive failing during the rebuild process.

 

Note that is not necessary to have pre-cleared drives that replace existing drives.  In such scenarios the pre-clear is really just acting as a confidence test of the drive.  Since you have no spare SATA slots you may find it more convenient to simply run the manufacturers tests on a separate Windows machine to act as the confidence check.

... Now that my array is full, any drives I add will be replacing others, which means as soon as I plug in the new drive my server will be degraded ...

 

That's always going to be true with a single parity system.  Only with dual parity do you still have protection against a drive failure while a drive is being rebuilt.

 

 

... Just so I'm clear, the process now is:

 

Let parity rebuild with my existing parity drive.

Run new permissions from Tools.

Preclear 6TB new parity drive and data drive.

Shut down server, swap 2TB parity with new 6TB drive.

Let parity rebuild.

Run a parity check.

 

This process will get you to the state where you have a 6TB parity drive with all of your current data drives ... which is basically the state you want to be at before starting further array upgrades [moving to larger drives; changing file systems from Reiser to XFS; etc.]  There's no need to preclear the new 6TB drive for parity; although it's not a bad idea to test it ... you could do that on your PC using WD's Data Lifeguard.

 

 

... What's the best way of replacing one of my data drives?  I'm assuming I just swap out a 2TB data drive with the other new 6TB drive, and let it rebuild (although naturally that's risky).  My current server is at capacity though, so it's not easy to move data around, although I could copy one drives entire contents onto my main PC, swap the drive and then I have a proper backup.

 

You could indeed swap out a 2TB for the 6TB and let it rebuild.  HOWEVER, the 6TB would then still be Reiser formatted.  There's nothing "wrong" with that -- but if your goal is to move to XFS I'd do it differently.    So the key question you need to answer before starting the migration to larger drives is whether or not you want to switch to XFS in the process of doing that.

 

 

 

 

I'm going to assume you'd like to do two key things in your migration:

 

(1)  Move to dual parity (a good idea with a large # of drives)

 

and

 

(2)  Move to the XFS file system

 

IF those are true, then I'd do the migration as follows:

 

(a)  Get THREE 6TB drives (not just the 2 you have coming)

 

(b)  Remove 3 of your current 2 TB drives -- the parity and two data drives -- and then install the 3 6TB drives.  Do a New Config, assigning two of the 6TB drives as parity drives and the other one as a data drive, and ALL of the rest of your current 2TB drives as data drives.

 

NOTE:  If you want to get a head start on this process, you could do this tomorrow (when your current 6TB drives arrive) with a minor mod => still remove 3 of the 2TB drives (be sure you know which 2 are data drives); and leave a space for the 2nd parity drive; but just assign one parity drive for now (and the 6TB data drive).    You can do everything I'm going to list here and simply add a 2nd parity drive later when you get another 6TB drive.

 

©  Start the array and let it do a parity sync; then do a parity check to confirm all is well.  You will also need to let it format the 6TB data drive -- confirm that it's done in XFS  (v6 defaults to that, so unless you've changed something that's what it will be).

 

At this point you'll have your array ready to use; but you'll have two 2TB data drives on a shelf with data you need to get to the array.

 

(d)  You can now copy the data from the 2 drives you removed to the 6TB data drive by, one-at-a-time, connecting them to your PC, and copying the data across your network to the server to the 6TB drive.  You'll need to use the free LinuxReader [ http://www.diskinternals.com/linux-reader/screenshots.shtml ] to access the data from the Reiser disks on your PC.    Note also that you'll need to enable Disk shares on your UnRAID server so you can "see" the 6TB disk.

 

Once you've copied all of the data from the 2 data drives that are no longer in the system to the new 6TB disk, you'll have all of your original data on the server; it will be parity protected; and the new, larger (6TB) data disk will be XFS formatted.    Fundamentally, your transition to v6.2.1 will be complete.

 

When you get another 6TB drive, you can simply shut down the server; install the drive; boot; assign the new drive as a 2nd parity drive; and Start the array ... and when it finishes computing the 2nd parity drive you'll be dual fault tolerant.

 

There are a variety of ways to convert your remaining disks to XFS, if that's something you want to do.  But what I've just outlined will take quite a few days, so get that done first, and then you can decide about that.    As I noted earlier, there's no real need to convert for disks that are essentially static -- which many of yours seem to be, since they're very close to full.    I'd be inclined to at least wait until you have dual parity AND a 2nd 6TB data disk available ... then you'll have a lot more "working space" available, which makes it an easier process.  Adding a 2nd parity drive will be a better improvement to the system than changing the file system.

 

 

Thanks Gary.

 

I've just ordered another 6TB drive; I have 2 arriving today, and one arriving tomorrow.

 

What are the advantages of moving to the XFS file system?  (it sounds like a lot of work).

 

In my head I still look at unraid like RAID, and I think that if I remove 2 drives it will kill the array...but of course it won't.

 

So I stop the array, shutdown, remove my parity drive and any 2 data drives, reboot and then immediately go to "New config", then setup the array again? (effectively allocating drives from scratch).  I'm assuming I need to setup everything including shares again?

 

 

What are the advantages of moving to the XFS file system?  (it sounds like a lot of work).

Reiserfs is end-of-life and no longer gets much maintenance.  The oriinal author is in jail for murdering his wife so not active any more.

 

XFS is now deemed to be more stable.  It also does not suffer form the issue that Reiserfs has where performance degrades significantly as drives get near full.

 

Having said that the Reiserfs recovery tools are by far the best if you ever get corruption at the file system level.

 

So I stop the array, shutdown, remove my parity drive and any 2 data drives, reboot and then immediately go to "New config", then setup the array again? (effectively allocating drives from scratch).  I'm assuming I need to setup everything including shares again?

You can do the New Config at any time as long as you know exactly which drives are which (using their serial numbers).

 

You should find that your shares (and other setup details) are still there on restarting the array as long as you have not wiped the configuration files from the USB drive.

Another stupid question - when doing a new config, I assume I can just allocate drives to whatever disk number I want?  (so long as I'm clear on what is a parity drive and what is a data drive).