July 25, 20178 yr I have finally decided to migrate my home server from windows 8.1 to unRAID. I have the Pro license, which I bought about a year ago, but never had the time to use it. The reason is - I have 130TB+ in 35 drives on my server now and I'm using Stablebit DrivePool. The biggest problem is DrivePool - it has scattered all my files on all drives. It did that intentionally - that's just how it operates. So for example: Star Trek Enterprise season 1 has 26 episodes - and each and every one of these episode files are on 26 different drives. How stupid is that? And that's how pretty much all 130TB of files are now. So the migration process is gonna be long and tedious. Here is the home server's hardware: Intel Xeon E3-1230 v2 3.3GHz BX80637E31230V2 Supermicro MBD-X9SCM-F-O Kingston 32GB (4x8) DDR3 ECC Unbuffered 1600 KVR16E11K4/32 LSI 9201-16e PCIe 2.0 x8 SATA/SAS HBA LSI00276, to this HBA a have connected 3 of these expanders: SGI Rackable SE3016 SATA SAS Expander 16 Hard Drive Bay and in these expanders I have 35 drives of various sizes: 7x WD Green 2TB 14x WD Red 3TB 14x WD Red 6TB Last week I bought 3 new WD Red 10TB drives. I had an old spare i7 pc, so I put an unRAID 6.3.5 Trial on it. I precleared those 3 new 10TB drives with a preclear plugin. Then added all 3 drives to an unRAID pool, without any parity or cache drives - just a simple 3 drives pool. And then I started moving the data off my windows server into these drives. I move the files using drive shares for now, not user shares (just to keep files together). I know unRAID can do that using user shares to, but it is simpler for now. Later on, when all the data has been moved, I will properly sort my files and will figure out user shares and their split levels, etc Questions: I was so very much pissed of by the DrivePool at this point, and was in so much hurry to start the migration, that I did not think to look this up before hand. Will I be able to simply take these drives, already filed with data and put them on my final unRAID server and add them to the new pool? I mean these drives have been precleared with unRAID plugin, have been formatted by unRAID and I will be adding these drives to the pool which has no parity... Basically, what I wanna do, is take the drives from my windows server in groups of 2 or 3, preclear them, add them to the unRAID pool and move the data of server... repeat... Finally, when all the data is migrated to the unRAID, I will make a new unRAID setup from scratch with my Pro license, create a new pool, add parity and cache drives and only then calculate parity... So is this OK? Can unRAID use drives filled with data from another unRAID system (again - no parity for now)? Copying 130TB of data over 1gig network (I'm getting ~80MB/s atm) is gonna take forever. After doing some thinking, I was wondering - could I put unRAID on my server machine right now and convert my existing windows 8.1 into a virtual machine running on unRAID? All the drives, that are needed by the windows VM would be on the same HBA. Could I pass-through the whole HBA to the windows VM with all the drives, so the DrivePool in windows could function without problems? I need Drivepool running and pooling the drives on windows, because the files are scattered all over the place, and without pooling I could not easily move the files to unRAID in proper file structure... I have a 2nd identical LSI 9201-16e HBA and an identical 4th expander box. I would use these to run the drives for the unRAID pool. These would be the destination drives. Would that be OK? If there's any problems using 2 identical HBAs, I have 2 LSI 9210-8i (IT mode) cards. I could use 1 or both of those. Depends on the answers to 2 to 4. But if I can do this windows in VM on unRAID migration - would I run into some serious problems, down the line? Because I will be removing drives from windows HBA and adding them to unRAID HBA, 2 or 3 drives at a time... This is gonna be my first unRAID and already it is gonna be with max drives - 28 data and dual parity. Not sure about cache yet (single or pool). I am wondering, how long will the parity checks gonna be? I am gonna have only 3 10TB data drives in it, but methinks in 2 to 3 years, all 28 data drives gonna become 10TB, I'm pretty sure. How long the parity checks gonna be then? I mean I've read tha it's recommended to do a monthly parity check... What would be better as parity drives - same WD Red 10TB or should I get WD Gold 10TB? As I understand parity drives are the most important, the most hard working drives and should be as fast as possible, yes? The whole reason to do this vm thing, is to avoid the 1gig LAN bottleneck. I'm not sure, but I think I could get at least 400MB/s transferring between HBAs or maybe even 800MB/s. Never tried it before, so we'll see. Thanks in advance for any answers and help.
July 25, 20178 yr Community Expert Looking at your questions: 1) You will have precleared them so in effect they no longer have data. Actually you do not even need to preclear them unless you want to test the drives - the act of adding them to unRAID will result in any existing data on them being lost. Note, however, that you will not be able to use all your drives on a single unRAID server as it currently has a limit of 28 data drives in the main array (plus 2 parity). You could add additional drives as a cache pool but not sure this will be what you want. Probably time to retire some of the smaller drives? 2). I think this is possible, but I do not think anyone has done it with this amount of data. Have you got backups if things go wrong? 3) This should be possible. However hardware pass-thru to VMs is always a bit system dependent. You could however test this out first to make sure it works OK on your system. 4) It should be possible. Since you seem to have sufficient hardware I would suggest carrying out a trial first. 5) This should not be a problem. I assume you would have 'hidden' the passed through HBA from unRAID so unRAID would not even be aware this is happening. 6) The time for parity is determined mainly by the size of the parity drive. I would guess that you would be talking about something like a day to carry out the parity check on 10TB but others may have a better view. In practise having mixed drive sizes will slow things down slightly as parity is determined by the slowest drive at any point and drives slow down as they get nearer the inner cylinders. With mixed drives sizes this will happen several times. 7) No idea. Any operation tends to be affected by the slowest drive involved so there is no point in using slow drives for parity. Very difficult to know what throughput you would really get - I think you will only get a firm answer by carrying out tests on your proposed system.
July 25, 20178 yr 3 hours ago, shEiD said: I have finally decided to migrate my home server from windows 8.1 to unRAID. I have the Pro license, which I bought about a year ago, but never had the time to use it. The reason is - I have 130TB+ in 35 drives on my server now and I'm using Stablebit DrivePool. The biggest problem is DrivePool - it has scattered all my files on all drives. It did that intentionally - that's just how it operates. So for example: Star Trek Enterprise season 1 has 26 episodes - and each and every one of these episode files are on 26 different drives. How stupid is that? And that's how pretty much all 130TB of files are now. So the migration process is gonna be long and tedious. Here is the home server's hardware: Intel Xeon E3-1230 v2 3.3GHz BX80637E31230V2 Supermicro MBD-X9SCM-F-O Kingston 32GB (4x8) DDR3 ECC Unbuffered 1600 KVR16E11K4/32 LSI 9201-16e PCIe 2.0 x8 SATA/SAS HBA LSI00276, to this HBA a have connected 3 of these expanders: SGI Rackable SE3016 SATA SAS Expander 16 Hard Drive Bay and in these expanders I have 35 drives of various sizes: 7x WD Green 2TB 14x WD Red 3TB 14x WD Red 6TB Last week I bought 3 new WD Red 10TB drives. I had an old spare i7 pc, so I put an unRAID 6.3.5 Trial on it. I precleared those 3 new 10TB drives with a preclear plugin. Then added all 3 drives to an unRAID pool, without any parity or cache drives - just a simple 3 drives pool. And then I started moving the data off my windows server into these drives. I move the files using drive shares for now, not user shares (just to keep files together). I know unRAID can do that using user shares to, but it is simpler for now. Later on, when all the data has been moved, I will properly sort my files and will figure out user shares and their split levels, etc Questions: I was so very much pissed of by the DrivePool at this point, and was in so much hurry to start the migration, that I did not think to look this up before hand. Will I be able to simply take these drives, already filed with data and put them on my final unRAID server and add them to the new pool? I mean these drives have been precleared with unRAID plugin, have been formatted by unRAID and I will be adding these drives to the pool which has no parity... Basically, what I wanna do, is take the drives from my windows server in groups of 2 or 3, preclear them, add them to the unRAID pool and move the data of server... repeat... Finally, when all the data is migrated to the unRAID, I will make a new unRAID setup from scratch with my Pro license, create a new pool, add parity and cache drives and only then calculate parity... So is this OK? Can unRAID use drives filled with data from another unRAID system (again - no parity for now)? Yes. When setting up a new array (I wouldn't call it a drive pool, because that sounds more like a cache pool in unRaid), the contents of the disks are not altered. Since you partitioned and formatted the new disks with unRaid, all of their content would be retained. However, if you already have a parity protected array, adding a new drive would result in unRaid zeroing it out. Quote Copying 130TB of data over 1gig network (I'm getting ~80MB/s atm) is gonna take forever. After doing some thinking, I was wondering - could I put unRAID on my server machine right now and convert my existing windows 8.1 into a virtual machine running on unRAID? Maybe, but I wouldn't try this. Are the disks in the Windows box formatted with NTFS? If so, you could mount the disks in your unRaid box, one or a few at a time, and using the unassigned devices plugin, mount the disks and copy their contents to the unRaid disks. But unless you have hot swap bays and are 100% confident you won't disturb the wiring, I'd probably stick to your slow but sure method. There is a relatively inexpensive 10Gb lan option you could explore. That might be the least invasive. ConnectX2 Quote All the drives, that are needed by the windows VM would be on the same HBA. Could I pass-through the whole HBA to the windows VM with all the drives, so the DrivePool in windows could function without problems? I need Drivepool running and pooling the drives on windows, because the files are scattered all over the place, and without pooling I could not easily move the files to unRAID in proper file structure... Again, these types of VM operations can be complex, involving a fair amount of trial and error. I would not be practicing on a production server that you are trying too extract data from. Quote I have a 2nd identical LSI 9201-16e HBA and an identical 4th expander box. I would use these to run the drives for the unRAID pool. These would be the destination drives. Would that be OK? If there's any problems using 2 identical HBAs, I have 2 LSI 9210-8i (IT mode) cards. I could use 1 or both of those. UnRaid can use multiple controllers of the same type. Quote Depends on the answers to 2 to 4. But if I can do this windows in VM on unRAID migration - would I run into some serious problems, down the line? Because I will be removing drives from windows HBA and adding them to unRAID HBA, 2 or 3 drives at a time... Again, I would not recommend. Quote This is gonna be my first unRAID and already it is gonna be with max drives - 28 data and dual parity. Not sure about cache yet (single or pool). I am wondering, how long will the parity checks gonna be? I am gonna have only 3 10TB data drives in it, but methinks in 2 to 3 years, all 28 data drives gonna become 10TB, I'm pretty sure. How long the parity checks gonna be then? I mean I've read tha it's recommended to do a monthly parity check... Parity check speeds are generally dictated by the size of parity. So a 10TB parity with 5 data disks would take about the same time as a 10TB parity with 10 dates disks. Assuming you are not constraining the bandwidth, and have sufficient horsepower, your parity check speeds with 28 data disks should be similar, but we don't really have good data to know for sure. Quote What would be better as parity drives - same WD Red 10TB or should I get WD Gold 10TB? As I understand parity drives are the most important, the most hard working drives and should be as fast as possible, yes? Parity drive is not the most important. In fact I'd argue the opposite. And reliability statistics generally don't bear out that more expensive disks are more reliable. Finding specific statistics can be difficult. But if you are going with REDS for your data disks, they are fine for parity as well. I prefer HGST, but many here like the REDS the best. The reason I say they are the least important is because it is only when a data drive fails that you are relying on parity for anything. And when a drive does fail, EVERY OTHER DISK IN THE ARRAY is equally important in the recovery. Parity is not special. Quote The whole reason to do this vm thing, is to avoid the 1gig LAN bottleneck. I'm not sure, but I think I could get at least 400MB/s transferring between HBAs or maybe even 800MB/s. Never tried it before, so we'll see. Thanks in advance for any answers and help. Good luck! Data migration is not the best time to experiment with VMs. If you had done it successfully in a test configuration, that might be different. But too much risk with so much data IMO. There might be an option to read and write XFS disks in a Windows environment. If so you might be able to put your unRaid formatted disks in your Windows box and copy the data over. Moving the physical disks to the unRaid box is also reasonable if the drivepool disks can be accessed individuality, and if you have hot swap cages. Have you considered two servers? Given the high drive count I might suggest it, especially as you continue to grow.
July 25, 20178 yr Community Expert My recommendation would be to consider the use dual parity with the number of data drives (eventually) that you are going to need to hold all of your data. That provides full data protection for up two drive failures. Dual parity provides protection against the possibility of having a second drive fail during a rebuilt of another failed drive. If you want to read more about securing of data, you can read through this thread. You could add the second parity drive at any point along your data transfer path. I personally would be looking at getting a few more of those 10TG drives and using some of the old 2TB and 3TB drives in a second backup server (kept offsite-- if feasible) of your files which are impossible to replace. One more thing. There is a way to add a drive filled with data (formatted and written on an unRAID server) to an unRAID parity protected array without loosing the data on that drive. At the time when you are ready to do this, ask the question and someone will help you out. (I know the basic procedure but I have never done it and I would have to do a lot more looking/thinking before I was confident that I had ALL of the required steps.)
July 25, 20178 yr Author Thank you guys very much for the answers, but I think you misunderstood my situation and questions (mainly the most important -1st one). I was really trying to be precise, but probably I wasn't. Let me describe it in more detail: I have no worries about my hardware. Pretty much all of it is enterprise grade and I bought all of it brand new (except the SAS expander cases). The Supermicro mobo and LSI HBAs - these are very reliable and should be supported very well, AFAIK. The SAS expander cases, even though I bought them old and used on ebay - they are also enterprise grade, not some Norco cases. Actually I bought them to replace the Norco case I was using before, and these are way better - only half depth and way better airflow and cooling for the drives. All my drives are in these expanders. And these expanders are connected to the HBA via a single external mini-SAS cable (SFF-8088 to SFF-8088). So basically, whenever I need to add or remove a drive - there is no fiddling with any wires or connections. I do not need to open any cases - neither server nor expanders - I simply remove/insert the tray in the expander and that's it. Also, I never do or trust hot-swapping - I always shutdown the server and that particular expander, to which I am adding or removing drive(s). I'm saying this just to be clear - I am not expecting any "loose or bad connection" troubles during the migration process. And during regular use down the line, probably too. Currently I have 2 servers. Migration source and destination. The source - server #1 is a windows 8.1 machine. This is my current home media server (it runs on the hardware I described). I guess you could call it production server (as in it is not a test machine just to play around - it has all my media on it). yes it has 130TB of data, but 98%+ of it is just media (movies and TV shows). So I do not have backup, but I do not need backups. I am not rich enough to have backups of media files They are pretty easy to replace. The 2% - the important files - they have backups, of course. The destination - server #2 is now running the unRAID 6.3.5 Trial. The singular purpose for this server is to be the destination for files from the windows server (#1) during the migration process. For this reason, this unRAID server does NOT have parity, nor any parity or cache drives. And it will NEVER have parity or cache drives, ever. After I have migrated/moved all my files from the server #1 (windows) to this server #2 (unraid), I will discard server #2. And then I will install a fresh unRAID server on the server #1 (windows be gone). Basically: I will move the files from server #1 to server #2, until I clear 3 drives in server #1. I will then remove those 3 empty drives from server #1 and add them to server #2. I will run preclear plugin on those 3 drives. I use preclear plugin for 2 reasons: 1) it tests the drives and 2) when drives are precleared, adding them to the pool is very fast. Then, after they have been precleared, I will add those 3 drives to the existing unRAID pool (again - the pool has no parity). And repeat this loop with the next 3 drives, until all NTFS drives from windows are in the unRAID. Again - when I move files to unRAID (server #2) - I must be moving files from a windows OS. I cannot simply remove a couple of drives from my windows server and attach them to the unraid server and mount them with something like Unassigned Devices. Like I described before - all my files have been scattered between all the drives by the stupid Stablebit DrivePool. So now, that I want all my files back together in proper file structure on unRAID - I have to be moving them from windows OS, running DrivePool. This and only this enables me to move the folders with the files inside them together. Basically - when I move a folder from DrivePool to unRAID, then DrivePool picks up all the files that are scattered around in multiple drives and combines them. I hope I am making myself clear. And yes - I know unraid supports only 28 data drives in the pool. I will have only 28 data drives. I have described my future unraid pool in my first post. Actually, fingers crossed for unraid supporting multiple pools in the future I will be using dual parity, of course. So, the questions in more detail: Can I make a brand new pool in a brand new, fresh install of unRAID with the drives, that already have data on them? Once again - brand new pool - still no parity. Can I add a new drive with data already on it to the existing pool on unRAID. That existing pool does not have parity yet. I repeat - no parity or parity drives. Basically - when unRAID adds a new drive to the pool - can that drive have data on it or not? I mean - if you add a drive to the unraid pool does unraid always clear and re-format the drive? I hope you can, because other-wise what happens, if you want to simply reinstall unraid from scratch... For example - the unraid flash drive dies... I hope you can use the existing data drives from last unraid pool to create a new one, without moving/copying the data over. Because other-wise it's silly... As for converting my current windows server into a VM running on unraid. I was reading the forums and watching the awesome unraid videos from Spaceinvader One - YouTube. So, basically, I should be able to do it. I was only wondering, if there was any trouble I don't know about, being an unraid newbie, when actually moving files from a windows VM into the unraid pool. If there isn't, it should be good, right? Also, these things you guys said worries me a little. I cherry-picked only the sentences in question: 6 hours ago, itimpi said: Actually you do not even need to preclear them unless you want to test the drives - the act of adding them to unRAID will result in any existing data on them being lost. This sounds like any drive added to unraid pool always looses any data already present on it. Even, a data drive from another unraid server? Seriously? I hope not. 6 hours ago, itimpi said: The time for parity is determined mainly by the size of the parity drive. I would guess that you would be talking about something like a day to carry out the parity check on 10TB but others may have a better view. In practice having mixed drive sizes will slow things down slightly as parity is determined by the slowest drive at any point and drives slow down as they get nearer the inner cylinders. With mixed drives sizes this will happen several times. Any operation tends to be affected by the slowest drive involved so there is no point in using slow drives for parity. Very difficult to know what throughput you would really get - I think you will only get a firm answer by carrying out tests on your proposed system. 4 hours ago, bjp999 said: Parity check speeds are generally dictated by the size of parity. So a 10TB parity with 5 data disks would take about the same time as a 10TB parity with 10 dates disks. Assuming you are not constraining the bandwidth, and have sufficient horsepower, your parity check speeds with 28 data disks should be similar, but we don't really have good data to know for sure. Maybe I'm wrong, but I thought that, during the parity check, all the drives need to be read? I understand, how the size of the parity drive (and the largest data drive(s) on the pool, probably) would dictate how long parity check would take. But I'm thinking this should be true only if you have a small enough number of drives, so that there is no I/O bottleneck. Hence my question - with 30 total drives - there will definitely be an I/O bottleneck... Not sure how big, of course. but other-wise, isn't that the case that parity check depends on your I/O bottleneck and the pool size (both in drive count and drive size)? I assume, that is what bjp999 means by "not constraining the bandwidth", yes? 4 hours ago, bjp999 said: Parity drive is not the most important. In fact I'd argue the opposite. And reliability statistics generally don't bear out that more expensive disks are more reliable. Finding specific statistics can be difficult. But if you are going with REDS for your data disks, they are fine for parity as well. I prefer HGST, but many here like the REDS the best. The reason I say they are the least important is because it is only when a data drive fails that you are relying on parity for anything. And when a drive does fail, EVERY OTHER DISK IN THE ARRAY is equally important in the recovery. Parity is not special. I was basically thinking, that parity drives do the most of work - they get written to whenever there's anything getting written to any of the data drives... Hence, I assume they need to be more enduring and reliable. And, of course, fast - at least not slower than any data drives, right? Edited July 25, 20178 yr by shEiD
July 25, 20178 yr Community Expert 41 minutes ago, shEiD said: Basically - when unRAID adds a new drive to the pool - can that drive have data on it or not? I mean - if you add a drive to the unraid pool does unraid always clear and re-format the drive? It can be added with data if the array has no parity, if the array has parity disk has to be cleared (it's also possible to do a new config and re-sync parity).
July 25, 20178 yr Wow - kind of a monster to reply to ... I'll take this out of order. Rethinking the VM. I think you could install a Windows image, pass through the HBA(s), get it working, and reform the DrivePool within the VM. This might work great but my spidey sense is going off, probably because I don't know how sensitive the drivepool is. I have zero experience. But if a mistep is going to invalidate the pool and leave you with all your data gone, it is not worth messing with. If the drivepool is robust, and you'd be able to reclaim the data even if the drivepool config but messed up, maybe worth a try. You'd have to be running your unRAID server on that same physical machine to get the speed advantage. It might get tricky to move a few drives over at a time - as you'd have to free up an entire HBA to not pass it through - or you'd need ample drive slots on the unRAID HBAs to accept the drives until you're able to un-pass through an HBA to the VM. I think this sounds like a risky approach, but YMMV. I said before, when you establish a new array, none of the drives are zeroed. All of them are accepted as is. When you add a drive to an existing PARITY PROTECTED array, any disk you add will be zeroed (or confirmed to be precleared). To add a populate disk to a parity protected array, you'd have to do a new config, add the populated disk, and then recompute parity. If the array is not parity protected, the zeroing does not happen, and you can just slap a new populated disk into place. Bottom line - nothing to worry about here so long as you understand the rules. I understand your point of wanting to reorganize your data as you copy it off of DrivePool and onto unRAID. But you mentioned you'd be emptying the DrivePool disks and then loading them into unRAID. If the data is coming from hither and yon, how do you know you'll empty anything? If you copyied that data off in their jumbled mess, there is an unRAID tool called "unbalance" that might take a jumbled mess and do the reorganization in place for you, but would need some study to determine. But if there is a not a lot of slack space it might get bogged down. And no doubt it would take a very long time. Bottom line - copying over the files and getting them organized on the way is is a good approach so long as you are able to empty disks to incrementally reallocate to unRAID. Parity checks result in all drives being read in parallel, constrained by the slowest disk at that point in the parity check. So if you are parity checking with a 10TB parity, your system should be able to do the reads in parallel at full (or near full) speed, it would not matter how many data disks are present. The parity computation might take a very small extra amount of time, but would not elongate it much. It is true that parity disks get more write action. And if you are doing more than one write at a time (to multiple disks), a faster parity disk can speed things up marginally. (Rotational speed is particularly important). BTW, turbo writes (a.k.a.reconstructed writes) are a faster way to copy data to a protected array. But some benchmarks have shown better performance with a parity disk of the same model as the data disks. Bottom line - it is not going to make much of a difference.
July 25, 20178 yr Author 3 hours ago, johnnie.black said: It can be added with data if the array has no parity, if the array has parity disk has to be cleared (it's also possible to do a new config and re-sync parity). 3 hours ago, bjp999 said: I said before, when you establish a new array, none of the drives are zeroed. All of them are accepted as is. When you add a drive to an existing PARITY PROTECTED array, any disk you add will be zeroed (or confirmed to be precleared). To add a populate disk to a parity protected array, you'd have to do a new config, add the populated disk, and then recompute parity. If the array is not parity protected, the zeroing does not happen, and you can just slap a new populated disk into place. Bottom line - nothing to worry about here so long as you understand the rules. Phew... I was beginning to worry Thank You guys for making this clear. So, even if you already have parity - in an emergency, you can simply "disband" the array, take the drive(s) with data and re-create the array and parity from scratch... Am I understanding this correctly? 3 hours ago, bjp999 said: Rethinking the VM. I think you could install a Windows image, pass through the HBA(s), get it working, and reform the DrivePool within the VM. This might work great but my spidey sense is going off, probably because I don't know how sensitive the drivepool is. I have zero experience. But if a mistep is going to invalidate the pool and leave you with all your data gone, it is not worth messing with. If the drivepool is robust, and you'd be able to reclaim the data even if the drivepool config but messed up, maybe worth a try. DrivePool is nothing special. The drives are simple NTFS drives with a hidden folder in the root, where all the files and folders get placed. Basically, it is very similar to how user and disk shares operate in unraid. With DrivePool you can access your files pooled over the pool drive (which get's created and presented to the windows as a regular drive), same as user shares on unraid. Or you can access the drives directly via their mount points (eg: I have all the regular letter mount points removed from all my pooled drives, and instead I have all of them mounted to folders in D:\Drives\...), same as disk shares in unraid. Basically - I can remove DrivePool completely, and still easily access all my files on the drives without it. It's just that all of them will be in hidden folder in the root of each drive. Simply show hidden - move everything out to root - done - a regular NTFS drive. So, there shouldn't be any kind of problems from DrivePool. If I passthough the HBA - all the drives should show up in windows as regular drives and DrivePool simply picks them up - no problem. Actually, DrivePool would be quite good and adequate, if not for 2 stupid mistakes: The stupidest thing ever - DrivePool scatters the files all over the drives by default - with pretty much no real way to keep them together, as in unraid. And the dev is ignoring any pleas for implementing an option to keep the files in a folder together. The numerous explanations for this that I have read over the years are ridiculous. The second problem is that the only type of protection is duplication. Which is stupid for media. And because of the reason #1 - you effectively can't even use third party parity solutions, like eg: snapraid. Because you never control the file placement - hence you basically will be out of sync all the time. There's a shitload of other problems with DrivePool and pretty bad too, but that is irrelevant. 3 hours ago, bjp999 said: You'd have to be running your unRAID server on that same physical machine to get the speed advantage. Exactly the reason I want to run the windows in a VM on the unraid I'm pretty sure I could get 800MB/s per HBA or even more, maybe. 3 hours ago, bjp999 said: It might get tricky to move a few drives over at a time - as you'd have to free up an entire HBA to not pass it through - or you'd need ample drive slots on the unRAID HBAs to accept the drives until you're able to un-pass through an HBA to the VM. I think this sounds like a risky approach, but YMMV. Actually, with 4 expanders I can put 64 drives online just using them (16 drives per expander). I can connect all 4 expanders to a single HBA. Or, as I was gonna do - 2 HBAs with 2 expanders each. 32 drives per expander. As unraid only supports 30 drive pool, I'm covered. I even have 2 extra slots for cache drives later on. Theoretically (as I'm new to unraid, hence no practical experience) there should not be a problem... I clear the drives in windows, remove them from the DrivePool Shutdown the windows VM and power down the server (unraid) itself Remove the drives from expander connected to the HBA, that is passthrough to the windows VM Add those drives to the expander connected to the other HBA, which is used by unraid Power on server (unraid) The new drives are ready to be precleared Power on windows VM - those removed drives are not even missed... And continue moving files onto the newly precleared drives on unraid... Again - removing and adding drives in my case is simply sliding out the HDD trays from one expander and sliding them into another. No potential loose connection/cable problems, whatsoever. 3 hours ago, bjp999 said: I understand your point of wanting to reorganize your data as you copy it off of DrivePool and onto unRAID. But you mentioned you'd be emptying the DrivePool disks and then loading them into unRAID. If the data is coming from hither and yon, how do you know you'll empty anything? Bottom line - copying over the files and getting them organized on the way is is a good approach so long as you are able to empty disks to incrementally reallocate to unRAID. Yes, that is a pickle with DrivePool Anyway, DrivePool has a balancer (the whole DrivePool works on "balancers") which fills the drives one by one. That is one of the "answers" they give, when asked how you can keep files together. This balancer dumbly fills the drives one by one in a row, instead of scattering files. Which may keep the files together, while you initially filling the drives. But later on - during normal usage - deleting files, adding new ones - the files still dumbly go to whatever drive needs to be filled now. Hence - no real keeping files together, no matter how much they want people to think that it is. In my situation - I simply move the 3 drives worth of files from the DrivePool, and yes they come from all over the place - so no drives get nicely emptied. But then I use that balancer in the background to fill some drives back up, in the process almost clearing the others (the next batch to be removed). At the end, I just use another balancer to actually clear the drives completely. And thats the whole convoluted mess DrivePool has gotten me into Oh, and btw - that singular balancer which fills the drives one by one, by disabling the most important default behavior of DrivePool - the scattering of the files all over the place - it is not even available by default. You need to install it separately, and it is not recommended TBH, the whole DrivePool business would be funny, if it wasn't so sad... 3 hours ago, bjp999 said: you copyied that data off in their jumbled mess, there is an unRAID tool called "unbalance" that might take a jumbled mess and do the reorganization in place for you, but would need some study to determine. But if there is a not a lot of slack space it might get bogged down. And no doubt it would take a very long time. Wow! That sounds extremely interesting. is this what you're talking about: [Plug-In] unBALANCE? In the description it says: Gather (Coming Soon ... ) Consolidate data from a user share into a single disk So it has Scatter implemented, but Gather - no? That would be very ironic I hope not, gonna read though it after posting this... I was looking for something - anything to do this kind of thing. I even started writing my own program, but dropped it, because I figured it'll will not save me more time during migration, that I would waste writing it. 3 hours ago, bjp999 said: It is true that parity disks get more write action. And if you are doing more than one write at a time (to multiple disks), a faster parity disk can speed things up marginally. (Rotational speed is particularly important). BTW, turbo writes (a.k.a.reconstructed writes) are a faster way to copy data to a protected array. But some benchmarks have shown better performance with a parity disk of the same model as the data disks. Bottom line - it is not going to make much of a difference. Please, from your experience with unraid, what would you recommend? I am thinking of only 3 options: WD Red 10TB - 5400 rpm, hence running cooler. Exactly the same as the data drives, and probably my favorite option WD Red Pro 10TB - 7200 rpm, this would be the most expensive option, but probably the most reliable? WD Gold 10TB - 7200rpm, more reliable than regular Reds? Edited July 25, 20178 yr by shEiD
July 25, 20178 yr The 10 TB drives are at a big premium. The 10TB reds are $380. The 10TB golds are $450. The 10TB HGST (Helium) are $350 (at Server Part Deals). I think I'd have to go for the HGST. But those are $35/T. I just bought an 8TB REDS for under $20/T. 30T in 10s is 3 drives = $1050 - $1350 32T in 8s is 4 drives.= $640
July 26, 20178 yr Author 1 hour ago, bjp999 said: 32T in 8s is 4 drives.= $640 Where did you buy these 8TB Reds so cheap? I'll buy some on the spot, if they're still available at this price. 12 hours ago, bjp999 said: There is a relatively inexpensive 10Gb lan option you could explore. That might be the least invasive. ConnectX2 Do you mean these? MNPA19-XTR 10GB MELLANOX CONNECTX-2 PCIe X8 10Gbe SFP+ NETWORK CARD W/CABLE | eBay - these are suspiciously cheap And cable included... Could I just direct connect my 2 servers with 2 of these cards? I mean without a 10gig switch, because I don;t have one... Edited July 26, 20178 yr by shEiD
July 26, 20178 yr The 8T REDS are from best buy. You have to shuck from WD easystore drives that are on sale for $160. But the Seagate externals and sometimes these WD externals have been going on sale around $180 for a while now. But $160 is a new low. You'll have to ask @johnnie.black about there ConnectX2s. He has a pair and having good luck with them.
July 26, 20178 yr Community Expert 6 hours ago, shEiD said: Could I just direct connect my 2 servers with 2 of these cards? I mean without a 10gig switch, because I don;t have one... Yes, I used them in 4 of my servers, have 2 dual port NICs on my desktop.
July 26, 20178 yr Author 1 hour ago, johnnie.black said: Yes, I used them in 4 of my servers, have 2 dual port NICs on my desktop. Cool, ordered 3 (just in case 1 is DOA). @johnnie.black Will these be easy to setup and use on unRAID and windows? Any guides? I have no experience with cards like these... Edited July 26, 20178 yr by shEiD
July 26, 20178 yr Community Expert They are plug n play in both unRAID and Windows (I believe Window 64bits only), then you just assign static IPs to both, on a different range than your existing gigabit lan, and add the new 10GbE IP to the windows hosts file to make sure all traffic goes through that NIC.
Archived
This topic is now archived and is closed to further replies.