August 29, 201213 yr Tom, I have a huge array with 1, 2 and 3 Tb drives. With parity checks/rebuilds I notice that the speed goes as follows: 0 Tb : 90Mb/s 0.99Tb: 50Mb/s 1.01Tb: 95Mb/s 1.99Tb: 55Mb/s 2.01Tb: 100Mb/s 3.00Tb: 75Mb/s This is because a drive is fast when it read/writes at the outer side of the platter, and is slowest when it near the spindle. The array is now configured as follows: 0.........1.........2.........3 TB parity drive 1Tb: 0.........1 1Tb drive 2Tb: 0.........1.........2 2Tb Drive 3Tb: 0.........1.........2.........3 3Tb drive Why not do the following: 0.........1.........2.........3 TB parity drive 1Tb: 2.........3 1Tb drive 2Tb: 1.........2.........3 2Tb Drive 3Tb: 0.........1.........2.........3 3Tb drive This way, all the 'slow' parts of the different drives come together, and since smaller drives usually means older (and thus slower) drives, I think you would gain quite some speed. The speed profile 'could' look like this: 0 Tb : 130Mb/s 0.99Tb: 110Mb/s 1) 1.01Tb: 105Mb/s 1.99Tb: 93Mb/s 2) 2.01Tb: 90Mb/s 3.00Tb: 50Mb/s 1) In this location you would need to spinup the 2Tb drive 2) In this location you would need to spinup the 1Tb drive Best leave the smaller drives idle as long as they are not needed (less heat/wear/...), easy solution would be to spin all drives up when parity check/rebuild begins and keep them running. Things that complicate this: - smaller drives should be running just before they are needed in the parity calculations, but it is preferred that they are idle when not needed - some extra calculations when a smaller drive is needed (array size - drive size = start where needed for parity calculations) - when array size changes (size parity drive), need to recalculate start - when you go from 5.x to 5.x+1 (version that includes this change), you will need to recalculate parity Since rebuild speed is faster, rebuild times would be shorter, so less time for something to go wrong (overheating, second drive goes bad, etc...). Usefull when you have a very mixed array, but this is already usefull with an array with 1 drive that has another size as the other drives. IMHO, not needed for 5.0, but maybe in 5.1? Comments please...
August 29, 201213 yr Your way means there is no way to resize the drive and it greatly complicates the code to do parity calcs and drive rebuilds onto a larger drive is incredibly difficult. You also forgot how to calculate the initial boot sectors that are currently not parity protected. Sent from my Nexus 7 using Tapatalk 2
August 29, 201213 yr Author Your way means there is no way to resize the drive and it greatly complicates the code to do parity calcs and drive rebuilds onto a larger drive is incredibly difficult. You also forgot how to calculate the initial boot sectors that are currently not parity protected. Resize a drive is only a little bit more difficult, just recalculate start of the new size as needed for parity calculations. Thats all. Why would that be impossible (am I missing something?)? It doesn't complicate parity calc's, calc's are done just the same way as now, you do need a list with starts of the drive (address), and update this list when you work on the array (change drive size, add/remove drive, etc...). I would hardly call that 'greatly complicates the code'?! At this moment, unraid also has to know when it doesn't have to use a drive in parity calculations (because the drive is smaller than the address where unraid is doing it's parity calculation at that moment). Instead of remembering the end address of a drive, it now has to remember its start address. Initial boot sectors: Then instead of array size - drive size = start where needed for parity calculations do array size - drive size + 63 (or 64) sectors = start where needed for parity calculations and let the parity protection start from there. I don't see what is so difficult about that? ('array size' is maybe a bad use of words, better would be 'parity drive size', I'm not a native english speaker...) Anyway, it's a feature request, I just listed some things that could complicate this (not a complete list, I know :-)), but it is by no means impossible and very hard to do (I just don't have the unraid source code handy to prove that ;-))... It will require some work, but that's for all the feature requests here, and I think that faster rebuilds/checks are a good thing.
August 30, 201213 yr The thing that immediately jumps out at me is enlarging a drive. With your diagram, if parity is 2 TB and data drive 1 is 2 TB, data drive 2 is 1 tb, drive 2's parity info would be held on the last half on the parity drive, correct? If this is the case, if you later decide to expand disk 2 by replacing it with a 2 TB drive, instead of just adding it to the array, letting data rebuild, and being done, parity would have to rebuild and then be recalculated as the parity information now has to be moved to the first half of the drive to allow for the larger data drive's parity info. This would increase wear & tear on the parity disk as well as increase data vulnerability due to the need to recalculate.
August 30, 201213 yr Author The thing that immediately jumps out at me is enlarging a drive. With your diagram, if parity is 2 TB and data drive 1 is 2 TB, data drive 2 is 1 tb, drive 2's parity info would be held on the last half on the parity drive, correct? If this is the case, if you later decide to expand disk 2 by replacing it with a 2 TB drive, instead of just adding it to the array, letting data rebuild, and being done, parity would have to rebuild and then be recalculated as the parity information now has to be moved to the first half of the drive to allow for the larger data drive's parity info. This would increase wear & tear on the parity disk as well as increase data vulnerability due to the need to recalculate. Again, I don't see a problem with this. Edit: OK, I now see the problem Does a drive care where data is located? I think not, just as long as the filesystem has a working index, it can retrieve files from anywhere on the drive. A 1 Tb drive is slowest at the spindle, just as a 2Tb drive. ********************************************* Now Old system Parity 0........1........2 1Tb 0........1 1 indicates spindle 2Tb 0........1........2 2 indicates spindel Expanded drive system Parity 0........1........2 2Tb 0........1........2 2 indicates spindle 2Tb 0........1........2 2 indicates spindel What happend when we upgraded the drive: - 0 till 1 tb, drive was rebuild from parity - 1 till 2 Tb, contains zero's from preclear/format, nothing has to be done ********************************************* My suggestion Old system Parity 0........1........2 1Tb 0........1 1 indicates spindle 2Tb 0........1........2 2 indicates spindel Expanded drive system Parity 0........1........2 2Tb 0........1........2 2 indicates spindle 2Tb 0........1........2 2 indicates spindel What happens when we upgraded the drive: - 0 till 1 tb, contains zero's from preclear/format, nothing has to be done - 1 till 2 Tb, drive was rebuild from parity - some extra calculations for the new startaddress for this drive, update of the list with startaddresses Edit: either - move all data OR - adjust the filesystem index so that it can find the files (add 1Tb with each startaddress of each file/directory). Don't know if this is easy, should cause no wear on the drive. Still, how often do you enlarge a drive, and how often do you do a parity check/rebuild? The spindle stays at the same address, but the data stands now on the second half of the 2Tb drive, compared with the current situation. I don't think that the drive would mind that we first fill up the second half, and then the first half.
August 30, 201213 yr Interesting idea! (in theory, but flawed in practice [more below]) Long ago, when there were only 10 computers running Unix (outside of Bell Labs, where it was invented), 1973-4, I did something rather similar to this but for a different reason. [head-localization, vs. transfer-rate-parallelization (??)] Notwithstanding the additional coding/maintenance needed to handle your idea (including the adaptation for re-sized "drive positions"), you will have lost drive portability! Any unRAID data drive can be taken to any other Linux machine; ie, for disaster recovery, or bulk transfers (SneakerNet). Yes, you can accomodate this by writing a drive-copier-with-geometry-unmangler, but that requires an added drive--or much more hairy "in-place" implementation, and much slower execution. Also, will you be able to squeeze your "geometry-translations" into the kernel, such that fsck does not go bonkers? Where does it end? (Cost/benefit analysis ...) Good thinking, though! [i don't use unRAID. It's an excellent concept, but the implementation does not fit [b]my[/b] style. Too many training-wheels, and not enough flexibility. This is not a criticism, per se, but a personal opinion/"observation".] Wimpie, if you want to re-direct your (unRAID) brainstorming in a slightly different direction, figure out what I meant in this post [link], which is related/follow-up to this post[link2]. --UhClem "Forward! ... into the past."
Archived
This topic is now archived and is closed to further replies.