TheJJJ42 Posted May 30, 2021 Share Posted May 30, 2021 Hi. My server uses several drives with different sizes and I think, parity and parity checks can be done in segments, if the drives are structured differently in the array. Currently, parity checks take two days. I think, this can be better: Example: |-------------------------------------------------------------Parity drive 14tb------------------------------------------------| |-----------------Drive1 5tb --------------||---------------Drive1 5tb ---------------||-------Drive1 3tb ----------| If drives are arranged this way, parity checks can be done drive by drive. - Check one drive per night. - Check just the drives with a specific amount of changes. (every 100gb change) (- Just do parity in sectors that contain data. - may not be possible) Is this a plausible idea? Quote Link to comment
itimpi Posted May 30, 2021 Share Posted May 30, 2021 I think you mid-understand how parity checks works? When writing new data parity is calculated in real-time and written immediately to the parity drive. What a parity check involves is reading a given sector off all array drives, calculating what the parity sector should contain and then comparing that to the actual sector on the parity drive. As such all array drives are always involved in a parity check. You might be interested in the Parity Check Tuning plugin which allows a parity check to be run in increments spread over several days. This is typically used to run a scheduled check only outside prime time at the cost of it taking longer to complete. Quote Link to comment
TheJJJ42 Posted May 31, 2021 Author Share Posted May 31, 2021 (edited) Hi. Thank you for your answer. I think, your explanation is how I understand parity (checks). So, when lots of drives are structured in parallel, all the drives have to be read to calculate parity - for example: for their first sectors. |--------------------------------14tb drive------------------------------| |-----------5tb drive-------| |-----------5tb drive-------| |----3tb drive---| Now all drives need to spin to check/validate the first 3tb of parity. To check between tb3 and tb5, the 3 larger drives have work to do - the small one can call it a day. (I hope, this is correct?) When only the 3tb drive had been used for the last two weeks (since the last parity check), there is no need to validate the parity for the 5tb drives. So, in my usecase, I would prefer the drive structure of my original post. Most often just the appdata and domains shares show alerts, that the data is unprotected / need a parity validation. Is there a way to validate just this data? I think it would be possible, if this data is not allocated at the beginning of the drives - because this includes all drives to validate the parity. In the 'serial' allocation of the drives (first post), just the drive containing these shares and the parity disk got work to do. It adds another perspective, if we take into account the way, how shares are handled: In my example, a share, that spans across the two 5tb drives and uses the fill-up allocation method, these drives factually are structured 'behind' each other (like jbod), like in my first post. If the first isn't full, the second should not contain any data. Why include it in a parity check at all? Do you get my thoughts? Are these assumptions how parity works, even correct? Edited May 31, 2021 by TheJJJ42 clearer wording Quote Link to comment
itimpi Posted May 31, 2021 Share Posted May 31, 2021 33 minutes ago, TheJJJ42 said: Most often just the appdata and domains shares show alerts, that the data is unprotected / need a parity validation If there is an indication that files are unprotected (and you have a parity protected array) then this means that files are on a cache that is is a single drive which is not redundant and this disappears when the files are later moved to the protected array. If the cache pool has multiple drives configured to give redundancy then you should never see such an indication. Not sure what you mean by ‘need a parity validation’? 36 minutes ago, TheJJJ42 said: t adds another perspective, if we take into account the way, how shares are handled: In my example, a share, that spans across the two 5tb drives and uses the fill-up allocation method, these drives factually are structured 'behind' each other (like jbod), like in my first post. If the first isn't full, the second should not contain any data. Why include it in a parity check at all? Even a drive that is ‘empty’ from a file perspective still has non-zero sectors that are used to contain the files system structure information and thus cannot be ignored for parity purposes. It is only sectors that contain purely zero bytes do not affect parity. Also any sectors that belong to files that are now deleted will rarely be zero filled. There is also an assumption in your statements that the drive sectors are used serially starting from the beginning - this is typically not the cases as the file system selects sectors in a more random manner than that. Unraid can avoid accessing all drives to update parity when writing new data but this imposes a significant performance penalty. You might find this section of the online documentation useful in describing the two different approaches Unraid has to updating the parity drive. Quote Link to comment
TheJJJ42 Posted May 31, 2021 Author Share Posted May 31, 2021 Thank you, very much! Quote Link to comment
TheJJJ42 Posted June 6, 2021 Author Share Posted June 6, 2021 I can´t find any information, if it is ok to keep using unraid when a parity check is running. Or what use cases do or dont affect a running parity check. (this information could be provided in the unraid gui, when a parity check is running.) Until now, I do not - which means more than 2 days 'downtime'. Sadly, I experience the issue, that shares cant be unmounted, when shutting down. (I read the threads to manually shut down docker and apps, what does not help.) So every shutdown/reboot results in 2+ days parity check downtime. Or, does it? Quote Link to comment
Squid Posted June 6, 2021 Share Posted June 6, 2021 You can keep using the system as-is. Performance will suffer though. Many people use the parity tuning plugin to run the checks in batches, only during off-hours If a share can't be unmounted, then something is stuck in a state where it won't shut down. Could be a crashed container, or something as simple as a command prompt with the current working directory some where within /mnt/user (or a symlink to there) Quote Link to comment
itimpi Posted June 6, 2021 Share Posted June 6, 2021 40 minutes ago, TheJJJ42 said: I can´t find any information, if it is ok to keep using unraid when a parity check is running. It IS documented in the online documentation that can be accessed via the 'Manual' link at the bottom of the Unraid GUI. Quote Link to comment
Necrotic Posted June 6, 2021 Share Posted June 6, 2021 I guess as an extension of this, my parity drive is bigger (12TB) than the other drives (<10TB). When doing a parity, once it gets past the max size of the other disks, they end up shutting down but I think I saw it still continue the parity with just the parity drive for those last 2TB. I don't usually pay attention to parity so I haven't confirmed that is the absolute case but it seemed odd that it would waste the time rechecking the parity area that isn't protecting a data disk. Quote Link to comment
trurl Posted June 6, 2021 Share Posted June 6, 2021 It makes no assumptions about the remaining parts of the parity disk, so it checks them. They would be important if you ever replaced a data disk with a larger data disk. Quote Link to comment
Squid Posted June 6, 2021 Share Posted June 6, 2021 Or if you added a pre-cleared 12TB to the array. If the system didn't check the remainder of the parity, then the parity may or may not be in sync with the precleared new large data drive. 1 Quote Link to comment
TheJJJ42 Posted June 6, 2021 Author Share Posted June 6, 2021 (edited) 20 hours ago, itimpi said: It IS documented in the online documentation that can be accessed via the 'Manual' link at the bottom of the Unraid GUI. Thank you for that hint. btw: I did not claim, that is is not documented, just that I could not find it, where I looked for this information. For everybody else, looking for it: >You can continue to use the array while a parity check is running but the performance of any file operations will be degraded due to drive contention between between the check and the file operation. The parity check will also be slowed while any such file operations are active. Edited June 7, 2021 by TheJJJ42 removed a partly false statement. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.