comet424 Posted August 10, 2018 Share Posted August 10, 2018 wish for manual or automatic pause of parity checking when other tasks needs the disk capacity. since copying 2TB to the unraid.. unraid at same time is doing a PArity check with 12TB parity drive.. and trying to access Plex server all at the same time and plex stops working it breaks down from this scenario Link to comment
-Daedalus Posted August 10, 2018 Share Posted August 10, 2018 Dell's PERCs have a user-defined 'rebuild rate' (default of 30%), which as far as I understands is a QoS system for the I/O to the disks. I've no clue if something like this is possible with unRAID, but it would be functionally pretty similar to what you're suggesting: If other stuff is happening, prioritise that, rather than the parity check. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 not sure what all that means what you said I just a simple person simple words lol.. just frustrating lol took 3 days 2 hours 45min to parity check 12TB drive of 24TB of data so had to wait Link to comment
JorgeB Posted August 11, 2018 Share Posted August 11, 2018 6 hours ago, comet424 said: took 3 days 2 hours 45min to parity check 12TB drive That's a long time for a 12TB parity, I have a server with 8TB drives and it takes 15 Hours. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 im guessing because i have 2 2TB drives 1 6TB 1 1.5TB and 1 12TB data drives and 1 12TB parity drive plus while it was doing parity i transfered 2.7TB of data to the array oh and it averaged parity speed was like 40mb/s it said Link to comment
pwm Posted August 11, 2018 Share Posted August 11, 2018 24 minutes ago, comet424 said: while it was doing parity i transfered 2.7TB of data to the array This is the main factor for the slowdown - during this time, the disks had to seek like crazy between the parity scan position and the file transfer position. And it was because of this that I suggested you should open a feature request thread - it would have taken way less time to pause the parity scan, perform the file copy, and then continue the parity scan. And with less disk wear. The file /proc/diskstats shows how much work different disks does. So it would be quite easy for unRAID to look at this file and notice that it's meaningful to step down on the parity scan work. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 well i kinda already did ask for a feature request. at the top i asked for a automatic or pause feature of the parity scan.. is not what you mentioned the same.. ya i dont really check the server i just copy to it and let it be so i noticed when plex want working parity was like 26 done and i was copying at that time 2.7tb of data to the system.. i still very new to unraid i used to windows Link to comment
pwm Posted August 11, 2018 Share Posted August 11, 2018 4 minutes ago, comet424 said: i still very new to unraid i used to windows Windows also suffers badly when doing multiple transfers using the same disk. Linux too. The OS doesn't have any optimization to prioritize one transfer before another. So it's mostly up to the end user. In enterprise installations, it's often handled by tiered storage using cache drives where some storage manager software spreads the load over multiple cache drives. And it's up to the system designer to decide max burst capacity that the caching system should be able to handle, and design enough overcapacity of the main storage to be able to synchronize between cache and main storage while maintaining the required real-time requirements. This is also why many larger storage solutions skips disk arrays and instead uses large pools of disks and stores multiple copies of the file data spread over "random" disks. Then a load balancer can spread read/write jobs over the disk pool. One note is that the native RAID support in Linux do support bandwidth-limitation of rebuilds. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 ok some of that confused the heck out of me lol i have a 250gb cache drive i added to the unraid i found if i copy the 2tbs of stuff it filles that cache but it doesnt transfer right away to say the 12tb drive and i had to manually transfer it.. so i guess doesnt work for big file copies like that i do ultimately want to have say on my unraid media server here if 3 or 4 people in the home accessing the server or watching the same show by odd chance i dont want no slow down.. do i need large pool and the file spread over random disks.. or would upgrading to like 10gb nic and 10gb switch fix that or do you need this link agression switch with multiple nics as id like only 1 copy of the file on the network but to speed up performance i plan on changing out like the 1.5tb drive with another 12TB drive i using WD Gold 12TB and would have tried these SASs but i cant get them from canada computers.. and not many comps support it anyways Link to comment
pwm Posted August 11, 2018 Share Posted August 11, 2018 16 minutes ago, comet424 said: i do ultimately want to have say on my unraid media server here if 3 or 4 people in the home accessing the server or watching the same show by odd chance i dont want no slow down.. The disks have more bandwidth than what single views needs. And use of read-ahead (in the viewer and by Linux itself) means each viewer will normally read in enough data that the drive has ample time to perform a seek and move to a different file to stream out to another viewer before later return back and fetch more data for the first stream. And if multiple people are viewing the same show, then unRAID will also make use of RAM to buffer data. When parity is checked, RAM buffers doesn't matter because unRAID wants to walk through all data on the disks once. So every new read is for new data from the disks. And same with your 2.7TB copy - it can't make use of any RAM buffers because there is constantly new data to write. So this special situation ends up with the parity disk and the data disk you copy to constantly multiplexing between the two tasks. With infinite seek speeds, you would end up with around half the speed - 50% of transfer capacity consumed for the copy and 50% for the parity check (actual percentage depending on disk raw capacity and used network interface speeds). But HDD disk seeks are very, very slow. So a majority of the time, the disks will not transfer any data because they are waiting for the disk heads to move to the new location. You get something called thrashing - you might end up with 20 MB/s total transfer rate and 80% of the time waiting for the disks to finish seek operations. That is why SSD manufacturers likes to present their 80000 OP/s figures that allows a SSD to access multiple files "concurrently" without the transfer rates free-falling. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 ok i understand a little but for my questions if i wanna be able to access same file and not have a slow down what do i need to do with un raid would upgrading the nics help i did notice if i paused my copying of the 2tb plex worked a bit least it wasnt choppy quality i guess i was maxing out the 1gbps nic but if its better for SSD can plex or unraid if a file is accssed plex copys to the ssd and then reads it that way as they dont make 12TB SSDs yet Link to comment
pwm Posted August 11, 2018 Share Posted August 11, 2018 Best is if you start your copying before going to bed (but preferably avoiding a parity-check night). And if you have a cache disk, and configure your shares to use the cache (option "Yes") then you can add reasonable amounts of new data to unRAID by writing to the cache drive, and have unRAID move them to the array next night. Note that your current issue was a special case. You had two file operations that each wanted 100% disk transfer capacity that did run concurrently with you trying to use Plex. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 ah ok... and what about accessing same file 4 times odds are it wont happen for plex cant the hard drives handle that can the nic handle it as i doubt it could handle and copy files over a single nic without dropping quality i did try the cache i new to SSDs i used to regular hard drives i bought my first SSD a month ago 250 WD blue 3D nano not sure what 3d nano means but has 5 yr warrenty i found when i link all my shares to the cache and was copying the 2 tb it filled that 250gb cache then unraid complained it didnt automaticlly move it right away had to manually move it.. i figured it automaticlly move when it gets close to full so there be no performance slow down etc Link to comment
Squid Posted August 11, 2018 Share Posted August 11, 2018 2 minutes ago, comet424 said: i figured it automaticlly move when it gets close to full It will move the files over on a schedule you set (Settings -> Scheduler). You can have unRaid move files over when the cache drive gets to a certain fill level by installing the Mover Tuning plugin. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 ok ill try that thanks and is there anything i can do to make it so if multiple people accessign the unraid to watch tv shows or movies or the same movie there is now slow down in quality i dont need it to stream online just on the network via xbox computer cell tv etc Link to comment
pwm Posted August 11, 2018 Share Posted August 11, 2018 51 minutes ago, comet424 said: as i doubt it could handle and copy files over a single nic without dropping quality Normal 1080p quality video can be streamed over a 100Mbit/s network. And "standard" networking nowadays is 1gbit/s. So the NIC shouldn't be the limiting factor. In most situations you should be fine to stream multiple playback streams at the same time, unless Plex needs to transcode the data to a resolution and encoding standard that the different clients support. If transcoding is needed, then the limiting factor would normally be the available computational power since it takes a huge amount of computations to decode and encode video data. In the end, I'd say just set cache option to "Yes" for your media shares and you are likely to be fine without need to do anything else. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 ok ill give it a try then Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 oh and under cache where do u set if it gets full it transfers there is a Critical disk utilization threshold (%): set to 90 is that what it is if so i guess wouldnt work for me since when i was copying it was 2TB and the cache was 250 so it be always full as its transfering right Link to comment
Squid Posted August 11, 2018 Share Posted August 11, 2018 8 minutes ago, comet424 said: oh and under cache where do u set if it gets full it transfers there is a Critical disk utilization threshold (%): set to 90 is that what it is if so i guess wouldnt work for me since when i was copying it was 2TB and the cache was 250 so it be always full as its transfering right Once the CacheFloor setting is reached, all transfers will go to the array. The file(s) on the cache drive won't get moved until the mover runs (Scheduler - Mover Settings). The plugin previously mentioned you can schedule it to run far more often, and begin moving files once a % threshhold is reached. Link to comment
comet424 Posted August 11, 2018 Author Share Posted August 11, 2018 ah yes i forgot i thought was part of the unraid program thanks its installing now Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.