bkastner

I do understand what you are saying, and believe it makes sense. It's just the odds that so few disks would be hammered that confuses me. Regardless... I will just let it go. I am almost done the refresh and will manually move files around to be more balanced and upate my split settings. On the split level question is it recommended to just split any directory, or select a specific different level?

That all makes sense, and yes... it's OCD behaviour around the TV shows. I know it really doesn't matter, and should likely get over it. Even with this explanation I don't understand why those 3 disks I mentioned are being targetted excessively... I understand and expect some data to be written there, but not like it has. Also, I have 500GB minimum free space required set on those disks... even if it's writing to movie folders there already I could see it using an extra 25-50GB or even 100GB, but disk 10 is down to 552MB of space and disk5 is 193GB of space... both well under that threshold. That still confuses me.

Here is the diagram as well as a screenshot from my Movies share. The TV share is set up identically.

So.. this server has been running for a decade with these exact same share settings, and there are Movies folders on each of the 12 data disks and this has never been an issue before. When manually copying to the Movies share the files can go to any disk. As a test I just disabled the cache on my movies share so it writes directly to the array. I picked a movie at random that was on disk8 and moved it to my local machine, and then moved it back... and it ended up on disk9 So, is it fair to say that share settings/split level are not the issue I am having right now?

While I do see a number of disks have been written to, it's interested that I've gained almost half a TB on Disk4... presumably because the old version of a movie was deleted and the new version went to another disk (Other than mover I don't think I've touched any files). Though I am guessing, I don't know why else space would have gone up and not down. I do look at the cache disk. I'm constantly monitoring it to see what's been completed, what's still showing incomplete, etc. So right now I have a 173 movies in the Movies folder on the cache drive. Because the file names are different I don't think it's overwritting. It would treat them as seperate files (I've had movie duplicates this way before by accident).

So, I haven't done the one off test yet, but here is the picture I posted yesterday of the drives, and then again from just now. Not absoutely everything uses those 3 disks, but it looks like the vast majority does. Yesterday: Today:

I still tend to leave it at split level 1 for tv as I'd prefer to have everything related to a show in a single folder on a disk, but there is no real reason I suppose. My brain just likes things ordered like that. Movies\Movie Name (year)\Movie Name is how everything is structured, and with metadata not downloading it's only a single file in a given movie folder, so there is really nothing to split beyond the first level.

This is only if the file name is exactly the same though, correct? As mentioned in my original post there is a zero % chance of this. Up until this refresh project I just used movie.mkv, but now use movie, quality, audio setup, imdb link, release group, etc. Before this project I didn't use Radarr... I manually cleaned up movies, fixed metadata with Media Companion and then moves to Unraid. I am now trying to automate this and narrow my focus on finding the best quality version that meets my criteria. So, the folder would pre-exist, but the movie file name doesn't as it seems to get deleted once the new file is in the movies folder on my cache drive. I would still sort of expect that if the folder exists on disk2 for example that the new movie would be put into the same folder.... but this doesn't look like is what's happening. As I think more about this I likely need to do a test with a specific movie and follow it through the process. Right now I've just been searching all movies in a given year, and letting the 300-400 movies update as a single batch.

I don't use a media share... I have root shares for Movies & TV and they span all the disks. This is where I was confused about it being restrictive. I am not sure what change I am supposed to make to help this in this scenario

What are the recommendations for split level? I have a root movies share and tv share (i.e. not under \media or anything). For movies, historically, there were always jpg files for the movie, actors, etc, so I had this set to ensure they all sat on a common disk. I could see where the restriction would cause issue with tv shows as each season will try to ensure it's on the same disk, which can (and has) caused space issues. However, with movies, it seems to just be the movie file nowadays, and Plex finds all the metadata, so if it's a single file per movie, how is this too restrictive? I'm not sure I understand (though maybe my understanding of the split level set is incorrect). As for files being replace in situ... that is what I'd expect, and some of my confusion. I will end up with 100-150 movies in the \movies cache folder that mover will need to move. As mentioned, it seems that the original file is deleted, and the only copy of the updated movie sits on cache.. however presumably the folder still exists on the array somewhere, and I would expect the new movie to be placed in the existing folder, which would distribute across the array (to some degree). The fact that it always pegs the same 3 disks and maxes them out rather than distributing across the array is not what I'd expect. There is no way those 150 movies all are from the same disk.. and even if they were by fluke, I've refreshed 4,500 movies at this point with the majority trying to write to one of three disks.

Thanks for the thoughts. I just received the replacement drive from WD today and am pre-clearing it as we speak. I also have 2 parity drives so trust I should be good even with another failure (though I would rather not confirm that theory). I guess an emulated disk is still a 'disk' but would think it would be a safety feature to not write to an emulated disk, in case, as you mention, a second drive dies and you don't have 2 parity drives. You are increasing the amount of data at risk. However, I would have thought excluding disk8 from the share (for exactly that reason) would prevent the disk from being written to, but it hasn't. I see files written today, yesterday and the day before... even though it's been over a week since I excluded it after the failure. Hopefully Jorge can provide some insights into why this is happening.

I am having issues with how Mover moves contents to the array, and I don't understand the behaviour. My server is on Unraid 6.11 rc4 for reference. As background, I have a really large movie library and over the last couple of weeks I've been refreshing it to ensure I have the best 1080p version available vs whatever I may have grabbed 5-10 years ago. I have a ssd cache that my nzbget downloads uses and I am also using that drive as cache for my movies share, so when radarr finishes processing the movie it unders up under \movies on the ssd. I also have hardlinks enabled (though I don't know if this is any part of the issue). I've started using trash guides to help focus on the versions I want, and have a much more detailed file naming convention than I did before. So, when I have a new version of a movie it doesn't overwrite the old file. It seems to delete the old file, and the new file is initially found in \movies on the cache drive as noted. This is all fine, however when mover kicks in to start copying the data to the array it behaves in a way I don't understand. Even though I have the share configured for most-free mover seems to target a specific disk and just keep filling it rather than distributing across the array. Even though I have 12 data disks, it only seems to target 3 of them... which are now the most full and theoretically should never be picked. One of the drives is actually failed, shows as failed in the gui and I have it as an exclusion in the movies share, but mover will still copy to it for some reason. I did post about this a few weeks back and was told if a movie exists on a disk already then the new copy will be moved to that same disk under the existing folder, which makes sense, but this is not what's happening. Mover seems to pick a single disk (one of the 3 noted) and push everything to that one disk. I had 3TB of data on my cache that I needed to move to the array today... Mover picked the most used disk and filled it to 100% capacity (even though I have 500GB free space as a minimum). Once it maxed that out it moved to one of the other 2 disks that get used. I dont' know if once Mover starts using a disk it will move everything to that disk (rather than distribute based on most-free), but I would have expected it to at least respect the minimum free space setting, as well as the disk exclusion (ideally I'd hope Unraid would not write to a failed disk at all, but this may be asking too much). I am attaching images of my movies share settings as well as the array (as well as diags). Mover is running right now, and it maxed out Disk10 (again, well beyond the minimum free disk space), and is now writing to disk5 (one of it's other favorites), even though there are other disks in the array with much more free space. I understand the theory that if a movie exists it will use the existing folder, however I am refreshing 200-300 movies at a time, and there is no way they all resided on a single disk. I'm really confused, and not sure how to fix. As an aside, I did post somewhat about this earlier this month, but the disk in question failed, and I let this linger, and now have more information, so thought I'd start a new thread. I marked the other as solved, but it's really not. Any insights would be appreciated. I usually think I understand how Unraid works, but then situations like this pop up that prove me wrong. cydstorage-diagnostics-20220919-1930.zip

That makes sense, but shouldn't be the issue (I don't think). I am in the process of refreshing my movie library to get everything to 1080p, etc. I've processed around 1,300 movies, and almost 900 of them are now on disk8. I have 12 disks, so the odds of that are incredibly small I would think.

I am running Unraid 6.11 rc4, though I think my issue is independant of that. I have 2 cache pools - one for appdata and the other for backups/downloads, and have the mappings in NZBGET, Radarr and Sonarr to point to /mnt/user for everything so I can take advantage of hard links, etc. When I download a movie it moves from the Downloads directory to the Movies cache folder on the same drive, however when mover starts it seems to want to move everything to a single disk (disk8) for some reason. My share is configured as Most-Free, and disk8 is actually the least free disk, and I don't understand why it keeps putting data on it. I've even set disk8 as an exclusion for the movies share, but it still keeps moving files to that disk. I've also just realized I have some funkiness with my shares (certain movies are showing up as seperate shares for some reason). I only noticed that now, but assume it's unrelated to the overall issue. I only mention as anyone reviewing my diagnostics may wonder what I am doing (I wonder myself sometimes). cydstorage-diagnostics-20220907-1320.zip

The current cache drive is btrfs, so I should be good. Thank you for confirming.

Hello I currently have a 1 TB nvme drive as my cache, however I've bought 2 new drives (also 1 TB) to replace as a cache pool in a raid 1 and then use the old drive just for caching share data / new downloads. My plex metadata folder is large with a ton of files, so I am trying to find an easy way to do this. I wanted to confirm if my strategy would work, and if there are any potential issues. 1) Add one of the new 1 TB drives to the existing cache pool (which I assume will start to mirror the existing one). 2) Once replicated, remove the old cache drive, and replace it with the 2nd new 1 TB drive and let the mirror rebuild 3) Re-add the old cache drive in a new pool for downloads So, a couple questions: 1) Am I correct in how the RAID 1 system will work? Do I need to do anything to make this happen? 2) Is there a way to know once the 2 drives are completely mirrored 3) Are there any issues with the above approach

Sorry... as you likely noted above, this is now sold.

I'd do $800 for everything, but that's about as low as I want to go

Norco 4224 Server Chassis Bundle - Includes everything you need for a storage server other than CPU, motherboard, RAM and storage. Includes HBA adapters, SFF cables, power supply (all already cabled), plus 2 additional backplanes in case of issue, as well as additional SFF cables. 1x NORCO 4U Rack Mount 24 x Hot-Swappable SATA/SAS 6G Drive Bays Server Rack Mount RPC-4224 1x Upgraded Fan Backplane for 3x 120mm fans instead of 4x80mm fans 3x 120mm fans 2x 80mm fans 2x Replacement Norco RPC-4224 SAS Backplane SFF-8087 3x Dell Perc H310 Adapter - Storage Controller (RAID) - 8 transmitter/channel - SATA 6Gb/s /SAS 6Gb/s Low Profile - 6Gb/s - Raid 0, 1 1x AX860 ATX Power Supply — 860 Watt 80 PLUS® PLATINUM Certified Fully-Modular PSU 6x 2FT Mini SAS (SFF-8087) to Mini SAS (SFF-8087) Data Cable Located outside of Toronto, Canada Asking $900 CAD / $700 USD + Shipping (if required) Willing to meet in person, or ship, though this will likely be expensive due to size and weight. Not interested in selling parts. Only the complete bundle.

I've had Unraid running for 10+ years and have just added or updated disks as needed. Currently, I have a mix of WD Reds (12TB, 6TB, 4TB). All drives are xfs and sharing is configured to split top level only so everything associated with a show or movie is on a single disk. However, In this scenario I do have disks that get filled and I need to shuffle stuff around which can get annoying (though I know I can set a minimum free space). I was looking at adding a couple WD Gold 20TB, and with drives this large I have an opportunity to rebuild a pool from scratch. I was thinking of buying 5 of the 20TB drives which would give me 60TB usable space. I'd then move my movies over to it, and once the 12TB Reds are empty I'd add them to the new pool. The thought is I can switch to btrfs and can (potentially) switch to split at any level which I assume will work to fill my drives fairly equally. However, I have some questions: 1) Is this even worth doing? It seems like it may be a good idea, and it is a rare chance to basically start over as I have around 100TB of data, so it's not easily to redo. I was only going to buy 3 of the Gold 20TB, but figured by adding more I make this an option, but want to get thoughts on whether it's worth the effort 2) Am I better moving from xfs to btrfs? I've seen that both offer benefits, but btrfs seems to be where things are going so again.. easier to do this way. Moving from reiserfs to xfs was a pain in the butt, so I don't really want to do this drive by drive 3) Are there reasons I wouldn't want to use 'automatically split any directory' (I think it used to be called split level 0)? I do like the organization of having everything on a specific drive, but get tired of shuffling data around at times. I've never explored this split level before and am curious if there are any thoughts / recommendations 4) If I do split any directory am I correct that with the initial 3 drives it will just balance the data, but once I add one of the 12TB drives it will fill that to the same level as the original 20TB drives before equally distributing again? (i.e. if I have 5TB used on each of the 20TB, when I add a 12TB will it fill it to 5TB as well before equally distributing, or does it use % of disk capacity (so 3TB on the 12TB vs 5TB on the 20TB drives) 5) Are there thoughts on how best to manage this? 99% of the data is tied to a Plex server, which is going to add a challenge. I am thinking of starting with just moving all my movies over as the content is (fairly) static, and then move historical shows that won't update, and finally everything else. This will presumably mean Plex can't see some of the tv show data, but I am hoping it will minimize issues with data access. 6) Does anyone know how plex will manage this? I have a ton of movies and tv shows and don't want to completely screw up Plex or have it thinking that old data is being deleted and then re-added back in again on the new location (basically want to avoid churn). But I don't know the best way to manage this process (again... if it's even worthwhile). I'd be interested in any thoughts anyone has on the above. My Unraid server has been pretty static and stable for a long time and has run great. I like the idea of setting myself up better for the future, but don't want to screw things up (especially if it's not even worth doing at all). Thanks.

Great to see this function back again. I agree, it makes reviewing your log file much easier. Would it be easy to select/deselect the colors like we could in UnRAID? It would be great to just filter on warnings/errors for instance.

Current progress: Total size: 4 TB Elapsed time: 3 hours, 19 minutes Current position: 1,84 TB (46,1 %) Estimated speed: 145,1 MB/sec Estimated finish: 4 hours, 7 minutes Sync errors detected: 0 i'm on FW version 18 - flashed it 2.5 years ago and it's working so no need for change i think.. Looking at your sig, I see you have a RAID 0 array as your parity. I am wondering if that is giving you the better than expected results. I am sure it would have a positive impact, I just don't know how much.

I have M1015 cards in my setup too, just started a parity check, after 9 minutes the results are: Total size: 4 TB Elapsed time: 9 minutes Current position: 83,6 GB (2,1 %) Estimated speed: 162,3 MB/sec Estimated finish: 6 hours, 42 minutes Sync errors detected: 0 see all hardware details in my sig below. i'm on unRAID v6B15 - did not upgrade further cos i'm on esxi with USB drops issue with V6 final.. Wow... I would be interested in seeing the final results once the parity check is complete. That is definitely way higher than I start, though I usually start around 120MB/sec, but end up at 82MB/sec. I do see I have 2 drives with the 750GB platters, which I know are not ideal and I will try and replace down the road, but it would be great to get somewhere near where you are. For you and dikkiedirk, what firmware are you on? I am on P20 as it looked stable after the fix, but I am curious if this makes a difference at all (as well as the UnRAID version). It may be worth trying to go back to a beta build and see what happens. Can anyone confirm, if I am on 6.1.2 is there any potential issue with me moving back to a beta build for testing? I only have my one production box and don't want to risk my data at all.

The same tunables results for me happen when there is a bus bottleneck or it’s hitting the maximum speed of the slowest disk in all settings. Are all your WD30EZRX disks 1TB/platter your do you have some 750GB/platter? If all are 1TB/platter parity check speed should start at 145/150MB/s, if you have at least one 750GB/platter parity should start at about 125/130 Mb/s. But this can’t explain that according to what you posted in the SAS2LP thread, your starting speed is higher than the SAS2LP and in the end the average speed is lower. The parity check peaks around 125MB/sec, so it must include the 750GB/platters. Of course, I wasn't smart enough to keep the tunables report from the SAS2LP cards the last time I ran them, but there were much larger variations with those cards, and it had me applying much higher values for the various attributes. The tunables report I ran for the M1015 cards says the following: These settings will consume 136MB of RAM on your hardware. This is -422MB less than your current utilization of 558MB. So, you can imagine how much higher the settings were with my last config. It's definitely bizarre, and somewhat frustrating and I am not sure what to try next.

I recently swapped out my SAS2LP cards for several M1015 cards, which I flashed to firmware 20. However the parity check performance sucks. I was getting 95MB/sec with the SAS2LP cards, but I am not getting 82MB/sec with the M1015 cards. I re-ran tunables which had every setting identical (118.2 MB/sec). I am somewhat surprised that I've lost performance with these cards. I expected them to be at least as fast as my SAS2LP cards if not faster. Does anyone have any suggestions? It's by no means the end of the world, but a bit annoying to say the least.