Karools

It's been ticking along nicely. Based on the time it's taken and the amount of data it's done so far, this can easily take another 4 to 5 weeks to complete. But yes, I was thinking, do you have any plans on implementing some cleanup or 'refresh' feature. Something that can check for orphaned or outdated par2 files? If you were to check and compare the files on the FS with what's in the json. In the event of new files, or where enough of the protected files are missing to invalidate the par2 set, it would recreate it. If all of the files are missing, the par2 files can simply be deleted. Just a thought..

It's working very nicely so far, thank you @Rysz . Great work, really, this fills a very important gap in UnRAID. This poor machine of mine will be busy generating sets for the next month I imagine. I was waiting on the nested option. It makes the setup so much easier. A single file in the root of your Movies folder and away it goes. _par2cron: glob: "**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa}" mode: "nested" The only thing I haven't tested yet is what it does when it fails during verification. It looks like it repairs and then deletes the corrupt file(s), but there's no kind of notification or persistent log for these encounters is there?

Oh no, not at all. This is a critical part of UnRAID for me now, optimisation is important ;D It's interesting, doing the media share by iteself now the time has doubled, but doing the whole array, it's been cut in half (more or less): ./par2cron -v par2cron version 0.2.2-SNAPSHOT-d7812bf ./par2cron create -c /boot/config/plugins/dwpar2cron/config.yaml /mnt/user/media 13:21:18 INF Scanning filesystem for jobs... op=create path=/mnt/user/media walker=os 13:27:23 INF Starting to process 28 jobs... op=create path=/mnt/user/media maxDuration=0s ./par2cron create -c /boot/config/plugins/dwpar2cron/config.yaml /mnt/user 13:28:11 INF Scanning filesystem for jobs... op=create path=/mnt/user walker=os 14:03:25 INF Starting to process 28 jobs... op=create path=/mnt/user maxDuration=0s ./par2cron create -c /boot/config/plugins/dwpar2cron/config.yaml /mnt/user --pprof perf.pb.gz 14:31:58 INF Scanning filesystem for jobs... op=create path=/mnt/user walker=os 15:07:53 INF Starting to process 28 jobs... op=create path=/mnt/user maxDuration=0s It's certainly more manageable; and I guess under normal circumstances it won't matter if it takes 30 min longer to iterate through the entire array. But still, is it really necessary to do so. perf.pb.gz

Nope, no parity check or anything too IO intensive. I had to replace a drive over the weekend, which is actually what prompted me to go looking into the integrity plugin - and ultimately lead me to your fine par2 implementation. I should have tried this earlier.. Running par2cron against the media share, it completes the scan in 2m24s: par2cron create -c /boot/config/plugins/dwpar2cron/config.yaml /mnt/user/media 19:04:42 INF Scanning filesystem for jobs... op=create path=/mnt/user/media 19:07:06 INF Starting to process 28 jobs... op=create path=/mnt/user/media maxDuration=0s Against the whole array, we jump up to 1h7m51s (very similar to yesterday in fact): par2cron create -c /boot/config/plugins/dwpar2cron/config.yaml /mnt/user 19:07:36 INF Scanning filesystem for jobs... op=create path=/mnt/user 20:15:27 INF Starting to process 28 jobs... op=create path=/mnt/user maxDuration=0s I don't know, but there's definitely something about the other shares that throws the poor thing off. Maybe I'll run par2cron against each of them and try to narrow it down. If it's not the amount of files, could it be the amount of folders? During this scan the drives were practically idle btw. There's one guy streaming something from Plex (another machine) via NFS. -- Update -- Though not entirely to blame, it looks like the system share is the main culprit. This is the one used for container images and things. This folder exists only on SSD for me, it's 160GB in total, consisting of 3216305 files and 506124 folders. And it takes par2cron 34m39s to process: par2cron create -c /boot/config/plugins/dwpar2cron/config.yaml /mnt/user/system 20:57:56 INF Scanning filesystem for jobs... op=create path=/mnt/user/system 21:32:35 INF Nothing to do (will check again next run) op=create path=/mnt/user/system

I suspect it's the sheer number of small files and folders, but they all sit outside of the of folder I'm actually interested in running this on. If there is a way of limiting the plugin to specific paths, that would be great. This setup consists of 24 x 12TB SAS drives + 6 SSDs for cache. Overall speed is quite good. /mnt/user: time find /mnt/user -type f | wc -l 6570518 real 6m50.679s user 0m13.044s sys 0m36.744s time du -hs /mnt/user 142T /mnt/user real 17m1.591s user 0m17.075s sys 1m53.070s /mnt/user/media: time find /mnt/user/media -type f | wc -l 725466 real 0m22.400s user 0m0.862s sys 0m1.901s time du -hs /mnt/user/media 132T /mnt/user/media real 1m42.175s user 0m1.278s sys 0m9.025s As you can see the difference in duration there is rather significant. I honestly don't know how slow it was prior to the glob updates. I don't recall it being particularly slow or snappy. Most of my tests were done on a different machine.

Firstly, the curly bracket issue is indeed fixed now with v0.2.2 - thank you very much. It's working perfectly. And the nested idea sounds great, it would certainly be of use to people who use *arrs for automation, at least for Movies. For current/ongoing Series not as much, for obvious reasons.. but there are ways to work around that 🤔 By the way, what @paulio23 mentioned is a bit of a problem for me too. I added markers to 4 movies for testing. After manually kicking off the Create task, it took over an hour for it to get going.

Oh my goodness you're quick 😶 The recursion thing makes sense, recursion is basically assumed based on the glob. Right, bear in mind it's late evening and I've already got a few glasses of wine in me, but I'm fairly certain I haven't made any mistakes. I have one new issue now that we didn't have before. With Plex you can give media an anchor or a tag to help prevent things from being mismatched; or a tag to distinguish between editions, like Director's Cut, Extended etc., in the form of curly brackets - https://support.plex.tv/articles/naming-and-organizing-your-movie-media-files/ It seems happy to process a filename with curly braces, even subfolders, but not when it's the folder containing the _par2cron marker. └── Movies └── Big Buck Bunny (2008) {imdb-tt1254207} ├── Big Buck Bunny (2008) {imdb-tt1254207} - 1080p 30fps normal.mp4 ├── Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.idx ├── Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.mp4 ├── Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.srt ├── Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.sub ├── Featurettes │ ├── big_buck_bunny_720p_surround.avi │ ├── sub_folder │ │ └── BigBuckBunny_320x180.mp4 │ └── sub_folder with braces {hello} │ └── big_buck_bunny_720p_surround.avi ├── landscape.jpg ├── logo.png ├── _par2cron └── poster.jpg That results in an error where it claims it can't find anything: par2cron create -c ~/par2cron.yaml . 22:24:03 INF Scanning filesystem for jobs... op=create path=/home/acme/test/media 22:24:03 INF Starting to process 1 jobs... op=create path=/home/acme/test/media maxDuration=0s 22:24:03 INF Job started op=create job="/home/acme/test/media/Movies/Big Buck Bunny (2008) {imdb-tt1254207}/_par2cron" job_position=1/1 args=[] glob=**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa} mode=folder hidden=false verify=false 22:24:03 ERR No files to protect in folder (will check again next run) op=create path="/home/acme/test/media/Movies/Big Buck Bunny (2008) {imdb-tt1254207}" job="/home/acme/test/media/Movies/Big Buck Bunny (2008) {imdb-tt1254207}/_par2cron" job_position=1/1 args=[] glob=**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa} mode=folder hidden=false verify=false 22:24:03 ERR Job failure (will retry next run) op=create job="/home/acme/test/media/Movies/Big Buck Bunny (2008) {imdb-tt1254207}/_par2cron" job_position=1/1 args=[] glob=**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa} mode=folder hidden=false verify=false error="failed to find to-protect elements: no files to protect" 22:24:03 ERR Operation completed with errors (1/1 jobs processed) op=create error="partial failure: failed to find to-protect elements: no files to protect" successCount=0 skipCount=0 errorCount=1 processedCount=1 selectedCount=1 Error: create: partial failure: failed to find to-protect elements: no files to protect But if you remove the braces from the main folder, it happy does its thing, including subfolders with curly braces: par2cron create -c ~/par2cron.yaml . 22:24:21 INF Scanning filesystem for jobs... op=create path=/home/acme/test/media 22:24:21 INF Starting to process 1 jobs... op=create path=/home/acme/test/media maxDuration=0s 22:24:21 INF Job started op=create job="/home/acme/test/media/Movies/Big Buck Bunny (2008)/_par2cron" job_position=1/1 args=[] glob=**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa} mode=folder hidden=false verify=false Block size: 821036 Source file count: 8 Source block count: 2000 Recovery block count: 100 Recovery file count: 7 Opening: Big Buck Bunny (2008) {imdb-tt1254207} - 1080p 30fps normal.mp4 Opening: Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.idx Opening: Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.mp4 Opening: Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.srt Opening: Big Buck Bunny (2008) {imdb-tt1254207} - 2160p 30fps normal.sub Opening: Featurettes/big_buck_bunny_720p_surround.avi Opening: Featurettes/sub_folder with braces {hello}/big_buck_bunny_720p_surround.avi Opening: Featurettes/sub_folder/BigBuckBunny_320x180.mp4 Computing Reed Solomon matrix. Constructing: done. Wrote 82103600 bytes to disk Writing recovery packets Writing verification packets Done 22:25:11 INF Succeeded to create PAR2 op=create path="/home/acme/test/media/Movies/Big Buck Bunny (2008)/Big Buck Bunny (2008).par2" job="/home/acme/test/media/Movies/Big Buck Bunny (2008)/_par2cron" job_position=1/1 args=[] glob=**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa} mode=folder hidden=false verify=false 22:25:11 INF Job completed with success op=create job="/home/acme/test/media/Movies/Big Buck Bunny (2008)/_par2cron" job_position=1/1 args=[] glob=**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa} mode=folder hidden=false verify=false 22:25:11 INF Operation completed (1/1 jobs processed) op=create successCount=1 skipCount=0 errorCount=0 processedCount=1 selectedCount=1 Apart from that, it's nearly perfect 🙃

That should work a treat yes, it looks exactly like what we're after. I'd very much appreciate if you could look into this when you have a moment, I'm fairly certain other users would also find it very useful.

The only issue I have so far is that the glob seems rather limited. It only allows for a single extension. There's no brace expansion support, unless I'm doing it wrong. I'm using this in a media library setting, I'm using a recursive marker per movie/season - that's the idea for now anyway. The problem is that these folders tend to contain lots of images and nfo files that can change quite frequently. Ideally I would like to add a recursive marker with something like the following to the root of each movie - glob: "**/*.{mkv,avi,mp4,idx,sub,srt,ass,ssa}

This is bloody brilliant! I was considering switching to ZFS, but with this there's really no need. I'm still testing, but it looks very promising. Could arguably be one of the best plugins for UnRAID ever in my mind ;) Just out of interest though, why is the default redundancy set to 15%? If this is primarily to address bitrot, the par2 default of 5% is more than adequate, isn't it?