foo_fighter

I have about 100k photos/videos. I haven't figured out how to get the iGPU/quicksync to work yet so it's using software transcoding. ML is turned on for facial recognition but that just uses CPU I believe (I don't have any HW accelerators). Have you checked your paths? All generated data should be outside of the docker image right?

2.2Gb for me.

I see this endless cycle in syslog. But you're saying that the read SMART is caused by the spin up, not the other way around?: Feb 18 06:23:32 Tower s3_sleep: All monitored HDDs are spun down Feb 18 06:23:32 Tower s3_sleep: Extra delay period running: 18 minute(s) Feb 18 06:24:32 Tower s3_sleep: All monitored HDDs are spun down Feb 18 06:24:32 Tower s3_sleep: Extra delay period running: 17 minute(s) Feb 18 06:25:32 Tower s3_sleep: All monitored HDDs are spun down Feb 18 06:25:32 Tower s3_sleep: Extra delay period running: 16 minute(s) Feb 18 06:25:34 Tower emhttpd: read SMART /dev/sdf Feb 18 06:26:32 Tower s3_sleep: Disk activity on going: sdf Feb 18 06:26:32 Tower s3_sleep: Disk activity detected. Reset timers. Feb 18 06:27:33 Tower s3_sleep: Disk activity on going: sdf Feb 18 06:27:33 Tower s3_sleep: Disk activity detected. Reset timers. Feb 18 06:28:33 Tower s3_sleep: Disk activity on going: sdf Feb 18 06:28:33 Tower s3_sleep: Disk activity detected. Reset timers. Feb 18 06:29:33 Tower s3_sleep: All monitored HDDs are spun down Feb 18 06:29:33 Tower s3_sleep: Extra delay period running: 25 minute(s) Feb 18 06:30:14 Tower emhttpd: spinning down /dev/sde Feb 18 06:30:33 Tower s3_sleep: All monitored HDDs are spun down Feb 18 06:30:33 Tower s3_sleep: Extra delay period running: 24 minute(s) Feb 18 06:31:33 Tower s3_sleep: All monitored HDDs are spun down Feb 18 06:31:33 Tower s3_sleep: Extra delay period running: 23 minute(s) Feb 18 06:31:41 Tower emhttpd: read SMART /dev/sde Feb 18 06:32:34 Tower s3_sleep: Disk activity on going: sde Feb 18 06:32:34 Tower s3_sleep: Disk activity detected. Reset timers. Feb 18 06:32:34 Tower emhttpd: read SMART /dev/sdh Feb 18 06:33:34 Tower s3_sleep: Disk activity on going: sdh

I'm seeing the same thing in 6.12.8(also upgraded from 6.12.6). SMART reads on one ZFS drive every 20 or so minutes which prevents s3 sleep.

Is rsync rsync a typo? Would you consider adding -X to preserve extended attributes for things like the Dynamic File Integrity plugin?

the only way to do this with a live file system is with replication like on ZFS(or BTRFS). You can use sanoid/syncoid to setup replication of ZFS snapshots.

I have 6 MB Sata ports on an old MB with no M.2 slots and a LSI HBA, but I'm thinking of consolidating to only MB ports to save power. The HBA prevents lower C-states. I only have 4TB-8TB drives with 1 14TB Parity so I could easily consolidate to 2 or 3 18TB drives. 18TB drives were $200 during the BF sales and ~$150 for pre-owned server pulls. They were the sweet spot at the end of last year. Some M.2 slots are only PCIe 3x1....so 6 Sata drives would saturate the BW. I've also been looking at some pre-built systems with 4-6 HDDs and 2 M.2s. There I'd really need the high capacity HDDs.

Why would you need one, do you need more than 6 ports? There are 6 port M.2 to Sata adapters based on the ASM1166 chipset.

If 1 external drive can hold all of the data, that would be a convenient way to go. Rsync should work fine. You'd want to make sure /mnt/disk*/* is sync'd over to the backup drive. Another way would be to use Unbalace(d) to zero out and convert one drive at a time by moving one disk's contents to the other disks. That's assuming you have the spare capacity. Curious why you want convert the entire Array over. You'll get some benefits but not all the benefits of ZFS(you can check for corrupted files but not repair them for example)

You can find the PAR2 util in nerd-pack/tools.

Both NTFS and exFAT can store extended attributes so yes, DFI(or bunker from the command line) can run on those drives and files.

With Mover, they are treated as normal files, only the primary, most recent non-snapshotted files are moved over into the array. With Syncoid/Sanoid you can choose individual datasets and sub datasets. It can be used in the array. For example, I have my cache drive as ZFS and converted 1 array drive to ZFS so I could use it as a replication target.(ZFS->ZFS) The other drives in my array are XFS. I have my app data(dockers and docker data living on cache) replicated over to the ZFS drive for backups. It can be. RAID is not backup, so for catastrophic cases(any filesystem) that would be the recovery mechanism. I was only referring to the ZFS_Master plugin spinning up ZFS drives. It didn't touch any of my XFS drives. I actually set the plugin to manual refresh and it seems to have stopped the spinning up and writing.

ZFS snapshots take less space because they only store the deltas ZFS replication will be faster since it only sends the deltas The corollary is that deleting file won't save any space until all snapshots are also deleted ZFS replication of app data/cache drive doesn't require shutting down and restarting dockers. ZFS doesn't have a built in file repair system ZFS can detect bit-rot but can only repair bit-rot in a pool with redundancy(mirror/raidz, etc) ZFS Master plugin seems to spin up and write to drives often.

If you do an all ZFS pool, you would need to add 1 drive to the array. Could be a USB drive. One other option is a 2 parity but all ZFS array(all individual disks). That would give you the compression, not have to spin up all the disks, but you would lose the self-repair option of a ZFS Pool.

In my opinion, the best way of doing this is to use ZFS replication from a cache pool to another ZFS disk in the array or a ZFS pool. There is a sanoid/syncoid video for this:

I'm also getting these "errors" and the last 2 youtube videos I watched on unraid also showed these error scrolling by. I'm not experiencing any issues, but it seems this particular type of warning is pretty prevalent. Both drives and ports are USB 2.0.

Yes, always good to 100% understand what you're doing. You do need to stop the array to make changes, but that doesn't invalidate the parity. Here is a great video that might help: You actually don't need the extra step of erasing and then formatting, but you can follow it as is.

Then you would only need at most 2 rounds of conversions. If the total space used on all 3 reiserFS disks is less then the 1 parity disk then only one round. Here's how you would do it: Convert parity2 to a data disk.(Meaning temporarily change the config to a 1 parity system, and change parity2 to a data disk with 2x the space of any other drive.) For the above step I'm not 100% sure if you need to pre clear(write all 0s) the 2nd parity drive before you add it back into the array as a data disk. You may want to do it in 2 steps to be safe. Copy the contents of the reiserFS disks or 2 largest disks onto the new data disk. Verify all data(data integrity plugin) Reformat all the disks which data is duplicated to the new file system. Copy all of the data back. Verify all data(data integrity plugin) Repeat once more if needed. Change the temp data disk back to parity2 You'll still be protected by 1 parity disk the entire time and you're duplicating data so the risk is a bit less.

64hex = 100Decimal.

Bunker does not like spaces in dirnames when the dir is double quoted "/dir name/": \ /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker: line 91: $monitor: ambiguous redirect /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker: line 133: $monitor: ambiguous redirect but seems okay with it backslashed as /dir\ name/.

@bonienl May I make an enhancement request? Some of us like to run bunker from the command line and output STDOUT to a log file. I'd like a mode similar to -q (surpress all output) but to remove the "Scanning... spinning icon" output so it only outputs relevant information and also remove escape characters so that the output can go into a plain readable text file for a .log.

Thanks. Nice script. A couple of comments: You may want to run bunker -a on the $src_path before running the rsync command. bunker doesn't seem to really play nice with piping STDOUT to a log file. It keeps dumping "Scanning \-/-/" (spinning bar) into the log file. Your bunker -v command works for your case where everything is under /Test but it wouldn't work in general for different destination directories. Also it will verify Everything under /Test even though you only rsync'd /Test/Agoria and /Test/Peacock. For example if there's an existing /Test/HugeSizeDirectory, it would also try to verify that.

You can pre clear it over usb, with usb 3.0 it'll run just as fast as internally.

Use something like: find . -exec setfattr --remove=user.md5 {} \;

hashes are stored as Xattrs in the files. try getfattr -d filename. You can also export them to files on the boot usb.