drumstyx

Hooray! You did it! What a weird problem lol Now my first world problem is that my internet is actually faster than my SATA drives (not to mention the disk shelf they're in is a DS4246 so even turbo write peaks at about 85MB/s), so I can't rely on the mover to keep up with data intake. Maybe I can throttle nzbget at a certain cache usage threshold...

Fair enough -- I've attached diags. I'm not quite sure where to look in them for this issue. As far as I can tell, there have been no dropouts or errors on the cache other than a failure to write due to insufficient space. datnas-diagnostics-20221230-1334.zip

Right, but why is there a filesystem issue when it's "full"? Isn't the global reserve supposed to prevent this? I guess I didn't have min free space configured on cache, so my bad on that, but filling a filesystem shouldn't result in unresolvable readonly. If I accept the risk of some corrupted data in whatever was last written, is there any way to just say "to hell with integrity, delete this file"?

Long story short, I upgraded to 3gbps fiber internet, and accidentally filled my 2tb cache pool (dual 2tb nvme drives) with the automated download processes. The filesystem went readonly, and I tried to just reboot, and of course, readonly again. On top of this, a disk in my array happened to fail a couple days ago, and I didn't notice, so I'm running a rebuild on a warm spare I had in there to replace it. Problem is, now I can't run mover, and of course even after copying manually, I can't delete anything to free up space. The global reserve is at 512MiB and says 0 is used, but still, it's just not happy. The rebuild means I can't go into maintenance mode right now. I'll wait it out if I have to, but it'd be awesome if I could just figure out how to delete even a single file from the cache to free it up a bit. Scrub doesn't work (readonly) balancing doesn't work (readonly)....seems like I'm stuck? It's frustrating because I went dual btrfs on the cache to prevent issues like this -- the suggestions I see on here have been "best bet is to backup cache, reformat and rebuild cache".

Does this happen to have tools that used to be included in the old devpack plugin? Awesome to have an equivalent for nerdpack, but devpack was also indispensable for some uses! Seems to be a VERY similar codebase, so likely not a difficult port, if you're up for it!

config/stop runs very early in the shutdown process -- not sure if you meant to quote me where I said "To run things LATE in the shutdown process is hard" there, but it holds true. It works great for various full-stack cleanup, or external notifications/hooks, but is technically not the best place for commands that affect the status of the server hardware (like my much earlier posts about turning the AC power off at the plug).

Old, but still relevant -- bump?

The stop script is just about the earliest you could want anything to run. Shutdown is initiated with /sbin/init 0, which sets the runlevel to 0, at which point the absolute earliest hook is the first line of /etc/rc.d/rc.0 (after "#!/bin/bash", of course). The number after "rc." in the script name is the runlevel. Runlevel 0 is shutdown (aka "halt", officially) and runlevel 6 is reboot. In Unraid, rc.0 is just an alias for rc.6, and one script handles both things, as it looks to see which runlevel it was called as. The only scripts that run before /boot/config/stop are scripts within /etc/rc.d/rc0.d/ or /etc/rc.d/rc6.d/, depending on runlevel, which at this point contains only the flash backup of the myservers plugin, if applicable. All that to say, to run things EARLY in the shutdown process is easy -- /boot/config/stop. To run things LATE in the shutdown process is hard, and would involve either modifying the base unraid image on flash, or adding something in the go script to modify rc.6 to your liking on boot (either copying a complete file, which would have to be modified manually with each OS update, or some awk/sed/bash magic to modify the last if statement in rc.6)

I've experienced various hardware failures over the years, and they've been a pain in the arse! Not the fault of UnRAID, just because I didn't take all the precautions necessary -- failed cache, failed drives, failed USB. Thing is, I can mitigate almost everything by just having some redundancy -- dual parity, cache pool, etc. which I've now done. Got this thing almost bulletproof, but for one thing: A USB drive failure. Is it possible, or are there any plans to make it possible, to have a boot pool or redundancy for the boot drive? I know that, in theory, the boot drive is only needed for power on and power off (actually, not even sure it's needed for power off?) but I do write a rotating syslog to my boot drive because it's very useful for debugging array and system bugs. I could write to an unassigned device, or the cache, but the boot drive is guaranteed to exist as long as the system is running, and if any of those other drives drop because of a system-wide failure, I don't want to lose the logs. I'm willing to sacrifice a boot drive every few years, but I'd REALLY like to not have that cause an outage for which I have to be physically present to fix.

Long story short, I had a cache drive failure (ADATA nvme ssd, quite a surprise) and after recovery I'm looking to improve reliability. I've picked up a 2TB Samsung NVMe drive to complement the ADATA warranty replacement when it arrives (also 2tb) Thing is, I know the process when an array drive fails, but I have no idea what happens when a BTRFS drive fails in a pool. I've read some horror stories about a failed drive causing an unrecoverable situation. Of course I back up important stuff to the array (and important stuff on the array is backed up offsite) but recovering from that is still a pain in the butt. So, can anyone walk me through what a BTRFS cache pool drive failure looks like?

Does Unraid work fine running on cores other than 0 these days? I remember when I first started playing around with pinning/isolation years ago, Unraid didn't play nice if I pinned/isolated core 0, even if I left, say, core 3 (the last core on a quad core CPU) entirely untouched. So I've always just let Unraid have the first core, which is no problem with homogenous CPUs. If I went 12th gen though, I'd really hate to give up a P core if I could hand it an E core for its minor management stuff.

Got it -- runlevel is 0 for shutdown, 6 for reboot, 3 for normal operation For posterity, here's my hacked-together script for it: #!/bin/sh currentRunLevel=$(runlevel | awk {'print $NF'}) echo "*********** THE CURRENT RUN LEVEL IS ***********" echo $currentRunLevel if [[ $(echo $currentRunLevel) = 0 ]]; then snmpset -v 1 -c private 192.168.42.20 1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.1 integer 5 snmpset -v 1 -c private 192.168.42.20 1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.2 integer 5 snmpset -v 1 -c private 192.168.42.20 1.3.6.1.4.1.318.1.1.12.3.3.1.1.4.5 integer 5 fi The echos at the top were just for me to test that runlevel was changed before the script is executed, and indeed it seems it can be relied on. To add: this script runs pretty much immediately when a shutdown is triggered (at least from GUI). BEFORE unloading anything. Technically, that means it's not an ideal spot for it, but since unraid force kills things after 90 seconds (at least, that's what I assume when it says it's allowing 90 seconds for graceful shutdown) my timers getting triggered over snmp are fine at 180 seconds. The only downsides to the timeouts is that coming back up after power loss is a rather long process (I've got a raspberry pi that listens for UPS OFFBATTERY state and brings things back, but it needs to wait longer than the timeout to make sure the server isn't currently trying to shut down), and power is consumed from the UPS for longer than is really necessary, but only a minute or two extra. You just can't make the timings too tight, otherwise if there are any hangups during shutdown, it could kill the power too soon. EDIT: This isn't a thread for SNMP specifically, but I thought I'd note what the heck those long strings are: OIDs for the outlets on my PDU. APC provides MIB files to use in an MIB browser, and with some digging you can figure out how to control various things. It's a pain, not to mention archaic, but it's neat once it's set up. I'm positive I'll hate myself if I ever move outlets around and forget I set this up, but for now it's a blast!

Come to think of it, this will probably run on both reboot and shutdown -- if the outlet timeout is long enough, and the scripts robust enough, it shouldn't be a problem for functionality, but it could slow down reboots considerably. Any ideas as to how I might ensure it only runs on shutdown?

Ah, this could be perfect -- do you know if it runs before or after networking is unloaded, or can you point me to what script calls the stop script? The most ideal place to put this script is RIGHT before the poweroff call, just like a UPS killpower command, but just like the UPS killpower command, this won't work over SNMP because networking is already unloaded by then. Annoyingly, filesystems appear to be unmounted after stopping networking, so the only real option is to have a long poweroff delay on the outlets and send the request at the beginning of the shutdown. Probably fair to assume it's before the rc.0 script is called at all, so I'll give it a shot!

Looking to run a script that shuts down my disk shelves on unraid shutdown. I've configured poweroff delays on my switched PDU, and I'm sending snmp commands to the PDU to make it work. Problem is, I'm using /etc/rc.d/rc.0 as the script location, and this is in the RAMdisk, so it doesn't persist across reboots. There used to be a powerdown script, but that's apparently been deprecated, and rc.0 just calls /sbin/poweroff, which is a binary. Basically, I'm looking for the shutdown equivalent of the go script

I'm trying to fine tune my UPS power down and power restored workflows. I run 2 DS4246 disk shelves, and they need to be powered up for a minute or two before unraid boots, to avoid panics about missing disks. I've got a switched PDU, and a UPS with an outlet group. The outlet group on the UPS powers non-critical systems that I want to come back after a power outage, but not only my unraid server, so I can't just switch that off/on based on power status. So I've been working to directly switch the PDU outlets. Honestly, it's a long story, but the bottom line: I need to be able to power up those 2 disk shelves when unraid boots, and then tell it to wait a couple minutes for them to spin up, then continue with mounting the array. My understanding is that the go script runs asynchronously to array booting and mounting, so that's out, and user scripts simply specifies "AT" array start, not whether it's run before or after. What are my options here?

Heck, I'd even pay for multiple licenses to make this possible -- I already own 2, but I don't want to run 2 whole machines and deal with network drive mapping for sharing media between them

Very nice, super handy! A few feature requests: Ability to reorder disk tray layouts, even if just with an up/down button, or even as simple as an order field. I've got hotswap units in my main server (3x4 drives) and one of them is sideways, plus a DS4243 shelf, so I have 3 configs, and I accidentally ended up with a couple out of order. Minor issue, but it just looks strange to see 2 before 3. Line breaks in dashboard view and tools area.

Ah yeah, makes sense -- I couldn't figure out how to specify an older version of the docker via GUI, but I suppose I could do it in console...

Having issues with python cryptography -- looks like py3-openssl was updated just a few hours after the current latest version was updated, and is causing issues because py3-cryptography is outdated now? I'm no expert, just a bit of digging. Error is: pkg_resources.ContextualVersionConflict: (cryptography 2.6.1 (/usr/lib/python3.7/site-packages), Requirement.parse('cryptography>=2.8'), {'PyOpenSSL'}) EDIT: In the meantime, running this in console and restarting works fine, though it has to be done each time the container is recreated (edited, etc) apk add gcc musl-dev libffi-dev openssl-dev python3-dev; pip install cryptography --upgrade apk add gcc musl-dev libffi-dev openssl-dev python3-dev; pip install cryptography --upgrade

Ah! Yes, I'll do exactly that. I was thinking that might be a problem, but I couldn't find where to change what folder Sonarr uses for /tv

I'm trying my best, but I just can't figure out why the heck hard links aren't working as expected -- No matter what I try, sonarr will make a .backup~ version of the file before moving it to its final resting place. My setup: Downloads come via usenet onto the cache ssd, and are direct unpacked to the non-cached array (essentially, final resting place, ideally). I would expect Sonarr to do a simple mv on the file, which would just move the pointer, or at the very least just make a hard link, and delete the old hard link. It does neither (despite settings saying use hardlink instead of copy), copies to the same folder with a different extension, and then apparently moves THAT and deletes everything in the initial folder to clean up. NZBGet's output directory, Sonarr's /downloads directory, and Sonarr's /tv directory are all on the same share...

Not that I don't appreciate this container, but I've put a script together to bypass this whole problem with rclone. If you have issues with speed, this will help. It requires that you've set up your FTP/SFTP server as a remote with rclone, but set this up with a cron of whatever your schedule in davos would have been, and tailor it to your own needs. remote_dir="remotename:/path/to/complete/items" # wherever your files are coming from -- make sure only completed files are here local_dir="/local/working/directory" # davos' initial download folder local_completed_dir="/local/complete/directory" # equivalent of davos' move after complete log_dir="/log/dir" function scrape () { if rclone lsf $remote_dir | grep . -c then if rclone move $remote_dir $local_dir -v --delete-empty-src-dirs --buffer-size=2G --stats 5s --transfers 4 2>&1 >>$log_dir then echo "scrape succeeded into $local_dir" if mv $local_dir/* $local_completed_dir then echo "mv succeeded into $local_completed_dir" else echo "mv failed to move to $local_completed_dir" return 1 fi else echo "rclone failed on $local_dir" return 1 fi else echo "No new files" fi } scrape || exit 1 EDIT: And here's a sloppy rsync version, which allows for as-you-go 'completion', where the other script doesn't have anything in 'complete' until the whole job is done. This one's probably more delicate as-is (like I said, sloppy) but it does what I want better. Depends on parallel, rclone, and rsync remote_dir="remotename:/path/to/complete/items" # wherever your files are coming from -- make sure only completed files are here local_dir="/local/completed/files" # where things will end up -- rsync uses temp files, so no need for a working dir mount_dir="$local_dir/tempmountpoint" # a temp folder where the remote server gets mounted while syncing function scrape () { echo -n "files found: " if rclone lsf -R --files-only $remote_dir | grep . -c then echo "attempting to mount" find $mount_dir/ -depth -type d -empty -delete &>/dev/null fusermount -u $mount_dir &>/dev/null mkdir $mount_dir &>/dev/null if rclone mount $remote_dir $mount_dir --daemon then echo "mount to $mount_dir succeeded" echo "sleeping a few seconds for mounting to be complete I guess" sleep 5 echo "making local directory structure to fill" find $mount_dir -type d | sed 's|'$mount_dir/'|'$local_dir/'|' | tr '\n' '\0' | xargs -0 mkdir -p echo "trying to rsync with parallel" if find $mount_dir -type f | sed 's|'$mount_dir/'||' | parallel -v -j8 --progress --line-buffer rsync -v --progress --stats --remove-source-files $mount_dir/{} $local_dir/{} then echo "rsync succeeded into $local_dir" find $mount_dir/ -depth -type d -empty -delete &>/dev/null fusermount -u $mount_dir &>/dev/null echo "attempted to remove empty directories from $mount_dir" else echo "rsync failed to move to $local_dir" return 1 fi else echo "rclone failed to mount $remote_dir" return 1 fi else echo "exiting" fi } scrape fusermount -u $mount_dir &>/dev/null find $mount_dir/ -depth -type d -empty -delete &>/dev/null exit 0 EDIT: Tailored the script above for better performance on large sets of files. I'm hitting speeds in excess of 800mbps if there's enough data to keep transferring -- it happens so fast that it sometimes can't even hit its peak. To @Stark, if I could figure out how to get a better handle on the parallel output, something like this could replace the internal FTP client (which to be fair is very difficult to wrangle, it seems), while keeping the management and scheduling system. rclone's configs are machine readable/writeable, and rsync *does* provide pretty progress output when it's not parallelized, but parallel butchers it. Worst case, that could probably be faked by reading the file sizes and measuring the total transferred, but that would just give you batch-level progress.

Hey @Stark, really appreciate the response! I definitely feel ya that it's a personal project -- maybe I'll see what I can do to chip in with enhancements. Speeds are confirmed by the overall network speed in the stats plugin, the readout on davos is buggy, so I knew to look elsewhere, but ultimately not a concern. It's quite possible that it's an issue of lack of parallelism. IIRC, even using Filezilla, transfers from the seedbox don't reach full potential unless I'm downloading either multiple files at once, or large files in chunks. For my use-case in particular, I often download large sets of files, so dead-simple parallel downloads might be the way to go. I'll see if I can get some time for this. Heck, realistically, rclone could probably handle my case, it just didn't have a pretty UI at the time.

After finally getting everything tidied up on my server such that the hard drive write speed can't possibly be the bottleneck, I can finally see that, indeed, Davos only copies at 24mbit/s at max, often around 8-12. Again, this isn't a *huge* problem because of my automated workflow, but I have a gigabit connection, and drives that can write at at *least* 30MB/s. Faster if I ever take my 1TB out of the array. What could possibly be causing this?