little_miss_goth

hi, ok, sooo..... just in case anyone has read my last post..... erm ....so... obviously I don't use the webgui very much... turns out that just issuing killall -g cache_dirs can end up b0rking emhttp... :-/ Apart from that, it seems to work perfectly! :'( I only came across this today because I was trying to add scripts to make cache_dirs go up and down with the array rather than from the go and stop scripts... and somewhere in my experimenting I pressed the 'stop array' button and the moment I did, I ran slap bang into a completely unresponsive webgui.... when the array is stopped as part of system shutdown, you're expecting the webgui to stop working, and because I tend to shutdown from the shell, not the webgui, its even less apparent.... I think I've worked out whats going on though; what I think is that when its being started by scripts that live in /usr/local/emhttp/plugins/*some_plugin*/event/ , cache_dirs runs as part of the same process group as emhttp, and when you issue the killall -g cache_dirs to kill the process group, it takes out emhttp at the same time, what with them being in the same group.... If the processes are stopped manually (using ps and kill to stop cache_dirs and find in sequence, lowest PID to highest), emhttp doesn't b0rk... Been trying to figure out what to do about it for a good few hours, even wrote the large part of a script that would carefully the individual kill the main cache_dirs process, its children cache_dirs and the find processes (and even check the at-queue for scheduled cache_dirs jobs) And then I fell across setsid , which I'd forgotten alllll about...! Turns out that if you start cache_dirs by doing setsid /path/to/cache_dirs -w ... then cache_dirs gets its own process group, and killall works as expected, even when stopping the array from array event scripts :-) So sorry if anyone has run into this. -Jo

Hi all, OK, so this is my first post in the forums, so please be gentle! 1st up, many thanks to Joe for the cache_dirs script; most definitely a Good Thing to have on an unraid system :-) So, I've been running unraid v5 since about 5b5 (I think), and have been on the stable 5.0 release since not long after it came out. *was* running a combination of 2TB and 3TB SATA array drives for a total of 17TB, with a non-array 300GB IDE disk providing me a swap partition (not strictly necessary with 4TB, I know, but I've been using Linux since 1996, and old habbits die hard!) and somewhere to keep data that didn't need to be protected cos I have it burnt to dvd). When I set the machine up for unraid, I went down the "oh, that could be useful" route, so I had unmenu, powerdown, various additional unmenu packages (like htop, powerdown, etc) and a rather customised way of launching them so that I could store the majority of them on the non-array disk and kind of inject them into the live unraid filesystem, inspired by the way Tiny/Micro Core linux can (or atleast could when I last used it) be configured to do package persistence. Anyway, a couple of months back, I had a combination failure of an array drive and a molex/sata adapter (with impressive/scary amounts of smoke billowing from the machine!)... the result being a completely b0rked 3TB drive and some very melted/charred wires. The long and the short of it is an upgraded array with 25TB on 4TB Parity with the 300GB drive now set up as cache (although not actually *caching* any of the shares.... its doing the same job as before (even down to having a sneaky 2nd partition set up for swap), its just that its now formatted with BTRFS so that I have the option of playing with Docker if I fancy) and a bigger PSU with lots of SATA power connections (eliminating what would have been 6 molex/sata adapters for the now 10 SATA drives) and running unraid 6b6 with powerdown 2.04 the only 'extra' added to the stock unraid setup (except for the minute tweak of moving emhttp onto port 8000). While I'd been poking in the forums to find out as much as I could about the changes in powerdown 2.x *before* I tried it, I came across the cache_dirs script and, since I've got a reasonable amount of data on my array in a various shares, it seemed like it would be a very nice addition. I don't run my server 24/7 (mainly cos its audible from where I sleep, but also to keep electric use down), so there are only really two possibilities when its powered up; I'm either actively doing something that uses the array or I'm not (because I'm either going to or have already done so)... so I have my spindown time set to 15 minutes, which suits this usage pattern right up to the point when what I want to do is quickly check if I have already got something on the array or am hunting for a specific thing that I know I've got stored on there *somewhere*... at that point, it becomes a right royal PITA,... quickly rapidly becomes slowly..... as multiple spinup delays have to happen My unraid system is my main storage for *everything*.... so it has a Software share with all *sorts* of software installers, updaters, boot images, etc from/for all *sorts* of OSs on it, a TV share with dvr'd movies and tv shows, a Media share carrying music and e-books and a final share that acts as my archive of personal *stuff*.... its got A Level and Uni notes, software designs, photos, etc, etc thats been accumulated over some twenty years of having atleast one PC.... this share is by far the least organised and most messy of the 4, and its also the one that suffers the worst from spinup delays when I'm trying to find things, as its distributed across pretty much every disk in the array (and yes, I know I should amalgamate it on one disk, that would help; I just never seem to find the time to shuffle it all around!) So yeah, I've only been using it a couple of days but cache_dirs is *brilliant* for me as it has mostly eliminated those spinup delays and made me a very happy bunny! :-) However, I did come across a little niggle that I'd like to report... OK FYI: I'm running cache_dirs from my /boot/config/go script as /boot/tools/cache_dirs -w -u -B I am (purposefully) not using a maxdepth value with cache_dirs (because my archive and software shares are the trees that will benefit me the most from being cached as they are my "I know I've got that somewhere" go-to locations and they are both quite deep and range from being poorly organised to completely not organised at all...!). This unrestricted depth means that the first cache_dirs scape takes a pretty long time (ie, ballpark 20mins). Not (in and of itself) an issue, but because my first cache_dirs scrape takes a long time , I come across something that I suspect most people won't.... which is that if I attempt to use cache_dirs -q in preparation for shutting down the unraid system while the first scrape is still in progress (ie, there is a 'find -noleaf' process running that was spawned by cache_dirs), only the cache_dirs process who's PID is stored in the LCK file ends and I still have a cache_dirs process and a find process listed when I do: ps -ef | grep "\(find\|cache_d\)" | grep -v grep Attempting cache_dirs -q for a second time, outputs a statement: cache_dirs is not currently running (presumably because the LCK file no longer exists), even though ps shows it is; indeed, the cache_dirs process that is still running shows in ps as having now been re-parented to PID 1, since it was orphaned when the process referenced in the LCK file ended. Keeping an eye on the process list, the find that was running when the 'quit' was signalled runs to completion, and another find starts, so it looks to me as though the cache_dirs process that is still running will continue to run until it completes, regardless of the 'quit' having being signalled. But... I got bored waiting for that to happen, so I decided to kill cache_dirs and find. Just killing that orphaned cache_dirs process doesn't stop the find, so I ended up using killall -g cache_dirs which immediately stops the orphaned cache_dirs process group (ie cache_dirs ands its child find(s)). Now, admittedly this is a less-than-usual use pattern, but from my reading about powerdown, I know there is atleast one user who experiences frequent (ie, multiple powerouts in short succession) power outages; when their power goes out, the powerdown script is invoked (ie, via UPS signalling) to gracefully shutdown the array and machine, so I don't think its too far-fetched that someone could run into this issue by being unlucky enough to suffer a power outage during cache_dirs initial scrape... I am now using the following bit of scripting to stop cache_dirs on my own system (so I don't end up with a modified cache_dirs scripts to maintain): #!/bin/bash cache_dirs -q sleep 2 killall -s SIGTERM -g cache_dirs But if Joe felt it worthwhile, I think the 'quit' handler could simply be tweaked to use killall -g "$program_name" instead of kill "$lock_pid" I thought that I should probably report this, as it could be an issue that someone might run in to (albeit infrequently) that could end up with an array disk being ungracefully unmounted, thus triggering a parity check on the next boot for no obvious or easily-identified reason. Editted to include the possible tweak, and again to make it use $program_name not $lock_pid