RobJ

Read the second post on this thread about best practices. There is an explanation about UD showing the disk temperature. I think in your case this is what is happening. UD is not monitoring your drive so it looks to be spun down. Based on your quick test, the drive is spun up. You need to see what is spinning it up. If UD is showing '*' then it is not monitoring the disk at all. Ok so I added the default script to my disk and now UD aswell shows it as spun up and reports the temperature. Unfortunately I don't use that drive at all and it is empty. So I don't know what keeps it spinning. Is it possible to add a line to the script somehow that sets a default spin down delay or something? I had the same problem and finally gave up and uninstalled Unassigned devices. I wasn't using it, just liked having its extra capabilities in case I ever needed them. UD insisted on spinning up the drive constantly, even though I wanted it to just ignore the drive if it couldn't spin it down. There's a script option that is supposed to change the behavior somehow, but I never tried it (I know I should have, just to understand what it did and what the default script was). But that meant having to manage the drive and I didn't want to do that, I just wanted it to ignore the drive. There are times you'll have a drive currently unassigned, temporarily, for any number of reasons, and I shouldn't have to go into UD every time and 'manage' it, assign some kind of script to it, as it's probably a temporary situation. The script would be problematic anyway, as it's often just a random drive, and how would it know what its drive symbol was, unless Dan is passing its symbols in the local environment passed to the script. I don't think Dan believes that UD is spinning up the drives, and I don't know for sure what is causing it, but as soon as I uninstalled UD, the drive never spun up again. Ever. I use the spin down icon in the built-in Unassigned Devices section to spin it down shortly after boot, and it then stays down. It might be nice if both Unassigned Devices tools had a default action for 'unmanaged' drives, drives they don't have anything programmed for. The best default I think would be to issue a "hdparm -S60 /dev/sdX" to the drive. That would cause the drive to spin itself down in 5 minutes, and if anything spins it up, it will spin itself down again in 5 minutes. If desired, the -S60 could be changed to -S180 for 15 minutes, or -S12 for 1 minute. Hard coding the drive symbol is not going to work reliably, as it can change on each boot. How stable the symbol assignments are depends on the particular motherboard and the controllers added, and differs for everyone. You'll need to use a method that finds the drive by ID or serial number. When you hard code it to sdf, then the drive that is currently sdf will spin down, but it may not be the drive you expect.

Squid, it may be too early yet, but a check on the tunables for reasonable values might be good. The 6.2 series of unRAID releases have a new tunable md_sync_thresh that needs to be either half or very close to the value of md_sync_window, or you can get very poor write performance. It automatically defaults to 192, which is OK for users who haven't made any tunable changes, but if they used the Tunables Tester and raised md_sync_window much higher than its default, then they may see a large drop in performance. There are still only a few reports so far, so you may want to wait a bit, but once confirmed I think it would be a useful check, something users would be grateful for if it fixes a major slowdown.

We're waiting patiently for the 6.2 version.

Another idea - don't know if you'll like it or if it is easily feasible - 'target dilution'. Right now, each drive is target rich, and even after adding bait files, it is still almost entirely valid data files, relatively few bait files. So what if we add numerous dummy folders to the root with bait and additional subfolders to them with more bait. Let's say we add 100 dummy folders, each with 100 dummy folders, and each of them with 100 folders, all with bait files. Now our real folders are a tiny percentage of the possible branches from the root, even if we haven't added numbers comparable to the actual data file count. It has suddenly become a very diluted target, and that could allow us to remove all bait files from our true data folders and exclude them, possibly fewer watches needed. It still looks rich to the attacker, because they can't know which folder branch is valid, but the odds are much higher that they first go after bait files only. (I doubt unRAID will be important enough for quite awhile, for unRAID customized attack strategies.) There's always a downside, the first one I can think of is whether you can design the top level names such that Mover will ignore them. Does Mover still ignore folders that begin with a period? If so, then that would work great. If an attacker cares at all, then an attacker may actually be *more* interested in a dotted folder, because they may especially want to encrypt whatever the user is trying to hide.

I know you specifically asked gfjardim, but... you've hit the nail on the head. It's been a debate that's run as long as the preclear has been available. Some folks have had their drives fail on the 3rd preclear, so they always run 3+, some run 1 just to get it into the system. With version 6.x (maybe 6.1?) you don't actually have to preclear a drive anymore. unRAID will do it for you without taking the array off-line to do it. (It used to take the array offline while zeroing out the drive, thus you were down for hours to days, thus the need to preclear the drive.) I don't think there's a definitive answer to your question, though others are welcome to pipe in . I agree with this, and I think you'll get as many opinions as people you ask, good opinions too from knowledgeable users. I only Preclear once, and have never had a drive fail afterwards, but then again, I've never had a drive fail during a Preclear either! I'm guessing I've done about 6 to 10 drives. I know of another user, very respected, who Preclears 4 or 5 times, because he has had drives pass after 3 Preclears but fail shortly afterwards. My feeling is that you have to draw the line somewhere, and it's up to the user's personal preference. And besides, if you are adding the drive to your unRAID array, a fault tolerant system designed for drives to fail without data loss, then it doesn't really matter as much if it fails soon, just a little more work to replace than if it hadn't yet entered the array. Drives don't fail very often, and I suspect there are a few that feel the chance of failure is low enough, they don't bother to Preclear at all, just let normal operation expose the lemons.

Yeah, I couldn't decide where to put it, at all, so finally went for the safe choice. V6 support was the other best choice. I was hoping more would have opinions on that, yours is the first. I'll wait just a bit more, then probably move it there. I did think about that, haven't felt strongly enough about it I guess. If it starts to get buried pages deep somewhere, then yes, that should probably be done.

I understand. I felt I needed to make the point though, because the efforts above to restart SMB were concerning to me. I could see users watching a movie, notice a brief pause, then settle back as the movie resumed, and only check for notifications hours later, while the ransomware continues merrily encrypting other stuff it can find on the network. It really seems safer to bring everything to a halt, until an unRAID administrator makes the decision that it's safe to continue.

This has to be considered the most disastrous event that can happen to your server, worse than a hard crash, worse even than multiple drive failures and loss of parity. So I don't think you want to minimize the effect on the server, by trying to continue any streaming or transfers. A movie or music stoppage might be a terrific and instant 'notification' of imminent disaster. Perhaps it would be better to block all access, and put up a "Ransomware attack detected! Resume read-only access? (Y/N)" dialog, on the current unRAID screen (plus send all notifications of course). Then if they decide to, they can restart SMB access, resume the movie... But I suspect they will have more important matters to attend to! Like saving the server and figuring out which desktop is infected and now already encrypted! The first step has to be blocking access. I'd almost prefer a way to effectively take the server offline, "cut the network cable", but if stopping SMB, AFP, and NFS (and FTP?) effectively does that, and does it very quickly, then that's fine. The next step would be to reconfigure the server, all access and permission changes, and stopping unnecessary or risky services. I personally think the default action should be stopping the array, as it stops everything (except access to the flash disk(!), especially if the boot disk is exported read/write). I originally thought stopping the array should be the first action, but it takes too long spinning up the drives, so stopping server access has to be first. Then once the server is reconfigured to a safe state, send the notifications. Just thinking out loud how I think it should go, you may already have it doing all or most of that. You may want to spend more time with that CryptoDrop paper. There were a number of important points made from their studies of ransomware in the wild. One being that certain file types were attacked first by many of the attackers. As I recall, pdf's were possibly the first thing encrypted, and other similar high value doc types. I originally suggested jpg's because that's what we hear about most from anecdotal sources, but this paper says differently. Another important finding was that they tend to go for the smallest files first, presumably so they can encrypt more of them faster. Another very important finding was that many of them use randomized search strategies, ignoring the root folders, probably trying to avoid easy detections. This makes it a much harder target for you, and means the bait has to be pretty well everywhere.

I've PM'd lionelhutz, notified him of the problem and the suggested URL replacement, told him I'm leaving it up to him.

One problem, lionelhutz post leads with a URL that has now gone 404.

Apologies for the newbie question, but do I just add this line to the bottom of the syslinux.cfg file, below the: label Memtest86+ kernel /memtest Each labeled section of syslinux.cfg is a menu item on your boot menu. So adding there at the end would just be adding it to the memory test! You want to edit the append line for the menu section that you plan to run.

It might be nice to have the Bleeding Edge plugin mentioned in your first post for this thread, along with a description of the current added features, and a disclaimer that it's a constantly changing testbed, is temporary, and not as well tested, a beta thing? If you do add it, I imagine it would have its own little section, apart from the 'approved' ones. This is just my opinion, you may prefer it to stay somewhat obscure and private.

Yikes! Probably works though. Might want to appeal to Tom for a return to the simpler mdcmd commands. Eric gave them to me for another plugin thats fallen by the wayside. Its just hitting the webUI to do it. There is no "command" per se to accomplish starts and stops I think there used to be, long ago, something very simple like 'mdcmd start' and 'mdcmd stop'...

Yikes! Probably works though. Might want to appeal to Tom for a return to the simpler mdcmd commands.

I can easily implement a backup feature like that (good idea), but only with regards to autoupdates. I want to second this, as a great addition. Even with a delayed auto-update, I'd feel safer knowing I could always roll an update back. (I suspect a number of users would have liked to have this previously, so they could roll back the powerdown termination.)

No fundamental reason why they couldn't be that I'm aware of It could be more of an endpoint problem than an rsync problem. Can you display the commands you are using?

While you're watering the plants, a new feature to consider ... I have to admit the red error notifications are irking me some, informing me that a new plugin upgrade has not been performed. No need to go into it again, but I really feel this is wrong, as I'm rather sure that practically all veteran computer professionals turn off immediate auto updates, and wait a bit before deciding to upgrade. I don't need to bring up a recent auto-update that caused so much trouble. Even with the best of intentions and the best developers, stuff happens. I was thinking about it, and suddenly realized I would be fine with an auto update after a satisfactory interval, and if I had been fully notified. I would want: * full notifications about the update * a long enough delay to feel a sufficient number of other users had tested it * a long enough delay that the author had had time to pull it if needed * a long enough delay for me to check change logs and decide if I want to block the update I'm not positive what a good delay would be, but something in the range of 2 to 7 days seems good, with a default of 3 days (72 hours). I would possibly pick 3 days, but more likely 5 days. Because updates often happen in clusters, I would want the clock to reset with each update that arrived before the previous delay was up. So the feature suggestion is: add a third option to the CA Auto Update Settings per app - "AutoUpdate", "Don't Auto Update", and "Delayed Auto Update". Add a configuration option for "Auto Update Delay (in days)", with default value of 3 days or 5 days. Then change Fix Common Problems to check for "Delayed Auto Update", check whether the delay period is up, and only issue an Informational, not a Warning or Error, if a plugin is set to Delayed update and the delay interval is not over. I think this would satisfy many of us. One complication for me is that my server is often turned off for weeks at a time. Since I may not have received any notifications before the delay is up, I would prefer to have a chance to check first, before auto updates are performed. My case is probably not common, but it would be nice to have an additional internal check that notifications have been sent, before performing upgrades on the next run. I have it set to run daily, so that would give me one extra day, better than nothing.

I agree, that is awesome work, awesome graphs! Now if we just had some guides (hint, hint) Lots of smart people here, but many of us don't have time for more learning curves, would like already prepped and cooked products, ready to use, with a simple configuration screen for the particular components we want. The ideal would be a user designed Dashboard with those graphs, *easily* set up and configured. I suppose that that 'easily' and the time it takes is the problem that will block many of us. I'd love to see a component for long-term SMART history, among other things.

I have a new SAS drive. How do I get unRAID to recognize and assign it? Why can't unRAID find or add my SAS drive? Assuming you are running unRAID 6.2, go to Settings -> Display Settings and change Display world-wide-name in device ID from Disabled to Automatic. Click Apply then Done, and then go see if unRAID will let it be assigned!

Turbo Write technically known as "reconstruct write" - a new method for updating parity JonP gave a short description of what "reconstruct write" is, but I thought I would give a little more detail, what it is, how it compares with the traditional method, and the ramifications of using it. First, where is the setting? Go to Settings -> Disk Settings, and look for Tunable (md_write_method). The 3 options are read/modify/write (the way we've always done it), reconstruct write (Turbo write, the new way), and Auto which is something for the future but is currently the same as the old way. To change it, click on the option you want, then the Apply button. The effect should be immediate. Traditionally, unRAID has used the "read/modify/write" method to update parity, to keep parity correct for all data drives. Say you have a block of data to write to a drive in your array, and naturally you want parity to be updated too. In order to know how to update parity for that block, you have to know what is the difference between this new block of data and the existing block of data currently on the drive. So you start by reading in the existing block, and comparing it with the new block. That allows you to figure out what is different, so now you know what changes you need to make to the parity block, but first you need to read in the existing parity block. So you apply the changes you figured out to the parity block, resulting in a new parity block to be written out. Now you want to write out the new data block, and the parity block, but the drive head is just past the end of the blocks because you just read them. So you have to wait a long time (in computer time) for the disk platters to rotate all the way back around, until they are positioned to write to that same block. That platter rotation time is the part that makes this method take so long. It's the main reason why parity writes are so much slower than regular writes. To summarize, for the "read/modify/write" method, you need to: * read in the parity block and read in the existing data block (can be done simultaneously) * compare the data blocks, then use the difference to change the parity block to produce a new parity block (very short) * wait for platter rotation (very long!) * write out the parity block and write out the data block (can be done simultaneously) That's 2 reads, a calc, a long wait, and 2 writes. Turbo write is the new method, often called "reconstruct write". We start with that same block of new data to be saved, but this time we don't care about the existing data or the existing parity block. So we can immediately write out the data block, but how do we know what the parity block should be? We issue a read of the same block on all of the *other* data drives, and once we have them, we combine all of them plus our new data block to give us the new parity block, which we then write out! Done! To summarize, for the "reconstruct write" method, you need to: * write out the data block while simultaneously reading in the data blocks of all other data drives * calculate the new parity block from all of the data blocks, including the new one (very short) * write out the parity block That's a write and a bunch of simultaneous reads, a calc, and a write, but no platter rotation wait! Now you can see why it can be so much faster! The upside is it can be much faster. The downside is that ALL of the array drives must be spinning, because they ALL are involved in EVERY write. So what are the ramifications of this? * For some operations, like parity checks and parity builds and drive rebuilds, it doesn't matter, because all of the drives are spinning anyway. * For large write operations, like large transfers to the array, it can make a big difference in speed! * For a small write, especially at an odd time when the drives are normally sleeping, all of the drives have to be spun up before the small write can proceed. * And what about those little writes that go on in the background, like file system housekeeping operations? EVERY write at any time forces EVERY array drive to spin up. So you are likely to be surprised at odd times when checking on your array, and expecting all of your drives to be spun down, and finding every one of them spun up, for no discernible reason. * So one of the questions to be faced is, how do you want your various write operations to be handled. Take a small scheduled backup of your phone at 4 in the morning. The backup tool determines there's a new picture to back up, so tries to write it to your unRAID server. If you are using the old method, the data drive and the parity drive have to spin up, then this small amount of data is written, possibly taking a couple more seconds than Turbo write would take. It's 4am, do you care? If you were using Turbo write, then all of the drives will spin up, which probably takes somewhat longer spinning them up than any time saved by using Turbo write to save that picture (but a couple of seconds faster in the save). Plus, all of the drives are now spinning, uselessly. * Another possible problem if you were in Turbo mode, and you are watching a movie streaming to your player, then a write kicks in to the server and starts spinning up ALL of the drives, causing that well-known pause and stuttering in your movie. Who wants to deal with the whining that starts then? Currently, you only have the option to use the old method or the new (currently the Auto option means the old method). But the plan is to add the true Auto option that will use the old method by default, *unless* all of the drives are currently spinning. If the drives are all spinning, then it slips into Turbo. This should be enough for many users. It would normally use the old method, but if you planned a large transfer or a bunch of writes, then you would spin up all of the drives - and enjoy faster writing. Tom talked about that Auto mode quite awhile ago, but I'm rather sure he backed off at that time, once he faced the problems of knowing when a drive is spinning, and being able to detect it without noticeably affecting write performance, ruining the very benefits we were trying to achieve. If on every write you have to query each drive for its status, then you will noticeably impact I/O performance. So to maintain good performance, you need another function working in the background keeping near-instantaneous track of spin status, and providing a single flag for the writer to check, whether they are all spun up or not, to know which method to use. So that provides 3 options, but many of us are going to want tighter and smarter control of when it is in either mode. Quite awhile ago, WeeboTech developed his own scheme of scheduling. If I remember right (and I could have it backwards), he was going to use cron to toggle it twice a day, so that it used one method during the day, and the other method at night. I think many users may find that scheduling it may satisfy their needs, Turbo when there's lots of writing, old style over night and when they are streaming movies. For awhile, I did think that other users, including myself, would be happiest with a Turbo button on the Main screen (and Dashboard). Then I realized that that's exactly what our Spin up button would be, if we used the new Auto mode. The server would normally be in the old mode (except for times when all drives were spinning). If we had a big update session, backing up or or downloading lots of stuff, we would click the Turbo / Spin up button and would have Turbo write, which would then automatically timeout when the drives started spinning down, after the backup session or transfers are complete. Edit: added what the setting is and where it's located (completely forgot this!)

You do make me laugh! I still laugh *every* time I see your 'hoovering' comment!

I noticed that every time I used the script to clear a 250GB drive, some time after it unmounted it (didn't watch so don't know how soon), the Main screen showed a valid disk mounted there but with a little over 2GB size drive there, and a little of it used, don't remember the numbers. I just assumed normal wonkiness when I confused it by unmounting it, and never investigated further. What you found sounds exactly the same. I never noticed any issues. When the clearing finished, I stopped the array and went to New Config, which essentially reset everything. A parity check found nothing wrong. I saw the same lines in my syslog. I think this is something that Limetech needs to look into. Not your script, but the auto creation of a disk folder not linked to the underlying md system. I can imagine some very strange things happening if a disk is unmounted and things are written to the phantom drive. I didn't test to see if the newly minted disk1 would participate in the user share system, but I assume it could. I don't think this is right. I think if LimeTech was asked about it, they would correctly reply "Don't do that! Never manually unmount an array drive". What the script is doing is completely outside the normal operation, could be called a hack. I suppose it's like taking a hammer but using it backwards, hitting the nail with the handle. If the handle cracks, would you complain that the handle is no good? It wasn't meant to do that. I should probably put disclaimers somewhere that this is a completely unofficial hack, tested and seems to work but unapproved by LimeTech. None that I know of. Maybe one, if the drive is detected to have failed, unRAID may unmount it. Perhaps I should somehow cause the drive to appear to have failed...

Have you tried the Docker FAQ?

I noticed that every time I used the script to clear a 250GB drive, some time after it unmounted it (didn't watch so don't know how soon), the Main screen showed a valid disk mounted there but with a little over 2GB size drive there, and a little of it used, don't remember the numbers. I just assumed normal wonkiness when I confused it by unmounting it, and never investigated further. What you found sounds exactly the same. I never noticed any issues. When the clearing finished, I stopped the array and went to New Config, which essentially reset everything. A parity check found nothing wrong. I saw the same lines in my syslog.

Cool. I always thought it was powerdown that added that To the best of my knowledge, using 6.2, the Powerdown plugin first saves a dated diagnostics to /boot/logs, then a dated syslog to /boot/logs, then an undated zip of that syslog to /boot/logs, then the unRAID shut down process saves an undated syslog.txt to /boot. The /boot/logs/syslog.zip and /boot/syslog.txt are overwritten each time, and the Powerdown plugin removes all previous diagnostics (something I want to change!), so there's not much growth.