SSD

I am bjp999, not bjp2006! With 28 drives I think dual parity is worthwhile, although i would stop short of saying absolutely necessary. Being here for a long time I've seen a lot of issues, and very very very few where dual parity would have helped. The vast majority of issues are related to bad or loose cabling, and worse yet, users that don't recognize that is what is going on and make matters worse. If you have drive cages, solid wiring, monitor smart reports, and periodically replace drives as they get old, I could argue that dual parity is a nice to have and not a necessity. And lacking those things, I'd recommend those things before putting dual parity in place. And I'd also say 28 drives is a lot. If you have strung together a fleet of aging small drives, and assembled into a large array, I don't recommend it. But if they are relatively large, modern drives it should be fine. unRAID does not give up, from what I understand (never had it happen myself). It would continue to rebuild on a best effort basis. Unfortunately, even if you knew that it had happened due to a log entry, you'd have not way to know what file was impacted unless you kept checksums. I'd say this is very different than a RAID rebuild, one error and it stops and invalidates the array. Maybe good for an enterprise where the requirement for absolute data integrity is paramount. But not for a media array, where you'd rather have a randomly corrupted JPG than be pushed into doing a full array restore.

I guess it depends on the definition of "bitrot". If bitrot means that data is returned by a drive that is different than the data that was actually written (due to decay of the signal on the media, not due to something like bad cache in the drive), I agree with you. But if bitrot is related to the gradual degradation in magnetic signal on a disk making that data harder and harder to read over time, I DO think that indeed does happen - and is, in fact, is happening on a continuous basis. And real "bitrot" is when the degradation reaches a point where the data cannot be reliably read even with the error corrections built into the drive, causing the drive to have a read error. The question then becomes, at what point has the signal degraded to the point that it becomes unreadable? Is that a year, 5 years, 25 years, 100 years? 1000 years? Is it consistent across the media, or are some spots weaker in magnetic material. Or are some heads less sensitive imprecise in head placement to make a less than perfect signal on the disk harder and harder to read and correct. When we rebuild a disk, we are in fact re-writing every sector. This, in itself, would help us protect against bitrot. In the old days, Spinrite had a mode where it would do exactly that - rewrite every sector on the disk with exactly the same data. But I agree we have not seen anything we can ascribe the bitrot in the unRAID forums. So have to believe the timeline for the magnetic decay to be pretty lengthy, at least 7-8 years. But if one were worried about it, they could rebuild each disk onto itself or around its 3rd or 4th birthday, and feel better that the signal is refreshed and bitrot pushed out in time. Who knows, drives may start working harder and harder in the presence of magnetic degradation, prior to true bitrot, triggering retry cycles we are not even aware of, and adding wear and tear to the drive. (Of course drives that are continuously updated with new files would be less susceptible to bitrot, although perhaps become susceptible to bitburnout, a very different phenomenon! )

@shEiD @johnnie.black @itimpi Oh how we love to be comforted! While it is true that the mathematics show you are protected from two failures, drives don't study mathematics. And they don't die like light bulbs. In the throes of death they can do nasty things, and those nasty things can pollute parity. And if it pollutes one parity, it pollutes both parties. So even saying single parity protects against one failure is not always so, but let's say it protects against 98% of them. Now the chances of a second failure are astronomically smaller than a single failure. And it does not protect in the 2% that even a single failure isn't protected, and that 2% may dwarf the percentage of failures dual parity is going to rescue. I did an analysis a while back - the chances of dual parity being needed in a 20 disk array is about the same as the risk of a house fire. And that was with some very pessimistic failure rate estimates. Now RAID5 is different. First, RAID5 is much faster to kick a drive that does not respond in a tight time tolerance than unRaid (which only kicks a disk in a write failure). And second, if RAID5 kicks a second drive, ALL THE DATA in the entire array is lost. With no recovery possible expect backups. And it takes the array offline - a major issue for commercial enterprises that depend on these arrays to support their businesses. With unRaid the exposure is less, only affecting the two disks that "failed", and still leaving open other disk recovery methods that are very effective in practice. And typically our media servers going down is not a huge economic event. Bottom line - you need backups. Dual parity is not a substitute. Don't be sucked into the myth that you are fully protected from any two disk failures. Or that you can use the arguments for RAID6 over RAID5 to decide if dual parity is warranted in your array. A single disk backup of the size of a dual parity disk might provide far more value than using it for dual parity! And dual parity only starts to make sense with arrays containing disk counts in the high teens or twenties. (@ssdindex)

Just watched part2 last night. Great video, as always! A couple questions / comments: 1 - I know that unRAID prefers core0, so as a result I had avoided pinning VM cores there. In an earlier incarnation of unRAID VM a couple years ago (Xen), there were cases where core0 would get overloaded and unRAID would starve and fail. Is that a concern with KVM? Assuming no, I will start using core0 for VMs. (I had not been assigning Dockers by core or load, but may start to look at that). 2 - We continue to recommend that cores be pinned with their hyperthreaded twins. There are cases where multiple cores cause problems, and I can only pin one logical core. Does it matter whether pinning core0 vs its twin? What is the negative of assigning one VM to one core, and another to its twin? (I know its not recommended, just don't understand exactly why and if you had done any testing.) 3 - Very interesting approach of having multiple Docker containers linked to same appdata. (I had done this when moving to a different Docker for the same app). Makes sense to have a "High Transcode" configuration for Plex! You'd want to update to the next version in tandem (although you could update one, check it out to make sure it is working properly, and only then update the other). And be careful to not let them run together, as you said. Thanks!

Why? Inside the VM you'd be able to access the disk using SMB. Since it is local to the same machine, it is quite fast. But if you really needed to pass it through this would not be the place to ask. Check out SpaceInvader One's (on youtube) videos on passthrough. I know you can pass through a SATA controller and pass the disk through that way. Not sure if you can just pass through a single disk.

The whole idea of Dockers is that they contain functioning code and no configuration data. With the docker templates stored on your flash, you can rebuild them very fast. I recently recreated my docker on a fresh SSD. Took about 10 minutes. I'd focus your backups on your irretrievable data. I'd probably including the "appdata" share on the cache, at least for some Dockers. If you have VMs (domains share), you might want to periodically back them up as well. But they can be big and it really depends on how much time it would take you to reconstruct. Even with VMs, you should be externalizing your data onto the array so backing them up may very well not be necessary, at least not very often.

Hey - we all give our time. Have any idea how many hours I put into unmenu/mymain? The faster preclear script? The wiki? I've handled some of the most complicated data recovery scenarios in the forum taking hours of research and extensive step-by-step instructions. I'm not demanding anything of anyone. I just made a suggestion - anyone should be able to do that - and expect a minimum of a friendly response thanking for the insight. @johnnie.black also has requested the use of the unRAID disk format. It would be quite useful and users would benefit. @dlandon - thanks for your excellent contribution on UD. Putting it all together into this plugin is quite a lot of benefit to unRAID users. On this topic, we'll just agree to disagree. Peace.

You're a hard man @BRiT! But it is a fair point and I agree with you this far - users should have backups of their critical data. But problem is, many do not have backups at all. And even those that do, may not backup voluminous data that is "recoverable in other ways", that they take a calculated risk they'd never have to recover. Does that mean the support forum should not promote an option to maximize the value of unRAID and offer the very best chance of recovering the data? And give the user the control to decide which data is most valuable and direct that it be the priority to recover? Offering such a user a way to put in a new disk, partition and format it so that it could be added into unRAID later, and using Krusader, mc, cp, or something else to copy the data over in a selective manner, is what I would be advising. And today, in order to give that option, I need to explain a complicated multi-reboot process to format their new 8T drive for unRAID and mount it outside the array. I'd rather explain to install UD, do these three clicks, and they are ready to go. If the array is too far gone, and the user doesn't have backups, shame on them. But if the array is recoverable to at least some degree, and we as the support forum do not make the option available to give them the best chance of recovery, shame on us. That's how I feel anyway.

I never said that I didn't trust unRAID to rebuild a disk. Or I was desperately in need of this feature (I explained three ways to do it!) Please reread my post. I think I articulated my use cases pretty well. And made a reasonable suggestion. You can take it or leave it, but I'd prefer you didn't insult the idea and tell me my its not worth your time. And I asked not on behalf of myself. And I am not even a UD user! I just think it belongs in UD. Your users do not know of your decision to partition disks incompatible with unRAID disks. A number of users have used UD to format disks, and then copied data to them, only to find out they can't and having to redo a lengthy set of operation. They are mystified and blame unRAID because they don't know enough to trace it back to you. And I believe for no good reason except it wasn't worth your time to understand the unRAID partitioning and formatting commands. How would you have a user with a 2011 vintage array with 4 disks showing reallocated sectors, one drive dropped from the array, and priceless (from their perspective) data to recover? Add dual parity? (ROFL) Scenarios like this has happened 2-3 times in the last couple months. My suggestion would help those users who you refer to as edge cases. Later.

When helping people with some tricky array problems, being able to have a get them to prepare an unRAID compatible disk would be useful. This is particularly needed with older arrays demonstrating multiple drive issues. I'd rather not have them do multiple rounds of rebuilds, but instead copy the data to new disks and then form a new array when complete. I also have a relatively frequent need. I will often buy a set of new drives (say 2 8T drives), that I want to use to replace several smaller drives (say 3 4T drives). I want to install the 8T drives outside the array, copy the data to the new drives, run md5 comparisons and ensure all is good, and then do a new config excluding the 4T drives and including the 8T drives with a single parity build. The I/O to the unassigned drives is fast. And the array only has to do one rebuild cycle. And I am free to shuffle the content as desired. Any disk fragmentation is taken care of with the fresh copies (unlike a rebuild that "inherits" the fragmentation). And I am left with the 3 4T drives with data intact that I can use for backups. Doing this in other ways is more complicated and time consuming, and overall more disruptive to my usage. (I do take add'l precautions to monitor smart reports and run a parity check soon before the exchange to protect against failure during the rebuild, and have other techniques to recover even if there is a failure during the build.) Formatting in unRAID format is not THAT difficult without this feature. The cache slot can be used - but if you have a cache drive or cache pool supporting VMs and Dockers it is not so obvious that you can do so without creating a problem for yourself. But I recently did exactly that with my single cache drive that contained a set of Dockers. Replacing it with a real disk that I wanted to format disabled docker functionality. After formatting it, I stopped the array and install the next disk to format, until all disks are formatted, And re-installing the correct cache drive everything was put right. I have not tried with cache pools or with VM feature enables, and would not wholeheartedly recommend this method if you are not confident in your ability to recover if things went wrong. Another way is to use a trial key. This is foolproof but a bit of a PITA. A good way, IMO, is to create a temporary config folder on your real USB key, and boot with that. With the array stopped, rename the config folder (e.g, "config-real") and create a new "config" folder. Copy the primary config files, like ident and network and disk, as well as your USB-specific ".key" file. No "super.dat" file. But have no plugins or other stuff that might confuse things. And no super.dat (although that is not really a problem if you include it). You can boot, create an "array" of the few disks to format, format the disks, stop the array, rename the config folder to "config-format" or something, and rename the "config-real" to "config". When you reboot all is as it was, except the disks are formatted. UD works great with the disks formatted like this. And unRAID is happy with them as well. I believe this would be a nice enhancement to UD to avoid all this rebooting and renaming, and make these techniques easier for users.

Forgive if this has been asked before. But would it be possible to make unassigned devices partition and format disks in an unRAID compatible format? There are a lot of use cases where being able to create a compatible disk would be very handy, and instructions to do so can get complicated (e.g., booting a trial version).

Deserves to be. Thanks!!

@Squid- Add to CA? Thanks!

Sounds like an interesting setup. How does it cool? I have a 92mm fan in each of 4 drive cages, pulling air in over the drives, and several case fans evacuating. My cooling is very good, with parity check temps maxing in upper 30s, and occasionally lower 40s on warmer days. (That's with 20 drives). Your fans should be generating negative pressure in the case, but I'd be a little concerned that air would flow through least resistance and with air currents some of the tighter spots would have lesser flow. Do you have some drives that run hotter than others? Also, not all fans have enough umph to generate enough pressure to move a lot of air against significant resistance. And the ones that do are not the quietest. What is your criteria for exhaust fans? My server lives in an unfinished basement and quiet is not so important. A beefy fan in each cage really works well. But many users need a very quiet server and your method sounds a good way to go!

The SuperMicro 5in3s have 92mm fans.

Seems that the trial and error nature of figuring out what version of what would drive one insane!

Here were the versions of the packages that I had, compared to the versions included in the devpack, compared to the versions I found at http://slackware.oregonstate.edu/slackware64-current/slackware64/. package / myversion / devpack / oregon-state gcc / 4.8.2 / 5.4.0 / 7.1 glibc / 2.24 / 2.17 / 2.25 binutils / 2.23 / 2.27 / 2.28 make / 3.82 / 4.2.1 / 4.2.1 (same) kernel-headers / 3.10.17 / 4.4.38 / 4.9.35 cxxlibs / 6.0.18 / 6.0.18 / (not found) libmpc / 0.8.2 / 1.0.3 / 1.0.3 (same) attr / 2.4.47 / 2.4.47 / 2.4.47 (same) glib2 / na / 2.46.2 / 2.52.3 sqllite / na / 3.12.2 / 3.19.3 I am not sure what happened to cxxlibs, or if it is even needed. It was part of the older package install. The ones I bolded are the ones that are newer in the Oregon State "-current" version. It looks like a fairly major version upgrade to gcc. I only checked the packages above, looks like you have a bunch more in devpack. With that link above you might find newer versions. If you look in the PACAKGES.TXT file, it will provide the info and tell you what subdirectory the package file is located in. Thanks again!!

Awesome! Thank you. Will give it a try.

I'd get the latest (6.4). I will be moving to that soon anyway. And will report back if it is producing an executable that gives errors on 6.3.5, which is what I am running now.

@dmacias @eschultz Would it be possible to add the C compiler and associated libraries to the NerdPack. They the last vestiges of the unmenu package manager I am still using. And they are as nerdy as you can get! These may not all be the latest x64 versions, but this is the set I have ... gcc-4.8.2-x86_64-1.txz glibc-2.17-x86_64-7.txz binutils-2.23.52.0.1-x86_64-2.txz make-3.82-x86_64-4.txz kernel-headers-3.10.17-x86-3.txz cxxlibs-6.0.18-x86_64-1.txz libmpc-0.8.2-x86_64-2.txz attr-2.4.47-x86_64-1.txz Thanks a lot!!

Yes, that is the version that I am running. I see now that the individual package shows updated. I updated from there and it is now working! Thanks!!

Thanks @dmacias! I am looking at the plugin, though, and it does not indicate needing an update. What is required to install the new version?

@eric, @jonp I was trying out the tmux tool, but when I run it with unRAID 6.3.5, I get the error ... "tmux: error while loading shared libraries: libevent-2.0.so.5: cannot open shared object file: No such file or directory" Any suggestions?

Most of the config files deal with setting like spindown, auto start array, array name, IP address, local master, etc. etc. These are pretty easy to redo if you start with the default set of config files, but better to use the real ones. The super.dat file maintains the array configuration (which disks are tied to which slots, whether parity is valid, last parity check and # parity errors, whether array was properly shutdown. The last one can cause an inconvenience - because if your super.dat was from a time that the array was started, and you reuse it, it will report a dirty shutdown and want to do a parity check on boot. Suggestion - when you take a backup of the stick, do so with array stopped!) The other configs you might not want to loose are your docker templates and share configurations. Nothing you can't redo with some time, though, but I'd just as soon keep them. You should also pull the plugins or you'd have to reinstall them. Glad worked for you! Everyone should do these little things once - so you have the knowledge to recover.

Get a new flashdrive and prepare it as per instructions, using the unRaid zip file for your version of unRaid. If you can copy the config directory from the old flash to the new one, that would be best. At a minimum you'd need the .key file that, depending on when you made your purchase, you might have in an old e-mail or in a backup. Be careful using an an old flash backup of your entire config directory. Only use it if you are sure that your array configuration has not changed (I.e., you have not added or removed disks) since the backup was taken. With the old key file in place, when you boot you should be able to request to transfer your license to the new flash drive. If you don't have your key file, you would need to reach out Limetech for assistance.