Dependency on Lime Tech when flash drive fails?

[dev_guy]

The unRAID licensing model is different than most Linux or Unix based server based products as there's no way to backup the operating system with a working license and the license is tied to a flash that drive that will eventually fail from reaching its write cycle or data retention limit. I know you can transfer to a new flash drive up to once per year but that requires Lime Tech still be in business and offering that service.

 

Unless I'm missing something, any unRAID server running 7/24 will eventually permanently fail if Lime Tech is shut down or potentially if they are purchased or otherwise stop offering the ability to get a new license key tied to a new flash drive? I assume one could buy multiple licenses on multiple flash drives but that's still a temporary solution as all flash drives in this application will eventually fail. The inherent problem is the license key is tied to a device destined for failure, and it cannot be backed up, which makes every installation dependent on Lime Tech remaining in business and offering a new license when, not if, the flash drive fails. No offense to anyone, but Lime Tech isn't Microsoft and unRAID doesn't have a very significant market share making it much less certain Lime Tech will be around in say 3 years.

 

Does Lime Tech have any position on this situation? For example some companies have a policy of removing licensing restrictions and/or switching to an open source model if they no longer will support the product for whatever reason. This is a significant issue for more critical and/or elaborate unRAID installations where it would mean a lot of downtime and work to move to a new server operating system.

Edited February 10, 2018 by dev_guy

[JonathanM]

I understand where you are coming from, but your concerns are largely unfounded.

1. Flash drives as license media aren't particularly prone to failure, since unraid doesn't actually run off the drive. Great care has been taken to reduce writes to the flash to an absolute minimum, and reads aren't done often, most read activity is at boot when the OS is read from the USB and loaded into RAM. I'd say average life expectancy of a quality USB key for unraid is over 5 years. I personally have a couple that are older and still running fine. I realize that doesn't help if your key fails, but statistically it's not a big concern.

2. When the flash drive fails, and you no longer wish to run unraid for whatever reason, your data is quite safe. Any decently technical person could spin up a server using totally free software and get access to all the array data in a very short period of time, basically as long as it takes to install and set up practically any linux OS.

3. Limetech has been in this arena for MANY years, and has expanded operations greatly in the last 4 years. During that time they have always been more than fair to their customers, it feels more like a family than a tech company. As a newcomer, it's not possible to see this from the current state, you really just have to experience it over time. Every policy decision that could have been made differently have come down on the side of keeping the majority of customers happy, after discussion on the forum.

4. I can't point to a post, and maybe it wasn't stated publicly on the forum, but I'm pretty sure I remember Tom saying in the past that if the company ceases operation, he would release the licensing and code into the public domain.

Maybe @limetech will comment, but that's how I see things after being around here for 10 years.

[S80_UK]

Agree entirely.  Lime Technology are very supportive of their customers.  And as for USB memory reliabilty - just buy from trusted brands - I chose Lexar based on Limetech's recomendations back then.  I have four USB sticks with unRAID licenses: three are in use, one is for experiments, playing around, etc.  The three that are in use are all over 5 years old, and show no signs of problems.

Edited February 10, 2018 by S80_UK

[dev_guy]

@jonathanm and @S80_UKI appreciate your replies and thoughts. It's good to know at least a few have gone many years on a given flash drive. I don't really know what unRAID is doing with the flash drive and I've only seen vague descriptions from others. More than 99% of flash drives, including the ones recommended by Lime Tech, use MLC or TLC flash and lack any manufacture specifications for write cycles or data retention. Generally data stored on MLC and TLC flash is considered unreliable after 1 year even with the drive sitting in a drawer unused due to the charge gradually bleeding off that stores the data. In practice data retention is typically better but it's still a valid concern. If unRAID, or the drive itself, periodically re-writes the data then retention isn't an issue. But I don't know if that's the case?

 

Even if data retention isn't an issue the number of write cycles may still be. The flash used in most USB drives is rated for around 3000 - 5000 erase/write cycles per block. Even changing one byte in a block requires the entire block to be erased and re-written. That works out to about 10 writes a day to the same block for 1 year. While SSDs typically use higher quality flash, sophisticated wear leveling, and excess spare (hidden) capacity, to largely mitigate this limitation USB drives generally do not. SanDisk and Lexar say nothing about any of these things (or data retention) for their USB flash drives. Does anyone have factual information how often the flash drive is written to? I know configuration data is stored there.

 

To ask a related question, I know the initial key installation requires an internet connection and I assume there's a handshake with the Lime Tech mother ship when that happens. If I purchase a second license for a spare USB drive, what happens if I plug it in to replace a failed drive? Will it need to handshake with Lime Tech as the new key won't match what might be stored elsehwere? If so, even having a spare licensed flash drive isn't a solution if Lime Tech is no longer around.

 

It would be great if @limetech can weigh in on these issues? It also would be interesting to know how many unRAID users have had a flash drive fail?

Edited February 10, 2018 by dev_guy

[S80_UK]

unRAID only connects to LImetech for a trial license.  An unRAID installation using USB with an purchased .key file does not require to have an external internet connection in order to fully function.  So once you have a key set up, it will continue to work regardless.

 

Wriiting to USB is not frequent.  The software installs to RAM and runs from there.  There are files on the USB that describe the server configuration, disks, shares, etc, and they get updated when you make changes.  Small changes are made when you start and stop the server, so that it knows whether the filing systems were shut down cleanly or not.  Other writes are typically for plugins that you may install and they may have their own write behavior, athough some will primarily be writing to shares or perhaps to a cache drive.  

 

I take your points regarding data retention and SLC vs MLC, TLC, etc.  In my experience, (40 years with EPROMS, 20+ years using flash) leakage issues are rare, and are mostly a high temperate / high usage issue.  Yes, there are wearout mechanisms, but the frequency of writes in a typical unRAID sytem is really quite low.  I don't have hard data, but a significant number of the failures reported on the forums are mechanical in origin - sticks being physically broken.  Hint, use a very short stick, or if available, an internal USB socket. 

 

Lastly, since the disks in the server are directly readable by pretty much any modern Linux system as @jonathanm pointed out, I see this as a non issue.  So long as I have my data in 5 years or so, that is too far away for me to worry about whether my sever is going to use unRAID or something else.  Having said that, if unRAID still works, the chances are high that I will still be using it.

 

 

[dev_guy]

 

@S80_UK I'm not worried about the raw data as much as the entire configuration, VMs, docker containers, users, security, plug-ins, etc. Those things can be just as critical as the data and typically take considerable time to get set up, configured, working properly, etc. Much of that goes out the window if you lose the ability to run unRAID because your flash drive dies and Lime Tech has been bought up and discarded by Google (as Google has done with countless other companies with loyal followings). You would have to start over on a new OS. If a user just needs a file server there are arguably better overall choices than unRAID. A big reason to choose unRAID is all the other functionality.

 

It is good to know a spare drive with a spare license might be good insurance if it will work without checking in with Lime Tech.

Edited February 10, 2018 by dev_guy

[JorgeB]

I have 10 unRAID licenses, some of which are more than 10 years old, never had a flash fail (though full disclosure had 2 replaced a long time ago because I used some lesser brand flash drives and they kept getting corrupted after each power cycle), just get good quality flash drives and don't worry.

[pwm]

2 hours ago, dev_guy said:

Generally data stored on MLC and TLC flash is considered unreliable after 1 year even with the drive sitting in a drawer unused due to the charge gradually bleeding off that stores the data.

This isn't correct, unless your drawer is 80°C warm - every 10°C halves the retention time but starting from a quite significant number of years at room temperature.

 

The majority of products you buy and have around you have their firmware stored on MLC flash. Your DVD player, amplifier, mobile phone, ... aren't considered unreliable after 1 year. Temperature matters a lot - but your flash drive will not have to suffer the same temperatures as some devices in vehicles.

Said by an engineer who develop hardware with flash-based firmware and file systems for use in infrastructure and with 10+ years expected lifetime in the field. And not just having 10+ years of expected lifetime - lots of the equipment have already been out for 10+ years with the software in MLC flash. I obviously have to spend time with the actual flash memory datasheets when selecting what components are selected for the next generation hardware since I can't rely on personal guesses. For normal use, it isn't retention but number of erase cycles that counts, and how well the flash controller spreads the wear.

 

Anyway - best is to buy small and old flash drives of known brands to reduce the probability of TLC flash.

[dev_guy]

@pwm In fairness I did say flash drives typically do better. I'm also an engineer who specializes in embedded systems so I'm very familiar with flash. And it's worth pointing out if the flash drive is located inside an enclosure with 20+ TB of hard drives and quiet cooling for a server beefy enough to handle some real Docker/VM workload, it's going to get fairly warm in there. It will be running 7/24/365 at elevated temperatures which isn't true of most consumer electronics or thumb drives. The recommended Sandisk Cruzer Fit, for example, is in no way designed for that sort of application.

 

Regardless of the cause of a flash drive failure the issue still remains. You might be just in the unlucky 1% or whatever. Hard drives are also relatively reliable for at least a few years but not having a good backup strategy for those is considered foolish. Yet most of the advice in this thread so far is "don't worry about it" despite the fact it's a similar risk and will take the entire server down. Granted your data should still be safe but the server, and everything it used to do, will be dead in the water because of Lime Tech's licensing/anti-piracy scheme.

 

If it's indeed correct you can have a licensed spare flash drive that will work without needing anything from Lime Tech that helps a lot but does require buying two licenses. I'm not sure what happens if you wait a year and license a new flash drive as a spare with the original drive still working? Will they both work? Or is the original key disabled somehow? That's one reason I suspected unRAID has to check in with Lime Tech to authorize a new key.

 

As I said before, hopefully @limetech can weigh in on this issue as they're in the position to provide the best answer.

[Hoopster]

This is one reason why I use a Kingston MobileLite SD card reader as my unRAID flash drive.  If the SD card dies, I just copy the backup to a new card.  Since the license is tied to the GUID of the card reader, as long as it does not die, I can replace SD cards "forever" without needing to license another device.

 

Card readers are not "officially" supported by Limetech as there is no guarantee they will work, but, as many of us have discovered, the MobileLite G2 and G3 work well since each one has a unique GUID.  The G4 does not have a unique GUID and cannot be licensed.

Edited February 11, 2018 by Hoopster

[SSD]

11 hours ago, jonathanm said:

 

4. I can't point to a post, and maybe it wasn't stated publicly on the forum, but I'm pretty sure I remember Tom saying in the past that if the company ceases operation, he would release the licensing and code into the public domain.

Maybe @limetech will comment, but that's how I see things after being around here for 10 years.

 

Tom has said that he has taken steps to ensure that users will be able to swap their keys should he go out of business or get hit by the proverbial bus.

 

But keeping it up to date, especially with the security issues that seem to be an ongoing challenge, would require someone at the helm. Let's all hope that Tom stays healthy, and unRAID continues to be a profitable venture!

[remotevisitor]

9 hours ago, dev_guy said:

If it's indeed correct you can have a licensed spare flash drive that will work without needing anything from Lime Tech that helps a lot but does require buying two licenses. I'm not sure what happens if you wait a year and license a new flash drive as a spare with the original drive still working? Will they both work? Or is the original key disabled somehow? That's one reason I suspected unRAID has to check in with Lime Tech to authorize a new key.

 

When you move the license to a new flash drive, the old flash drive GUID is added to the blacklisted drives which will be included in the next Unraid update.   This therefore disables the old drive in subsequent releases of Unraid.

[dev_guy]

@remotevisitor Thanks for that bit of info. Being blacklisted wouldn't be an issue if Lime-Tech was no longer supporting the product. I also agree with @SSD's point that we all would rather be using a supported product, be able to get at least security updates, etc. Still, my main concern is not putting a bunch of eggs in one basket and having it all broken in a way I can't self-recover from by just a flash drive failure.

 

@Hoopster's suggestion is an interesting one. I didn't realize there were any USB SD readers with unique GUIDs. That eliminates the concern about the flash memory failing but the reader itself could still fail leaving you in the same situation. But that's less likely than a flash failure.

 

I have found a couple sources of SLC (single level) industrial USB flash drives rated for 10 years data retention and 100,000 write/erase cycles (vs about 5000 for typical consumer flash drives) over their full range of operating temperature. They're designed for even military applications. They're relatively expensive at about $45+ for a 1GB drive but that might be cheap insurance if I don't go with the Kingston Mobile Lite SD reader.

 

I appreciate everyone's input on this. Hopefully @limetech will also comment this week.

[SSD]

unRAID reads the boot and configuration files once per boot. And there is a tiny file on the drive that gets updated with each starting and stopping of the array. There are plugin updates that occur from time to time.

 

This is extremely light usage.

 

Everyone that kicks the unRAID tires is concerned about this risk. In the early days unRAID came with 2 keys - one primary and one for a backup. But with the quick and easy keyswap solution, these dual keys are not longer offered. But the best backup is a second key IMO.

 

But I have to say that failing USB sticks are exceedingly rare. It is more likely that one would have physical damage, and therefore you might look at a stick that is very short and not subject to being snapped off.

 

https://www.amazon.com/gp/product/B005FYNSZA

[dev_guy]

@SSD in my case it's a Supermicro server motherboard with an internal USB jack on it. I plan to have the flash drive inside the rack enclosure so it won't be vulnerable to damage, loss, etc. As for thumb drives failing I've had a few. My car has a USB port for the audio system. I had a tiny HP 16GB thumb drive become corrupted after about 2 years of being left in the car. Removing it and testing it revealed extensive read errors. I've also had a tiny USB 3.0 Sandisk Ultra Fit fail. That particular drive is a really stupid design as it runs extremely hot with no surface area to dissipate heat. If you look at the reviews on Amazon they frequently fail in under a year despite being from Sandisk. Lime-Tech should probably make it clear to only use the otherwise identical looking USB 2.0 Cruzer Fit which runs much cooler.

[uldise]

3 hours ago, dev_guy said:

I plan to have the flash drive inside the rack enclosure so it won't be vulnerable to damage, loss, etc.

I had the same config some time ago - when i rearranged PCIe Cards and placed one very close to internal USB port, USB Stick starts drop out and then completely dies. i replaced it for new one and noticed the same drop outs.. then i put it outside a server on usb hub, and all is working since.

[dev_guy]

I'm giving this a bump in the hope @limetechmight weigh in on this issue?

[JonathanM]

6 minutes ago, dev_guy said:

I'm giving this a bump in the hope @limetechmight weigh in on this issue?

Which specific question(s) do you want him to answer?

Perhaps restating the question(s) more succinctly now that you've seen the discussion would be helpful.

[dev_guy]

@jonathanm and @limetech ideally I'd like to know the following. Some of it has been discussed but is still speculation rather than a definitive answer:

 

1 - Can a flash drive be replaced with a previously licensed spare without needing a connection to Lime-Tech or for Lime-Tech to even be in business?

 

2 - Does Lime-Tech have an official policy on licensing if they no longer will be providing license activation services?

 

3 - How often is the flash drive read from and written to? Some have said, at the least, when the system is booted, when the array is stopped/started, and when the configuration changes. Those all make sense. Are there are other times it's accessed? If I pluck the flash drive out of a running system what happens assuming it's just left running with no changes? I don't really want to try that at the risk of corrupting the authorized flash drive but what happens when a flash drive dies while unRaid is running? @uldisemay also know the answer to this one as he's had multiple flash drive failures.

[JorgeB]

2 minutes ago, dev_guy said:

1 - Can a flash drive be replaced with a previously licensed spare without needing a connection to Lime-Tech or for Lime-Tech to even be in business?

That one was already answered, yes it can.

[Squid]

7 minutes ago, dev_guy said:

How often is the flash drive read from and written to

Written to: only when a setting is changed

 

Read From: Pretty much only at boot up time and when navigating to different pages in the UI.  "Technically", various files are read pretty much continuously during operation, but the way Linux works is that those files wind up being cached in RAM, and the actual read comes from RAM instead of from the flash

 

10 minutes ago, dev_guy said:

If I pluck the flash drive out of a running system what happens assuming it's just left running with no changes?

The operation of the server itself remains complete, but the GUI will issue a ton of various errors, and you'll see in the syslog "Bread" errors.

[uldise]

2 minutes ago, dev_guy said:

@uldisemay also know the answer to this one as he's had multiple flash drive failures.

i had only one failed flash drive - and this was because of heating issues inside a server - flash drive was on port on the MB. I replaced flash with online procedure and then placed it outside a server on a small USB hub.

[dev_guy]

Thanks to all for the added info. @johnnie.black has someone actually used a spare licensed flash drive with the current version of unRaid and is certain unRaid didn't silently check in with the mothership? This one still seems more like speculation based on the replies so far and best answered by @limetech

[JorgeB]

Thanks to all for the added info. [mention=353]johnnie.black[/mention] has someone actually used a spare licensed flash drive with the current version of unRaid and is certain unRaid didn't silently check in with the mothership?

Yes, and LT already confirmed that only the trial license (and a few betas releases in the past) communicate with their servers.

[JonathanM]

51 minutes ago, dev_guy said:

is certain unRaid didn't silently check in with the mothership?

If that ever happened, trust me, there would be discussion about it. There are plenty of people here with locked down networks that inspect traffic. The latest addition of a custom domain for internal ssl use does use external communication, but that has been pretty thoroughly discussed, has nothing to do with licensing, and is totally optional.

 

Tom is more of a fellow tech hobbyist that decided to share his custom solution than a corporate IT type. Sometimes that hurts certain aspects, like hard release schedules, and beta / rc naming conventions, but it certainly helps with security and trust. When Tom says something, believe it. He has a long track record of working with and for the community, which is one of the main reasons his community is so strong. Limetech for many things still acts and reacts like a one man shop, for better or worse.