USB Flash Drive.. a bit of a rant....

[diehardbattery]

I'm not sure what my problem is.  Over the last 6 months I've had no less than 4 USB sticks of varying quality (ones that normally spend their life in a desk drawer, to recommended ones from other users, i.e. Samsung Bar) go corrupt requiring to shut down, pull the stick, insert it in my Windows PC, fix the errors and put back.  It runs for a while, then rinse, repeat.  It's mildly annoying but ok, whatever.  What's more annoying is that this happened at some point last night, during a data rebuild.  It's a 22TB drive that took 39 hours to do.  When I checked this morning it said it was finished but had the dreaded flash drive corrupted message.  I couldn't stop the array from the gui and thought it probably couldnt stop from the command line either because of the corruption.

 

So I did a clean shutdown and fixed the drive and rebooted.  The array was in a stopped state and warning that the drive that I just rebuilt was gonna be overwritten once I started the array.  Maybe I should have just stopped there and researched if there is a way around that.  But I figured the data might have been corrupted somehow anyway due to the flash drive issue, and started the array.  It is now rebuilding the same drive a second time.  39 hours wasted.  I'm not 100% discounting a hardware issue with my NAS (QNAP TS-873 that I've had now for 3 yrs), but the USB slots have only been used for Unraid and nothing else.  I guess I could try another port.  It's just frustrating because I have an 8 drive array and am replacing the drives one by one (It is the safest, correct?  Unless there's a faster way?)...

[Frank1940]

Make sure it is in a USB2 port.  (These generally have a black plastic insert.  USB3 usually have a blue plastic insert. If in doubt, RTFM.)  Unraid flash drives are often more stable in USB2 ports than USB3 ports.

 

Make sure you are not using the drive for any type of temporary storage.  Look at the count of read/write operations on the flash drive.  This is what mine looks like after being up four days:

 

'Writes" are normally the critical item you should be concerned about.  A drive that is corrupted by 'reads' is just plain defective.  Writes are what 'destroys' flash drives. 

Edited July 24, 2023 by Frank1940

[diehardbattery]

11 minutes ago, Frank1940 said:

Make sure it is in a USB2 port.  (These generally have a black plastic insert.  USB3 usually have a blue plastic insert. If in doubt, RTFM.)  Unraid flash drives are often more stable in USB2 ports than USB3 ports.

 

Make sure you are not using the drive for any type of temporary storage.  Look at the count of read/write operations on the flash drive.  This is what mine looks like after being up four days:

 

'Writes" are normally the critical item you should be concerned about.  A drive that is corrupted by 'reads' is just plain defective.  Writes are what 'destroys' flash drives. 

Once the rebuild is finished I'll double check, but I'm reasonably sure it's in a 2.0 port.  My writes look similar to yours though...

[Frank1940]

But your system has been up only a few hours.  Check those counts after it has been running normally a few days.  You could have a Docker app (or even a VM) using it a 'scratch pad'.

[Kaldek]

I burned through a few flash drives before using USB2 ports only.  My current unit has lasted 3 years now.  However, I am switching to a USB DOM (Disk On Module) shortly for some extra reliability. They are more expensive but use quality SLC flash.

 

The only downside is that it's mounted to a motherboard header, and harder to get to.  But, it should be unlikely I ever need to touch it.

[diehardbattery]

Just now, Kaldek said:

I burned through a few flash drives before using USB2 ports only.  My current unit has lasted 3 years now.  However, I am switching to a USB DOM (Disk On Module) shortly for some extra reliability. They are more expensive but use quality SLC flash.

 

The only downside is that it's mounted to a motherboard header, and harder to get to.  But, it should be unlikely I ever need to touch it.

My NAS already has a DOM with the QNAP OS on it.. wonder if it could be repurposed for Unraid?

[diehardbattery]

8 minutes ago, Frank1940 said:

But your system has been up only a few hours.  Check those counts after it has been running normally a few days.  You could have a Docker app (or even a VM) using it a 'scratch pad'.

Gotcha. I checked the manual and it seems that all the ports on my nas are USB 3.0... The BIOS is not updateable but its never been a problem... maybe I need to find a USB 2.0 expansion card...

[Gragorg]

I had a similar issue with my flash going corrupt and having to repair it in a windows pc. I formatted my flash (not quick format) and restored from my backup and my usb has been good for years. 

Edited July 24, 2023 by Gragorg

[Frank1940]

26 minutes ago, diehardbattery said:

Gotcha. I checked the manual and it seems that all the ports on my nas are USB 3.0... The BIOS is not updateable but its never been a problem... maybe I need to find a USB 2.0 expansion card...

 

Double check the manual.  Sometimes there are extra/spare USB headers on the motherboard and one (or more) of them could be USB2.  (I recall reading a post in another thread which reported this to be the case.)  

[diehardbattery]

6 hours ago, Frank1940 said:

 

Double check the manual.  Sometimes there are extra/spare USB headers on the motherboard and one (or more) of them could be USB2.  (I recall reading a post in another thread which reported this to be the case.)  

So far no luck on the headers.. and not looking too hot for something that is USB 2.0 specific... guess I'll try a different port...

[Frank1940]

It is not that fact that the flash drive is USB2 or USB3.  It is the fact that (for some unexplained reason) Unraid, Linux, USB3 ports and certain motherboards have an interaction that results instability of the entire flash drive connection.  The only common factor is the USB3 port.  When the problem occurs, using a USB2 port will fix the problem.

 

The reason that USB2 flash drives are recommended is that they generally run cooler.  (Lower speed = less power required.  It only takes my USB2 based system about 10-15 seconds to transfer the entire Unraid boot files to RAM when it boots!)  Smaller capacity drives were often another recommendation because the cells were larger.  Today, I suspect that all flash drives use small cells.  At one time, larger cells were cheaper to manufacture.  Today, there is little-to-no-difference in cost.  So they have probably standardized on the smaller cell.  I know if I were running the production line, I would...

[diehardbattery]

1 hour ago, Frank1940 said:

It is not that fact that the flash drive is USB2 or USB3.  It is the fact that (for some unexplained reason) Unraid, Linux, USB3 ports and certain motherboards have an interaction that results instability of the entire flash drive connection.  The only common factor is the USB3 port.  When the problem occurs, using a USB2 port will fix the problem.

 

The reason that USB2 flash drives are recommended is that they generally run cooler.  (Lower speed = less power required.  It only takes my USB2 based system about 10-15 seconds to transfer the entire Unraid boot files to RAM when it boots!)  Smaller capacity drives were often another recommendation because the cells were larger.  Today, I suspect that all flash drives use small cells.  At one time, larger cells were cheaper to manufacture.  Today, there is little-to-no-difference in cost.  So they have probably standardized on the smaller cell.  I know if I were running the production line, I would...

The flash drive I use now I think is several yrs old.. only 8GB... I cant imagine it's USB 3.0  I'll try a different port next time.. thats the only thing I have not tried... it had been several months since the last time before this so who knows.. I wouldnt have cared as much if I didnt have to rebuild the drive again...

[tjb_altf4]

3 hours ago, Frank1940 said:

It is not that fact that the flash drive is USB2 or USB3.  It is the fact that (for some unexplained reason) Unraid, Linux, USB3 ports and certain motherboards have an interaction that results instability of the entire flash drive connection.  The only common factor is the USB3 port.  When the problem occurs, using a USB2 port will fix the problem.

I would note many modern motherboards don't come with usb2.0 ports anymore, but sometimes they do allow those ports to be put into usb2.0 mode in bios settings.

[diehardbattery]

7 hours ago, Frank1940 said:

It is not that fact that the flash drive is USB2 or USB3.  It is the fact that (for some unexplained reason) Unraid, Linux, USB3 ports and certain motherboards have an interaction that results instability of the entire flash drive connection.  The only common factor is the USB3 port.  When the problem occurs, using a USB2 port will fix the problem.

 

The reason that USB2 flash drives are recommended is that they generally run cooler.  (Lower speed = less power required.  It only takes my USB2 based system about 10-15 seconds to transfer the entire Unraid boot files to RAM when it boots!)  Smaller capacity drives were often another recommendation because the cells were larger.  Today, I suspect that all flash drives use small cells.  At one time, larger cells were cheaper to manufacture.  Today, there is little-to-no-difference in cost.  So they have probably standardized on the smaller cell.  I know if I were running the production line, I would...

Well, happened to be up late, checked the progress and lo and behold it corrupted again.. no additional writes...

 

 

[diehardbattery]

4 hours ago, tjb_altf4 said:

I would note many modern motherboards don't come with usb2.0 ports anymore, but sometimes they do allow those ports to be put into usb2.0 mode in bios settings.

I thought that too.. but it wasnt until after I started the second rebuild... it just corrupted again so now is good time as any lol

 

UPDATE:  No option to change in the BIOS...

Edited July 25, 2023 by diehardbattery

[Kaldek]

19 hours ago, diehardbattery said:

My NAS already has a DOM with the QNAP OS on it.. wonder if it could be repurposed for Unraid?

Likely.  Also I'm a bit surprised your motherboard has *no* USB2 headers.  They're common even on new stuff, even it's only a single two-port header.

[diehardbattery]

3 hours ago, Kaldek said:

Likely.  Also I'm a bit surprised your motherboard has *no* USB2 headers.  They're common even on new stuff, even it's only a single two-port header.

If I corrupt again or the rebuild finishes I'll crack open the case to take a look.  But researching google and asking on the qnap forums didn't turn up anything on it.  Or in the unlikely event that someone else that has the same NAS sees this and volunteers LOL.

Edited July 25, 2023 by diehardbattery

[Kaldek]

10 hours ago, diehardbattery said:

If I corrupt again or the rebuild finishes I'll crack open the case to take a look.  But researching google and asking on the qnap forums didn't turn up anything on it.  Or in the unlikely event that someone else that has the same NAS sees this and volunteers LOL.

Ah, I seem to have missed the part where you wrote you're running unRAID on a QNAS box.

[diehardbattery]

2 hours ago, Kaldek said:

Ah, I seem to have missed the part where you wrote you're running unRAID on a QNAS box.

Yeah, and I just corrupted again.. time for a new usb stick me thinks...  On that note, The current USB is still usable, but I can't determine anything else thats causing this definitively except the USB itself.  I'm gonna need to contact limetech to have my key reset when I change it.  I've done it before but I cant remember if I was still able to use the NAS normally with the new usb until it's reset... if not is there a way I can get the GUID beforehand or will I be stuck until limetech resets?

Edited July 26, 2023 by diehardbattery

[Kaldek]

I will admit that my current flash drive - a SanDisk Cruzer Fit 32GB has lasted me a very long time without issue, when connected to a USB2 port. 

[diehardbattery]

9 hours ago, Kaldek said:

I will admit that my current flash drive - a SanDisk Cruzer Fit 32GB has lasted me a very long time without issue, when connected to a USB2 port. 

Therein lies the difference... you have a 2.0 port to utilize... evidently I either have to cross my fingers that the 3.0 ports do their backward compatibility reliably, or I just have really bad luck with USB flash drives.  I have a Cruzer Glide 16GB I'm using now... I mean I know Unraid is somehat known for running well on older hardware but eventually there will come a point where 2.0 ports won't exist period.  What then?

Edited July 27, 2023 by diehardbattery

[Frank1940]

16 minutes ago, diehardbattery said:

I mean I know Unraid is somehat known for running well on older hardware but eventually there will come a point where 2.0 ports won't exist period.  What then?

 

The issue is that some folks don't have a problem with USB3 ports!   It seems to be combination of things---  the bootup software used by LimeTech (contained in  /EFI/boot   folder on the boot drive), the Motherboard BIOS, and Linux USB drivers.  There could even be the USB interface chip set on the MB is somehow involved.   Or maybe two or three totally different combinations of these parameters is a possibility.  It is a complex system issue to solve. 

 

Plus, I think all of us have had issues with USB devices going awry and we just fix it by unplugging and replugging the device.  Unfortunately, when the boot drive goes offline, it creates major problems!

[diehardbattery]

26 minutes ago, Frank1940 said:

 

The issue is that some folks don't have a problem with USB3 ports!   It seems to be combination of things---  the bootup software used by LimeTech (contained in  /EFI/boot   folder on the boot drive), the Motherboard BIOS, and Linux USB drivers.  There could even be the USB interface chip set on the MB is somehow involved.   Or maybe two or three totally different combinations of these parameters is a possibility.  It is a complex system issue to solve. 

 

Plus, I think all of us have had issues with USB devices going awry and we just fix it by unplugging and replugging the device.  Unfortunately, when the boot drive goes offline, it creates major problems!

See, I really wouldn't have minded nearly as much if that's all I had to do.. but a 22TB rebuild takes ~40 hrs and having to start that from scratch multiple times now is starting to test my patience a little lol.  Then whenever I want to try a different stick I have to wait for limetech to reset my key because you can only do that once a year, so my one year is now reset.  I like the versatility that Unraid offers when mixing and matching drives so I really don't want to look elsewhere... The other issue being that, while you are 100% correct in that it's complex problem, we're paying for the license.  I'm not saying just because its paid its not going to be trouble free but it should at least be better than if it was completely free.  Just my 2 cents.

Edited July 27, 2023 by diehardbattery

[JonathanM]

5 hours ago, diehardbattery said:

I'm not saying just because its paid its not going to be trouble free but it should at least be better than if it was completely free. 

Definitely agree, but it's hard to fix the issue when you can't consistently recreate it. Thousands of Unraid setups go many years without any USB boot stick errors.

[foo_fighter]

I've had this same issue and it turned out to be bad memory. Running memtest found it and after replacing the RAM, the corrupted USB drive went away as well as other issues.