Lost 2 USB drives in 6 months and I don't know why

[ZenBuddha]

Back in November, I went to do an update to my system, rebooted and the USB drive would not boot.  No big deal, I download my backup and I'm up and running again.  In February I do the same thing and the same thing happens.  I didn't have time to troubleshoot it and just moved on.  I went to install the latest version of Unraid today, and I lost ANOTHER USB drive.  Now I can't replace my key because I've apparently done it too many times.  I've reached out to support to get a replacement key, however there is clearly something wrong.  The only thing I can read on the screen during the failed boots is:

Contains a free cluster (2).  Assuming EOF.

FAT-fs (sda1): error, fat_get_cluster: invalid cluster chain (i_pos 1)

FAT-fs (sda1): Filesystem has been set read-only

mount: /boot: can't read superblock on /dev/sda1.

           dmesg(1) may have more information after failed mount system call.

cannot mount /dev/sda1 - press ENTER key to reboot ....

 

Attaching diagnostics from the Feb 11 restore point.

tower-diagnostics-20240313-0850.zip

[itimpi]

Have you tried plugging the flash drive into a PC/Mac to see if it can still read it?     If so then rewriting it sometimes helps

[JorgeB]

So many flash drives failing can be a problem with the USB ports or flash drives you are using, unfortunately it's getting harder and harder to find good quality flash drives, personally I like the Sandisk Cruzer Blade 32GB, one of the few USB 2.0 drives remaining, which in my experience are more reliable.

[ZenBuddha]

3 minutes ago, itimpi said:

Have you tried plugging the flash drive into a PC/Mac to see if it can still read it?     If so then rewriting it sometimes helps

What does "rewriting it" mean? Just do a restore to the existing USB drive?

[itimpi]

3 minutes ago, ZenBuddha said:

What does "rewriting it" mean? Just do a restore to the existing USB drive?

In effect yes, although I would reformat it first if you try this.   Note that would mean that you would need to run the make_bootable.bat (in admin mode) afterwards if you want to boot in legacy mode.

[ZenBuddha]

I was able to format the original usb drive and restore to it. I did need to download my key and put it in the config folder, but my system is back up!  Thank you!!!

 

Now...can you also help me figure out why this has happend with 2 different USB devices in a 6 month period?  😬

[itimpi]

Just now, ZenBuddha said:

Now...can you also help me figure out why this has happend with 2 different USB devices in a 6 month period?

Not able to help with that I am afraid.   Seems to happen reasonably frequently with some people while others see their flash drives lasting forever.   I suspect it is something to do with the USB ports on the motherboards in some way,  As always we find that USB2 drives and/or USB2 ports seem to be more reliable.

[ZenBuddha]

My system froze up again.  I was able to get in via console and force a shutdown.  I'm attaching diagnostics in case anyone can help me figure this out.  I had also turned on the syslog server and told it to copy to the flash on shutdown but I'm not clear on how to retrieve them.  Would appreciate any guidance.

I have seen in the forums that SanDisk USB drives are not recommended and that is what I'm using. I'll work on getting a different one but it doesn't seem like there are a lot of viable 2.0 USB drives available any more.  I see a HP one, but not sure I trust that any more than a SanDisk.  Would a non 2.0 Samsung be a better solution?

Again, appreciate any help.  I do have a small feeling that this tends to happen after I use the binhex-krusader docker but that is just a small correlation I have noticed. 

Thanks!!!

tower-diagnostics-20240514-1133.zip

[itimpi]

3 hours ago, ZenBuddha said:

have seen in the forums that SanDisk USB drives are not recommended and that is what I'm using

This is not actually true.    What there is is a warning that SanDisk flash drives are a brand that are quite frequently counterfeited so you need to make sure you get them from a reputable supplier.   Genuine SanDisk flash drives are fine - it is the counterfeits that are a likely to not work well.

[ZenBuddha]

Ah.  Thank you for clarifying.  I saw this quote in the Unraid manual and it didn't mention anything about reputable sellers:
" Due to this, at this time, we cannot officially recommend SanDisk USBs due to the issue of generic GUIDs found in knock-off / counterfeit units."

 

Any guidance on how to access the syslog servers on the flash drive?  And anything else I can find in the diagnostics to figure out why my USB drives keep failing?

[itimpi]

5 hours ago, ZenBuddha said:

I saw this quote in the Unraid manual and it didn't mention anything about reputable sellers:

That quote was basically saying beware of counterfeits and that SanDisk is a brand is quite commonly counterfeited.  If the price seems suspiciously good there is a good chance it is counterfeit.  If a reputable seller supplies a counterfeit drive one assumes they would honour a RMA on it. 

[Frank1940]

7 hours ago, ZenBuddha said:

Any guidance on how to access the syslog servers on the flash drive?  And anything else I can find in the diagnostics to figure out why my USB drives keep failing?

 

First, have a look at the WRITES to your USB drive.  You can find them here:

 

USB drives are not design for large numbers of write cycles.  The write count for this drive is over a 33 day period.  If your boot drives has substantial numbers of writes per day (say more than 200 per day) , figure out what is causing them.  You can post back here if you can't figure it out. 

 

High temperatures are another failure accelerator.   For that reason, many Unraid users do not use 'Mini/Micro' drives.  They feel that the larger drives will have better heat dissipation properties.

 

EDIT:  Some users looked for USB2 drives as they tend to run cooler than USB3 drives.  (The extra speed that USB3 will provide will only save a couple of seconds during the boot process!)

Edited May 15 by Frank1940

[ZenBuddha]

Thank you Frank.  Appreciate the help.

This is what my boot devices says right now:

My current up time is 24 hours.  Clearly more than the 200 per day.

I don't know how to figure out what is writing to it so yes, if you can give me some guidance there I would be very grateful.

[Frank1940]

2 hours ago, ZenBuddha said:

I don't know how to figure out what is writing to it so yes, if you can give me some guidance there I would be very grateful.

15 hours ago, ZenBuddha said:

Any guidance on how to access the syslog servers on the flash drive?

How (and why) are syslogs getting written to your Flash Drive?  Now, I realize, that they are sometimes written to the flash drive on shutdown and this is an acceptable practice.   But setting up the Syslog Server (or some other process that does the same thing) just to keep a permanent record of it is not an acceptable practice.  Are you doing this? 

 

Updating Plugins will write to the flash drive.  This is normal.

 

However, sometimes Dockers containers are mapped to use the flash drive.  This should be avoided if possible by using the cache drive.

 

Twenty four hours following a reboot may not be completely typical of normal operations.  Keep a eye on it for the next several days

[ZenBuddha]

Thanks again for your help.  I will do as you suggest and keep an eye on it.  It's at 4069 right now so maybe it was just the boot reads.

In regards to the syslog server question I was referring to recommendation in the Troubleshooting section of the manual that says you can turn on the syslog server to write to the flash on shutdown:

And I was wondering since I recently had a forced shutdown because I couldn't get to the GUI, if there might be some data I could look at.

 

[Frank1940]

4 minutes ago, ZenBuddha said:

since I recently had a forced shutdown because I couldn't get to the GUI, if there might be some data I could look at.

 

I do believe that this option will only work if the shutdown is 'normal'.   (That was the whole reason for the syslog server plugin!  To try to catch some thing that might indicate the cause of server lockup...) 

 

What happens when you check that option is that the syslog is copied from its location in memory to the flash drive as a part of the normal shutdown procedure.   (I have been doing that for some time now and I am wondering why I am doing it.  My server has bee stable for years and therefore there will nothing of interest to see.)