Odd Array & Web Interface issues

[Matt.tr]

Shortly after upgrading to 6.8.0 (upgrade went fine from the web gui & reboot to my knowledge) from 6.7.2, I noticed that I was having issues trying to playing all media files on my cache drive. The file on a remote windows machine playing the file using VLC, would only play the first 3 seconds and restart over at the beginning in this loop. Then checking files/folders on the my array, I noticed there were missing files and folders, at that moment I knew something wasn't right.

 

Looking at the syslog, I noticed these errors below and thought my USB drive either had an issue in the port it was using or it was starting to die. Since I had no backup of the flash drive, I quickly backed it up. I shutdown the server, tried different USB ports (back 3.0/back 3.1 & USB 2.0 off of the motherboard) but all seem to still give me these errors in the log and array issues.

 

Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30552) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30553) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30554) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30555) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30556) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30557) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30558) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30559) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30560) failed
Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30561) failed
Dec 14 16:29:01 UNRAID2 kernel: fat__get_entry: 6 callbacks suppressed

 

I then proceeded to go through the "Replacing your Unraid USB Flash Device and Registration Key" process with that newly created backup zip file, to a new sandisk flash drive. However this hasn't resolved anything for me. I'm still having issues starting the array normally, I can start it in maintenance mode only but have issues stopping the array.

 

I do have a couple of drives with UDMA_CRC_Error_Count messages that appeared at the beginning of this issue but the #'s haven't increased over the past few days. SMART logs for these drives attached. Re-seated sas card/cables all other sata/power cables going to the drives. I did do a XFS_repair -n on these array disks and it didn't find anything. I was finally able to complete one parity check after this issue started, no issues reported when it finished.

 

I also have this odd web interface issue where after a bit (usually after attempting to start the array), I lose my custom banner (thanks spaceinvader!) and it reverts back to default. Sometime, I even lose everything underneath the top tool bar which forces me to go the terminal for logs & shutdowns.

 

Unraid Server Specs

Intel Xeon E5-2660 V3

Asus X99-E WS/USB 3.1

Hynix 4x8GB DDR4 ECC Registered

LSI 9400-8i P10 firmware in IT mode

Array:7xToshiba 5TB X300, 2xToshiba 6TB X300

Parity drives: 2xToshiba 6TB X300 (connected to motherboard sata ports)

Cache drive: Micron_1100_MTFDDAK2T0TBN (connected to motherboard sata port)

 

I'm a bit lost on how to proceed on resolving these issues. I'm not super tech savvy.

 

Please let me know if I need to grab any further information. thanks!

 

Before & After Web GUI woes

 

unraid2-diagnostics-20191214-2134.zip UnraidSyslogDec16.txt unraid2-smart-20191215.zip

[trurl]

1 hour ago, Matt.tr said:

Dec 14 16:28:01 UNRAID2 kernel: FAT-fs (sda1): Directory bread(block 30552) failed

Possibly just doing checkdisk on your PC would have fixed this.

 

1 hour ago, Matt.tr said:

I then proceeded to go through the "Replacing your Unraid USB Flash Device and Registration Key" process with that newly created backup zip file, to a new sandisk flash drive.

Did you prepare a new install then copy just the config folder from that backup, or did you attempt to use the complete backup zip? Config folder is all you need to add to a new install to get your configuration back.

 

1 hour ago, Matt.tr said:

I also have this odd web interface issue

Have you cleared browser cache and whitelisted your server in your adblocker?

[Matt.tr]

I did run check disk through the GUI on a win 10 machine on the original flash drive but I remember it not finding any errors to repair. For good measures, I went back and ran chkdsk /f /r but again no errors to repair.

 

I used the complete backup zip to prep the new flash drive. Should I have just prepped the new flash drive as a fresh install and copy the old config folder over top of it?

 

I installed Chrome for the first time on another pc on the network. Rebooted the unraid server, I logged in and was able to start the array. It auto started a parity check, a few moments later I "lost" the web interface.

 

Since I can't get to tools | diagnostics, I've attached the latest syslog via a text file.

 

Web interface goes back to this below.

Dec 16 New SysLog.txt

[trurl]

7 minutes ago, Matt.tr said:

Should I have just prepped the new flash drive as a fresh install and copy the old config folder over top of it?

Try this. Since you were having problems with the filesystem on flash when you made the backup then maybe you no longer have a good install.

 

And, in fact, that syslog shows you still have corruption of flash.

 

After you prepare a new install and copy config, be sure to use a USB2 port to boot from.

 

Also, see this sticky pinned near the top of this same subforum for instructions on how to get diagnostics from the command line:

https://forums.unraid.net/forum/index.php?topic=39257.0

[Matt.tr]

I prepped another flash drive with a fresh install and copied the old config folder over top of it. Everything was going the best it has over the past few days. Started the array in maintenance mode with no issues. As soon as I pressed the stop button for the array, the web gui "gave out" on me again. I rebooted the server and tried again, this time I could start/stop the array in maintenance mode but as soon as I tried starting the array normally, the web gui "gave out" on me again.

 

Hopefully I captured the diagnostics via terminal properly this time around.

Looks like I'm having a string on bad luck with flash drives. I'll purchase a brand new one tomorrow since all electronic stores near me are closed right now.

 

kernel: FAT-fs (sda1): Directory bread(block 30552) failed

 

Flash drives I've tried over the past few days: SanDisk Cruzer Fit B35 16GB & SanDisk Ultra USB 3.0 16GB

 

When it comes to USB 2.0 ports on my Asus X99-E WS/USB 3.1 motherboard, I'm quite limited in my options. All of the ports on the back of the I/O shield are 3.0 or 3.1. I've tried the USB 2.0 motherboard header with this fresh install, using a StarTech USB 2.0 A to USB 4 Pin to Motherboard Header Adapter (https://www.startech.com/ca/Cables/USB-2.0/Internal-and-Panel-Mount/6-USB-A-Female-to-Motherboard-Header-Adapter~USBMBADAPT). Which USB port would you recommend using in my scenario?

unraid2-diagnostics-20191216-2140.zip

[trurl]

I recommend a USB2 flash drive that isn't so small that it overheats, and a USB2 port. Specs for that board seem to say it has 2 USB2 headers so maybe try the other.

 

Other than that I don't know what to suggest. Most people don't have these issues and those that do can get past them by using a good flash in a USB2 port. Many people have no problems even when using a USB3 port. And I am even using one of those small Sandisk Fit, USB2 flash in USB2 port, for years with no trouble.

[trurl]

Your diagnostics definitely show it has lost the flash drive and that is the cause of your symptoms.