This morning, UNRAID was not accessible through gui, console, ssh anymore. But File Access still worked, VMs and Dockers were running.

 

No chance for a clean shutdown. the reset button was my friend.

 

From the Syslog I found that:

Sep 19 07:12:48 F kernel: usb 1-3: device descriptor read/8, error -110
Sep 19 07:12:49 F kernel: I/O error, dev sda, sector 25061216 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 2
Sep 19 07:12:49 F kernel: sd 0:0:0:0: [sda] tag#0 timing out command, waited 60s
Sep 19 07:12:49 F kernel: device offline error, dev sda, sector 25060976 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Sep 19 07:12:49 F kernel: device offline error, dev sda, sector 25071528 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
Sep 19 07:12:49 F kernel: device offline error, dev sda, sector 25071528 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Sep 19 07:12:49 F kernel: I/O error, dev loop1, sector 537496 op 0x0:(READ) flags 0x800 phys_seg 15 prio class 2
Sep 19 07:12:49 F kernel: I/O error, dev loop1, sector 548152 op 0x0:(READ) flags 0x800 phys_seg 10 prio class 2
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10673488: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10ba7490: -5
Sep 19 07:12:49 F kernel: device offline error, dev sda, sector 25071600 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
Sep 19 07:12:49 F kernel: device offline error, dev sda, sector 25071600 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Sep 19 07:12:49 F kernel: device offline error, dev sda, sector 25060976 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Sep 19 07:12:49 F kernel: I/O error, dev loop1, sector 548224 op 0x0:(READ) flags 0x800 phys_seg 14 prio class 2
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10bb0d58: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10673488: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10ba7490: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10673488: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10ba7490: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10681098: -5
Sep 19 07:12:49 F kernel: SQUASHFS error: Failed to read block 0x10673488: -5

(repeated over and over again)

Hard reset worked, the stick is working again. (No read errors so far).

The box was running for 60days already, this never happened before (yet, fingers crossed!)

 

Any idea what happened and how it can be prevented???

 

(before somebody complains, there are no diagnostics, I had to reboot the hard way. The syslog is still available, it is stored on a different machine)

Edited September 19, 20232 yr by MAM59

Try a new USB port, ideally USB 2.0, if it happens again you can try replacing the flash drive.

in between I have tracked down the issue a bit more.

It looks like the usb plug of the UPS cable was a bit too sloppy. It did not fit in place anymore and due to the vibrations of all these fans and drives, it loses contact now and then.

It seems that the kernel went crazy with all these "on" / "off" transitions of the device and turned off the usb controller.

The flash drive was using a port on the same controller.

 

I have now pulled out the UPS, squeezed the plug a bit with some pliers, and have put it into a raspberry in the same rack as the unraid box.

Installed NUT on the PI in master mode and NUT on UNRAID in slave mode.

We will see if this was the evildoer, the PI does not care about "ports running wild".

 

@JorgeB all ports, sticks and controllers are USB 2.0 only. I have learned my lesson last year already. And the flash drive is not corrupted (yet) at all, it was just disabled.