Hello,

 

I was on holidys and came back home to a frozen Server. So I have no exact point of time wehen this started to occur.

This server was runing with this config for about a year now, no problems what so ever. 

 

 

I can start the server normaly. I have autostart of the array of, because I use encypted disks and have to add the keyfile.

After I add the keyfile and start the array, it comes up normaly, all Docker container and VMs start normaly.

 

I tried allready booting, in safemode, still freezing after some time. I disabled docker and VMs, still freezing.

 

some odd things i encountered:

1. My Main VM had a "unkown USB Device" and could not start at the first try, but i removed the device from the config an not it is starting normaly with the rest.

2. I had some CRC errors on one of my drives in a seperate pool. So I remoed the drive, and started the server againe, bit still freezing after some time. Now I added the drive back to the pool, without any problems.

3. Sometimes all my devices in pools get unassigned and i need to reassign them.

 

I tried memtest allready and let it run for 5 passes without any errors. 

Since i use a seperate nvme SSD which is passed thout to my main VM seperatly I also tried booting from that and it worked (had some problems because none of the Networkshares are there, but it ran an showed all drives. Let it sit in Idle for about 2h.

Therefor I don't think it's a Hardware problem.

Edit: I also have no overclock, and c-state is set to "typical current idle"

 

Atatched diagnostics after array start.

And also some Logfiles I gathered after turning on the logserver.

 

Can anybody help me to figure out what's the problem with my server?

If you need any more Info please let me know.

 

syslog-192.168.0.10.log tower-diagnostics-20230821-2124.zip

Edited August 21, 20232 yr by GurKeSaLaT

Flash drive is dropping offline, try using a different USB port, ideally USB 2.0, if it keeps dropping replace it.

Thanks for the response, will try different USB-Ports this evening. 

Can you name me a tool to test the USB-Key? I Thinks it is about 7 years old, maybe it is dying.

5 minutes ago, GurKeSaLaT said:

Can you name me a tool to test the USB-Key?

Not really, but if it keeps dropping on multiple ports replace it.

I tried every USB-Port but still freezing.

Then i Tried a new USB-Key, and it still keeps freezing.

I doubt it has something to do with the controler on the Mainboard, i got a extra USB-Card which I normaly pass into a VM, so i can hotplug, but even in those ports its freezing after a short while.

Any other Ideas?

1 minute ago, GurKeSaLaT said:

Then i Tried a new USB-Key, and it still keeps freezing.

Post new diags to see if the flash drive is still dropping.

2 hours ago, JorgeB said:

Post new diags to see if the flash drive is still dropping.

Here are new diags.

I think by now it is even faster freezing, sometimes there is not even enough time to get the array started now.

syslog-192.168.0.10 (8).log tower-diagnostics-20230822-2115.zip

Aug 22 18:04:52 Tower kernel: xhci_hcd 0000:11:00.3: xHCI host not responding to stop endpoint command
Aug 22 18:04:52 Tower kernel: xhci_hcd 0000:11:00.3: xHCI host controller not responding, assume dead
Aug 22 18:04:52 Tower kernel: xhci_hcd 0000:11:00.3: HC died; cleaning up
Aug 22 18:04:52 Tower kernel: usb 5-1: USB disconnect, device number 2

It's still dropping, you are also having issues with multiple devices:

 

Aug 22 18:06:04 Tower kernel: sd 4:0:0:0: [sdb] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=77s
Aug 22 18:06:04 Tower kernel: sd 4:0:0:0: [sdb] tag#23 CDB: opcode=0x2a 2a 00 01 23 ae 80 00 01 40 00
Aug 22 18:06:04 Tower kernel: I/O error, dev sdb, sector 19115648 op 0x1:(WRITE) flags 0x1800 phys_seg 21 prio class 0
Aug 22 18:06:04 Tower kernel: btrfs_dev_stat_print_on_error: 6 callbacks suppressed
Aug 22 18:06:04 Tower kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 1, rd 676, flush 0, corrupt 0, gen 0
Aug 22 18:06:04 Tower kernel: sd 4:0:0:0: [sdb] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=77s
Aug 22 18:06:04 Tower kernel: sd 4:0:0:0: [sdb] tag#24 CDB: opcode=0x2a 2a 00 00 40 b1 00 00 00 20 00
Aug 22 18:06:04 Tower kernel: I/O error, dev sdb, sector 4239616 op 0x1:(WRITE) flags 0x1800 phys_seg 2 prio class 0
Aug 22 18:06:04 Tower kernel: BTRFS error (device sdb1): bdev /dev/sdb1 errs: wr 2, rd 676, flush 0, corrupt 0, gen 0
Aug 22 18:06:04 Tower kernel: BTRFS: error (device sdb1) in btrfs_commit_transaction:2418: errno=-5 IO failure (Error while writing out transaction)
Aug 22 18:06:04 Tower kernel: BTRFS info (device sdb1: state E): forced readonly
Aug 22 18:06:04 Tower kernel: BTRFS warning (device sdb1: state E): Skipping commit of aborted transaction.
Aug 22 18:06:04 Tower kernel: BTRFS: error (device sdb1: state EA) in cleanup_transaction:1982: errno=-5 IO failure

 

Aug 22 18:05:34 Tower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Aug 22 18:05:34 Tower kernel: ata5: EH complete
Aug 22 18:05:34 Tower kernel: sd 6:0:0:0: [sdd] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=77s
Aug 22 18:05:34 Tower kernel: sd 6:0:0:0: [sdd] tag#18 CDB: opcode=0x28 28 00 3f 9e 63 60 00 00 20 00
Aug 22 18:05:34 Tower kernel: I/O error, dev sdd, sector 1067344736 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 0
Aug 22 18:05:34 Tower kernel: md: disk1 read error, sector=1067344672
Aug 22 18:05:34 Tower kernel: md: disk1 read error, sector=1067344680
Aug 22 18:05:34 Tower kernel: md: disk1 read error, sector=1067344688
Aug 22 18:05:34 Tower kernel: md: disk1 read error, sector=1067344696

 

And there are more, also issues with the HBA, suggesting a more serious hardware issue, like bad PSU, board, etc.

Damn, I switched powersuplys, but it is still freezing. Now I orderd a new Mainboard, I hope it' not the processor. 

I'll report back when i tested the new board.

Thanks for the help this far!

Ok, i got a new Mainboard and not everything is working again. Thanks for the support!