So ive had a look around but other posts are either hard drives or sata ssds and usually its a power issue.

 

So every so often my server will become buggy, shares disappear, cant mount remote shares, dockers arent working. When i check it appears my cache drive has fallen offline. 

 

I try to add it back in but its not detected, however i can see it in unassigned devices.

 

A restart doesnt fix it, only a full shutdown and reboot does.

 

This happened with an old nvme drive i had also before i upgraded to a newer, faster one.

 

While the device is in unassigned devices the disk log shows

 

Quote

Nov  9 07:00:34 Hercules emhttpd: import 30 pool device: (nvme1n1) Samsung_SSD_980_1TB_S649NL0W176107A
Nov  9 07:00:34 Hercules emhttpd: read SMART /dev/nvme1n1
Nov  9 07:00:46 Hercules emhttpd: import 30 pool device: (nvme1n1) Samsung_SSD_980_1TB_S649NL0W176107A
Nov  9 07:00:55 Hercules emhttpd: shcmd (126): mount -t xfs -o noatime,nouuid /dev/nvme1n1p1 /mnt/cache
Nov  9 07:00:55 Hercules kernel: XFS (nvme1n1p1): Mounting V5 Filesystem a8afe6ed-d9d9-4fb8-b2c3-3d0892cbe211
Nov  9 07:00:55 Hercules kernel: XFS (nvme1n1p1): Ending clean mount
Nov  9 07:04:35 Hercules emhttpd: spinning down /dev/nvme1n1
Nov  9 07:04:54 Hercules emhttpd: sdspin /dev/nvme1n1 down: 1
Nov 12 13:32:47 Hercules kernel: I/O error, dev nvme1n1, sector 620560416 op 0x1:(WRITE) flags 0x104000 phys_seg 3 prio class 2
Nov 12 13:32:47 Hercules kernel: I/O error, dev nvme1n1, sector 5000 op 0x1:(WRITE) flags 0x1000 phys_seg 1 prio class 2
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): metadata I/O error in "xfs_buf_ioend+0x111/0x384" at daddr 0xb88 len 8 error 5
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): metadata I/O error in "xfs_imap_to_bp+0x52/0x74" at daddr 0x151e5100 len 32 error 5
Nov 12 13:32:47 Hercules kernel: nvme1n1p1: writeback error on inode 1811943943, offset 648556544, sector 1827778584
Nov 12 13:32:47 Hercules kernel: nvme1n1p1: writeback error on inode 1811943943, offset 4042924032, sector 1888867448
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): metadata I/O error in "xfs_imap_to_bp+0x52/0x74" at daddr 0x49232340 len 32 error 5
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): log I/O error -5
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): Filesystem has been shut down due to log error (0x2).
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): metadata I/O error in "xfs_buf_ioend+0x111/0x384" at daddr 0x6ed8 len 8 error 5
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): Please unmount the filesystem and rectify the problem(s).
Nov 12 13:32:47 Hercules kernel: nvme1n1p1: writeback error on inode 679475078, offset 0, sector 620562976
Nov 12 13:32:47 Hercules kernel: XFS (nvme1n1p1): log I/O error -5
Nov 12 13:32:47 Hercules kernel: nvme1n1p1: writeback error on inode 1811943943, offset 3274702848, sector 1775463136
Nov 12 13:32:47 Hercules kernel: nvme1n1p1: writeback error on inode 1811943943, offset 3274772480, sector 1775463272
Nov 12 13:42:50 Hercules kernel: nvme1n1p1: writeback error on inode 6706559, offset 0, sector 6727528
Nov 12 14:25:30 Hercules emhttpd: read SMART /dev/nvme1n1
Nov 12 14:28:01 Hercules kernel: XFS (nvme1n1p1): Unmounting Filesystem a8afe6ed-d9d9-4fb8-b2c3-3d0892cbe211
Nov 12 14:28:02 Hercules emhttpd: device /dev/nvme1n1 has size zero
Nov 12 14:28:02 Hercules emhttpd: offline: Samsung_SSD_980_1TB_S649NL0W176107A (nvme1n1) 512 1953525168

As far as i know the other drives were mostly spun down so there was virtually no activity other than myself streaming a show from plex (plex on a separate server).

i finally remembered this time to grab the diagnostics before rebooting so any help is appreciated 

hercules-diagnostics-20241112-2232.zip

Diags are not working, try attaching them again.