I'm having an issue recently where I can no longer mount my disks. Prior to this, my server was going inaccessible every ~2 to 3 days, forcing me to restart the whole system by manual reboot. I attached the most recent syslog. I had more syslogs saved to my server but I cant reach them without mounting all the disks.

 

Most of my issues before would be solved by deleting a container image and recreating it, or deleting the actual docker base image and re creating that. But they'd all eventually come back, almost like a memory leak or something. I'd see a lot of segfaults in the logs.

megaplex-syslog-20250404-2151.zip megaplex-diagnostics-20250404-1555.zip

Cache pool is crashing on mount, see if it mounts read-only, reboot and before array start type

 

zpool import -o readonly=on cache

 

@JorgeB thanks for helping out! It does seem to mount as read-only. After that I can start the array, but obviously I can't access any data on the cache pool. I do run app-data backups daily so I have that if needed.

37 minutes ago, ryantomlinson95 said:

After that I can start the array, but obviously I can't access any data on the cache pool

The GUI will show the pool unmountable, but the data should still be under /mnt/cache, copy anything important to the array or another pool/device and then re-format the pool

Thanks!

Quote

copy anything important to the array or another pool/device and then re-format the pool

I've copied over anything important to my drive array. To re-format the cache-pool do I need to unmount it or stop the array first? I haven't done that before and can't find concise steps anywhere

You would need to export the pool manually first with 

zpool export cache

After that, you should be able to format it with the array still running, if not, stop the array, click the first pool device, then "erase pool", start the array and re-format.

Apr  8 08:25:40 megaplex kernel: mdcmd (36): nocheck pause
Apr  8 08:25:42 megaplex emhttpd: creating volume: cache (zfs)
Apr  8 08:25:42 megaplex emhttpd: shcmd (58975): /sbin/wipefs -af --lock /dev/nvme1n1p1
Apr  8 08:25:42 megaplex emhttpd: writing MBR on disk (nvme1n1) with partition 1 offset 2048, erased: 0
Apr  8 08:25:43 megaplex emhttpd: re-reading (nvme1n1) partition table
Apr  8 08:25:43 megaplex emhttpd: error: mkmbr, 2329: Device or resource busy (16): ioctl BLKRRPART: /dev/nvme1n1
Apr  8 08:25:43 megaplex emhttpd: shcmd (58976): udevadm settle
Apr  8 08:25:43 megaplex emhttpd: shcmd (58977): /sbin/blkdiscard /dev/nvme1n1p1
Apr  8 08:25:43 megaplex root: blkdiscard: cannot open /dev/nvme1n1p1: Device or resource busy
Apr  8 08:25:43 megaplex emhttpd: shcmd (58977): exit status: 1
Apr  8 08:25:43 megaplex emhttpd: continuing...
Apr  8 08:25:43 megaplex emhttpd: shcmd (58978): /sbin/wipefs -af --lock /dev/nvme0n1p1
Apr  8 08:25:43 megaplex emhttpd: writing MBR on disk (nvme0n1) with partition 1 offset 2048, erased: 0
Apr  8 08:25:44 megaplex emhttpd: re-reading (nvme0n1) partition table
Apr  8 08:25:44 megaplex emhttpd: error: mkmbr, 2329: Device or resource busy (16): ioctl BLKRRPART: /dev/nvme0n1
Apr  8 08:25:44 megaplex emhttpd: shcmd (58979): udevadm settle
Apr  8 08:25:44 megaplex emhttpd: shcmd (58980): /sbin/blkdiscard /dev/nvme0n1p1
Apr  8 08:25:44 megaplex root: blkdiscard: cannot open /dev/nvme0n1p1: Device or resource busy
Apr  8 08:25:44 megaplex emhttpd: shcmd (58980): exit status: 1
Apr  8 08:25:44 megaplex emhttpd: continuing...
Apr  8 08:25:44 megaplex emhttpd: shcmd (58981): /usr/sbin/zpool create -f -o ashift=12 -O dnodesize=auto -O acltype=posixacl -O xattr=sa -O utf8only=on -m /mnt/cache cache  mirror /dev/nvme1n1p1 /dev/nvme0n1p1 
Apr  8 08:25:44 megaplex root: cannot open '/dev/nvme1n1p1': Device or resource busy
Apr  8 08:25:44 megaplex root: cannot open '/dev/nvme0n1p1': Device or resource busy
Apr  8 08:25:44 megaplex root: mountpoint '/mnt/cache' exists and is not empty
Apr  8 08:25:44 megaplex root: use '-m' option to provide a different default
Apr  8 08:25:44 megaplex emhttpd: shcmd (58981): exit status: 1
Apr  8 08:25:44 megaplex emhttpd: mounting /mnt/cache
Apr  8 08:25:44 megaplex emhttpd: shcmd (58982): mkdir -m 0666 -p /mnt/cache
Apr  8 08:25:44 megaplex emhttpd: shcmd (58983): rmdir /mnt/cache
Apr  8 08:25:44 megaplex root: rmdir: failed to remove '/mnt/cache': Device or resource busy
Apr  8 08:25:44 megaplex emhttpd: shcmd (58983): exit status: 1
Apr  8 08:25:44 megaplex emhttpd: cache: mount error: wrong or no file system
Apr  8 08:25:44 megaplex kernel: mdcmd (37): check resume

 

I tried both methods, but then started seeing this in the syslog when I tried to re-format, and the gui still showed `Unmountable: wrong or no file system`. Fix common problems was also suggesting the cache pool is still mounted read-only. When I tried:

zpool export cache

I got:

root@megaplex:~# zpool export cache
cannot export 'cache': pool is busy

 

 

 

 

 

I tried a few other things like stopping the docker service and deleting the docker vdisk. Finally a reboot and retry of the above steps worked and it looks like all is good at the moment:

 

Apr  8 08:44:41 megaplex kernel: mdcmd (36): nocheck pause
Apr  8 08:44:43 megaplex emhttpd: creating volume: cache (zfs)
Apr  8 08:44:43 megaplex emhttpd: shcmd (265): /sbin/wipefs -af --lock /dev/nvme1n1p1
Apr  8 08:44:43 megaplex root: wipefs: error: /dev/nvme1n1p1: probing initialization failed: No such file or directory
Apr  8 08:44:43 megaplex emhttpd: shcmd (265): exit status: 1
Apr  8 08:44:43 megaplex emhttpd: writing MBR on disk (nvme1n1) with partition 1 offset 2048, erased: 0
Apr  8 08:44:43 megaplex kernel: nvme1n1: p1
Apr  8 08:44:44 megaplex emhttpd: re-reading (nvme1n1) partition table
Apr  8 08:44:44 megaplex emhttpd: shcmd (266): udevadm settle
Apr  8 08:44:44 megaplex kernel: nvme1n1: p1
Apr  8 08:44:44 megaplex kernel: nvme1n1: p1
Apr  8 08:44:44 megaplex emhttpd: shcmd (267): /sbin/blkdiscard /dev/nvme1n1p1
Apr  8 08:44:45 megaplex emhttpd: shcmd (268): /sbin/wipefs -af --lock /dev/nvme0n1p1
Apr  8 08:44:45 megaplex root: wipefs: error: /dev/nvme0n1p1: probing initialization failed: No such file or directory
Apr  8 08:44:45 megaplex emhttpd: shcmd (268): exit status: 1
Apr  8 08:44:45 megaplex emhttpd: writing MBR on disk (nvme0n1) with partition 1 offset 2048, erased: 0
Apr  8 08:44:45 megaplex kernel: nvme0n1: p1
Apr  8 08:44:46 megaplex emhttpd: re-reading (nvme0n1) partition table
Apr  8 08:44:46 megaplex emhttpd: shcmd (269): udevadm settle
Apr  8 08:44:46 megaplex kernel: nvme0n1: p1
Apr  8 08:44:46 megaplex kernel: nvme0n1: p1
Apr  8 08:44:46 megaplex emhttpd: shcmd (270): /sbin/blkdiscard /dev/nvme0n1p1
Apr  8 08:44:46 megaplex emhttpd: shcmd (271): /usr/sbin/zpool create -f -o ashift=12 -O dnodesize=auto -O acltype=posixacl -O xattr=sa -O utf8only=on -m /mnt/cache cache  mirror /dev/nvme1n1p1 /dev/nvme0n1p1 
Apr  8 08:44:46 megaplex emhttpd: update_pool_cfg: 30 cache 0
Apr  8 08:44:46 megaplex emhttpd: shcmd (272): /usr/sbin/zpool export -f cache
Apr  8 08:44:46 megaplex emhttpd: mounting /mnt/cache
Apr  8 08:44:46 megaplex emhttpd: shcmd (273): mkdir -m 0666 -p /mnt/cache
Apr  8 08:44:46 megaplex emhttpd: shcmd (274): /usr/sbin/zpool import -f -m -N -o autoexpand=on  -d /dev/nvme1n1p1 -d /dev/nvme0n1p1 5886927328334564531 cache
Apr  8 08:44:46 megaplex emhttpd: cache: zfs verify devices
Apr  8 08:44:46 megaplex emhttpd: /usr/sbin/zpool status -P cache 2>&1
Apr  8 08:44:46 megaplex emhttpd:   pool: cache
Apr  8 08:44:46 megaplex emhttpd:  state: ONLINE
Apr  8 08:44:46 megaplex emhttpd: config:
Apr  8 08:44:46 megaplex emhttpd: 	NAME                STATE     READ WRITE CKSUM
Apr  8 08:44:46 megaplex emhttpd: 	cache               ONLINE       0     0     0
Apr  8 08:44:46 megaplex emhttpd: 	  mirror-0          ONLINE       0     0     0
Apr  8 08:44:46 megaplex emhttpd: 	    /dev/nvme1n1p1  ONLINE       0     0     0
Apr  8 08:44:46 megaplex emhttpd: 	    /dev/nvme0n1p1  ONLINE       0     0     0
Apr  8 08:44:46 megaplex emhttpd: errors: No known data errors
Apr  8 08:44:46 megaplex emhttpd: shcmd (275): /usr/sbin/zfs set -u mountpoint=/mnt/cache cache
Apr  8 08:44:46 megaplex emhttpd: shcmd (276): /usr/sbin/zfs mount -o atime=off cache
Apr  8 08:44:46 megaplex emhttpd: shcmd (277): /usr/sbin/zpool set autotrim=on cache
Apr  8 08:44:46 megaplex emhttpd: shcmd (278): /usr/sbin/zfs set compression=on cache
Apr  8 08:44:47 megaplex emhttpd: update_pool_cfg: 30 cache 0
Apr  8 08:44:47 megaplex emhttpd: cache subpool: root profile: mirror width: 2 groups: 1
Apr  8 08:44:47 megaplex kernel: mdcmd (37): check resume

 

I then recreated the docker vdisk, copied back over all my appdata and recreated my containers from my templates. Everything seems to be running well.

The only thing I noticed was a single segfault while copying back my appdata to the cache pool

Apr  8 08:50:38 megaplex kernel: rc.diskinfo[45979]: segfault at 40020 ip 0000148e5adb02b2 sp 00007ffd1bfeb6d0 error 4 in libcrypto.so.3[148e5ac76000+332000] likely on CPU 5 (core 5, socket 0)
Apr  8 08:50:38 megaplex kernel: Code: 48 8b 55 00 45 31 ff 4c 89 e3 48 c1 e3 06 0f 18 0c 1a 48 83 eb 40 4c 89 f9 48 c1 e1 04 48 01 ca 4c 8b 74 13 08 4d 85 f6 74 1a <49> 8b 56 20 4c 89 f7 ff 52 08 ba 7c 02 00 00 4c 89 ee 4c 89 f7 e8

 

For now I think I'm all set, but if I run into more issues (I had cache issues leading up to the full corruption) then I will open a new support item. 

Thank you so much for your help!! Really appreciate the time.