ryantomlinson95

Thanks! Up and running now with a single stick of ram and a second one on the way. Pools scrubbed and no data errors, recreated all my containers from backups, and monitoring for new corruptions. Appreciate all the help 🙌

Thanks for that! One of my two sticks boots and passes memtest with no errors. The other one I can't get past bios so I assume its bad in some way, even though its recognized in bios. Should I try running my server for a few days with just the single stick of RAM or should I try and replace it first?

Yeah I just checked my BIOS settings and they all look correct according to the FAQ post you linked, namely Power Supply Idle Control -> Typical Current Idle, and memory frequency at DDR4-3200MHz.

Ok got it, makes sense! I guess I'm wondering if buying new sticks is the first next step I should take? Sounds like I need to also get all my BIOS settings correct so I don't cause more issues with the new sticks.

Thank you both for the support & recommendations. I ran memtest first since that was the easiest thing to try...lots of errors. Results are tens of thousands of errors over multiple tests in all 32GB of RAM. Does this just mean I need new sticks (along with the Ryzen system recommendations)? And if so, are there any I should lean towards or away from given my current system specs? My understanding is that I would not be able to go with ECC RAM given my Ryzen cpu. edit: Didn't realize memtest is still running and runs multiple times. At ~400k errors so far, will update one more time once its done.

Link to previous issue where my cache would not mount so I wiped and restarted: After a few days of working with no issues, it appears that around April 10th/11th in the logs, I'm getting...I don't even know, just a bunch of faults/error codes that are indecipherable to me. This is the issue I was having before the issue that I linked above which eventually caused my cache to be unmountable. Typically I restart my server, fix any data errors in the cache and then keep going. Happens every couple days. The two logs I attached are the downloaded syslogs from after i rebooted the server (`megaplex` prefix), and the syslogslogs that I backup which show what happened prior to reboot (no prefix). I've also attached diagnostics. Please let me know if there's anything else I can upload. Not sure what's going on! syslog-192.168.0.10.log megaplex-syslog-20250412-1608.zip megaplex-diagnostics-20250412-1014.zip

Apr 8 08:25:40 megaplex kernel: mdcmd (36): nocheck pause Apr 8 08:25:42 megaplex emhttpd: creating volume: cache (zfs) Apr 8 08:25:42 megaplex emhttpd: shcmd (58975): /sbin/wipefs -af --lock /dev/nvme1n1p1 Apr 8 08:25:42 megaplex emhttpd: writing MBR on disk (nvme1n1) with partition 1 offset 2048, erased: 0 Apr 8 08:25:43 megaplex emhttpd: re-reading (nvme1n1) partition table Apr 8 08:25:43 megaplex emhttpd: error: mkmbr, 2329: Device or resource busy (16): ioctl BLKRRPART: /dev/nvme1n1 Apr 8 08:25:43 megaplex emhttpd: shcmd (58976): udevadm settle Apr 8 08:25:43 megaplex emhttpd: shcmd (58977): /sbin/blkdiscard /dev/nvme1n1p1 Apr 8 08:25:43 megaplex root: blkdiscard: cannot open /dev/nvme1n1p1: Device or resource busy Apr 8 08:25:43 megaplex emhttpd: shcmd (58977): exit status: 1 Apr 8 08:25:43 megaplex emhttpd: continuing... Apr 8 08:25:43 megaplex emhttpd: shcmd (58978): /sbin/wipefs -af --lock /dev/nvme0n1p1 Apr 8 08:25:43 megaplex emhttpd: writing MBR on disk (nvme0n1) with partition 1 offset 2048, erased: 0 Apr 8 08:25:44 megaplex emhttpd: re-reading (nvme0n1) partition table Apr 8 08:25:44 megaplex emhttpd: error: mkmbr, 2329: Device or resource busy (16): ioctl BLKRRPART: /dev/nvme0n1 Apr 8 08:25:44 megaplex emhttpd: shcmd (58979): udevadm settle Apr 8 08:25:44 megaplex emhttpd: shcmd (58980): /sbin/blkdiscard /dev/nvme0n1p1 Apr 8 08:25:44 megaplex root: blkdiscard: cannot open /dev/nvme0n1p1: Device or resource busy Apr 8 08:25:44 megaplex emhttpd: shcmd (58980): exit status: 1 Apr 8 08:25:44 megaplex emhttpd: continuing... Apr 8 08:25:44 megaplex emhttpd: shcmd (58981): /usr/sbin/zpool create -f -o ashift=12 -O dnodesize=auto -O acltype=posixacl -O xattr=sa -O utf8only=on -m /mnt/cache cache mirror /dev/nvme1n1p1 /dev/nvme0n1p1 Apr 8 08:25:44 megaplex root: cannot open '/dev/nvme1n1p1': Device or resource busy Apr 8 08:25:44 megaplex root: cannot open '/dev/nvme0n1p1': Device or resource busy Apr 8 08:25:44 megaplex root: mountpoint '/mnt/cache' exists and is not empty Apr 8 08:25:44 megaplex root: use '-m' option to provide a different default Apr 8 08:25:44 megaplex emhttpd: shcmd (58981): exit status: 1 Apr 8 08:25:44 megaplex emhttpd: mounting /mnt/cache Apr 8 08:25:44 megaplex emhttpd: shcmd (58982): mkdir -m 0666 -p /mnt/cache Apr 8 08:25:44 megaplex emhttpd: shcmd (58983): rmdir /mnt/cache Apr 8 08:25:44 megaplex root: rmdir: failed to remove '/mnt/cache': Device or resource busy Apr 8 08:25:44 megaplex emhttpd: shcmd (58983): exit status: 1 Apr 8 08:25:44 megaplex emhttpd: cache: mount error: wrong or no file system Apr 8 08:25:44 megaplex kernel: mdcmd (37): check resume I tried both methods, but then started seeing this in the syslog when I tried to re-format, and the gui still showed `Unmountable: wrong or no file system`. Fix common problems was also suggesting the cache pool is still mounted read-only. When I tried: zpool export cache I got: root@megaplex:~# zpool export cache cannot export 'cache': pool is busy I tried a few other things like stopping the docker service and deleting the docker vdisk. Finally a reboot and retry of the above steps worked and it looks like all is good at the moment: Apr 8 08:44:41 megaplex kernel: mdcmd (36): nocheck pause Apr 8 08:44:43 megaplex emhttpd: creating volume: cache (zfs) Apr 8 08:44:43 megaplex emhttpd: shcmd (265): /sbin/wipefs -af --lock /dev/nvme1n1p1 Apr 8 08:44:43 megaplex root: wipefs: error: /dev/nvme1n1p1: probing initialization failed: No such file or directory Apr 8 08:44:43 megaplex emhttpd: shcmd (265): exit status: 1 Apr 8 08:44:43 megaplex emhttpd: writing MBR on disk (nvme1n1) with partition 1 offset 2048, erased: 0 Apr 8 08:44:43 megaplex kernel: nvme1n1: p1 Apr 8 08:44:44 megaplex emhttpd: re-reading (nvme1n1) partition table Apr 8 08:44:44 megaplex emhttpd: shcmd (266): udevadm settle Apr 8 08:44:44 megaplex kernel: nvme1n1: p1 Apr 8 08:44:44 megaplex kernel: nvme1n1: p1 Apr 8 08:44:44 megaplex emhttpd: shcmd (267): /sbin/blkdiscard /dev/nvme1n1p1 Apr 8 08:44:45 megaplex emhttpd: shcmd (268): /sbin/wipefs -af --lock /dev/nvme0n1p1 Apr 8 08:44:45 megaplex root: wipefs: error: /dev/nvme0n1p1: probing initialization failed: No such file or directory Apr 8 08:44:45 megaplex emhttpd: shcmd (268): exit status: 1 Apr 8 08:44:45 megaplex emhttpd: writing MBR on disk (nvme0n1) with partition 1 offset 2048, erased: 0 Apr 8 08:44:45 megaplex kernel: nvme0n1: p1 Apr 8 08:44:46 megaplex emhttpd: re-reading (nvme0n1) partition table Apr 8 08:44:46 megaplex emhttpd: shcmd (269): udevadm settle Apr 8 08:44:46 megaplex kernel: nvme0n1: p1 Apr 8 08:44:46 megaplex kernel: nvme0n1: p1 Apr 8 08:44:46 megaplex emhttpd: shcmd (270): /sbin/blkdiscard /dev/nvme0n1p1 Apr 8 08:44:46 megaplex emhttpd: shcmd (271): /usr/sbin/zpool create -f -o ashift=12 -O dnodesize=auto -O acltype=posixacl -O xattr=sa -O utf8only=on -m /mnt/cache cache mirror /dev/nvme1n1p1 /dev/nvme0n1p1 Apr 8 08:44:46 megaplex emhttpd: update_pool_cfg: 30 cache 0 Apr 8 08:44:46 megaplex emhttpd: shcmd (272): /usr/sbin/zpool export -f cache Apr 8 08:44:46 megaplex emhttpd: mounting /mnt/cache Apr 8 08:44:46 megaplex emhttpd: shcmd (273): mkdir -m 0666 -p /mnt/cache Apr 8 08:44:46 megaplex emhttpd: shcmd (274): /usr/sbin/zpool import -f -m -N -o autoexpand=on -d /dev/nvme1n1p1 -d /dev/nvme0n1p1 5886927328334564531 cache Apr 8 08:44:46 megaplex emhttpd: cache: zfs verify devices Apr 8 08:44:46 megaplex emhttpd: /usr/sbin/zpool status -P cache 2>&1 Apr 8 08:44:46 megaplex emhttpd: pool: cache Apr 8 08:44:46 megaplex emhttpd: state: ONLINE Apr 8 08:44:46 megaplex emhttpd: config: Apr 8 08:44:46 megaplex emhttpd: NAME STATE READ WRITE CKSUM Apr 8 08:44:46 megaplex emhttpd: cache ONLINE 0 0 0 Apr 8 08:44:46 megaplex emhttpd: mirror-0 ONLINE 0 0 0 Apr 8 08:44:46 megaplex emhttpd: /dev/nvme1n1p1 ONLINE 0 0 0 Apr 8 08:44:46 megaplex emhttpd: /dev/nvme0n1p1 ONLINE 0 0 0 Apr 8 08:44:46 megaplex emhttpd: errors: No known data errors Apr 8 08:44:46 megaplex emhttpd: shcmd (275): /usr/sbin/zfs set -u mountpoint=/mnt/cache cache Apr 8 08:44:46 megaplex emhttpd: shcmd (276): /usr/sbin/zfs mount -o atime=off cache Apr 8 08:44:46 megaplex emhttpd: shcmd (277): /usr/sbin/zpool set autotrim=on cache Apr 8 08:44:46 megaplex emhttpd: shcmd (278): /usr/sbin/zfs set compression=on cache Apr 8 08:44:47 megaplex emhttpd: update_pool_cfg: 30 cache 0 Apr 8 08:44:47 megaplex emhttpd: cache subpool: root profile: mirror width: 2 groups: 1 Apr 8 08:44:47 megaplex kernel: mdcmd (37): check resume I then recreated the docker vdisk, copied back over all my appdata and recreated my containers from my templates. Everything seems to be running well. The only thing I noticed was a single segfault while copying back my appdata to the cache pool Apr 8 08:50:38 megaplex kernel: rc.diskinfo[45979]: segfault at 40020 ip 0000148e5adb02b2 sp 00007ffd1bfeb6d0 error 4 in libcrypto.so.3[148e5ac76000+332000] likely on CPU 5 (core 5, socket 0) Apr 8 08:50:38 megaplex kernel: Code: 48 8b 55 00 45 31 ff 4c 89 e3 48 c1 e3 06 0f 18 0c 1a 48 83 eb 40 4c 89 f9 48 c1 e1 04 48 01 ca 4c 8b 74 13 08 4d 85 f6 74 1a <49> 8b 56 20 4c 89 f7 ff 52 08 ba 7c 02 00 00 4c 89 ee 4c 89 f7 e8 For now I think I'm all set, but if I run into more issues (I had cache issues leading up to the full corruption) then I will open a new support item. Thank you so much for your help!! Really appreciate the time.

Thanks! I've copied over anything important to my drive array. To re-format the cache-pool do I need to unmount it or stop the array first? I haven't done that before and can't find concise steps anywhere

@JorgeB thanks for helping out! It does seem to mount as read-only. After that I can start the array, but obviously I can't access any data on the cache pool. I do run app-data backups daily so I have that if needed.

I'm having an issue recently where I can no longer mount my disks. Prior to this, my server was going inaccessible every ~2 to 3 days, forcing me to restart the whole system by manual reboot. I attached the most recent syslog. I had more syslogs saved to my server but I cant reach them without mounting all the disks. Most of my issues before would be solved by deleting a container image and recreating it, or deleting the actual docker base image and re creating that. But they'd all eventually come back, almost like a memory leak or something. I'd see a lot of segfaults in the logs. megaplex-syslog-20250404-2151.zip megaplex-diagnostics-20250404-1555.zip