jbrukardt

That makes perfect sense, as the video card that used to be on that slot was passed to the VM. You are awesome.

Repaired the filesystem on all disks in maintenance mode. Mounted disks in normal mode, entire machine froze. At this point, I want to recover any data I can and copy it to a different system.... best way to do that?

Diagnostics attached. Some very odd stuff going on, and I'm concerned I may have lost all my data. In the event log, I have continuous errors: Sep 28 09:05:43 citadel kernel: XFS: metadata IO error: 6697 callbacks suppressed Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS: metadata IO error: 452 callbacks suppressed Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md1p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md3p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS: metadata IO error: 533 callbacks suppressed Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 Sep 28 09:05:43 citadel kernel: XFS (md2p1): metadata I/O error in "xfs_buf_ioend+0x10d/0x380" at daddr 0x80 len 32 error 5 \ When i stop the array to try to do a filesystem check, it loses all the disks in the array: On a reboot, I can get them back, but when i go to stop it for maintenance mode, same thing happens. Rebooting into safe mode works, at which point I can enter maintenence mode. Upon doing file system checks, I get this: I run a file system check on the failed at random disk (it was Disk 11, which I successfully rebuilt in full, then it was parity 1, then it was disk 12) and they do have errors, which repair. I have checked all my cabling, and this is occurring across multiple cables, multiple backplanes. The only thing changed was the pcie slot i put my HBA in, and it was working well in the previous PCIe slot (PCIe x16 5.0 slot (working), vs PCIe 4.0 x16 slot (errors) citadel-diagnostics-20250928-0906.zip

The docker rollback in 6.12.2 fixed this issue thus far. Been stable for 4 days now, couldnt stay up for more than 40 minutes before that.

good to get rid of those too, but this is what filled the log, its almost 50 errors a second until the log fills and crashes. Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5834 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5834 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5852 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5852 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5867 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5867 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5891 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5891 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5907 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5907 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5912 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5912 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5935 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5935 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5947 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5947 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5967 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5967 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5981 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5981 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 5999 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 5999 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6015 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6015 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6038 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6038 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6048 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6048 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6063 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6063 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6079 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6079 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6084 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6084 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6129 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6129 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6143 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6143 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6153 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6153 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6183 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6183 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6191 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6191 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6207 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6207 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6221 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6221 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6232 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6232 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6249 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6249 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6257 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6257 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6276 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6276 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6292 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6292 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6301 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6301 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6319 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6319 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6328 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6328 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6338 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6338 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6357 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6357 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6364 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6364 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6384 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6384 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6395 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6395 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6411 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6411 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: worker process 6426 exited on signal 6 Jun 2 10:50:04 Media nginx: 2023/06/02 10:50:04 [alert] 13955#13955: shared memory zone "memstore" was locked by 6426

Im guessing this might be the /var/log fill issue that there are now about 6 or 7 different posts about. OP, if you're not headless, you should still have command line access physically on the server. check your /var/log fullness with du -sm /var/log/* You should also be able to cat /var/log/syslog from there and see if its filled with NGINX Error 6 spam. If it is, check my post here: https://forums.unraid.net/topic/138726-612-webui-inaccessible-and-varlog-full/#comment-1267931 @Eddie Seelkealso seems to be having a similar issue here:

Please see my post here... i found your cloudflared script in another post from earlier this year, and I expect you may be using cloudflared to connect to unraid. I also am, and whether it be the docker, or your script, thats what triggers the WEBUI issue. Disable cloudflared, and the problem goes away. Something changed in the 6.11 release to cause this, or has changed in cloudflared.

wget -nc http://s3.syncd.tech/files/unraid/supervisord ^^ this seems to be an invalid URL

I did, no response unfortunately since its not a universal problem with the container, but rather one with its interaciton with unraid.

Caught it process. Diagnostics attached before the server locked up with a full log several minutes later. At this point if this cant be resolved, I will have to switch off unraid. scarif-vault-diagnostics-20230601-1435.zip

I would also like to understand at a higher level, why unraid fails so badly when logs fill up. Ideally logrotate or something should kick in rather than the server locking up.

I have isolated this issue on my setup to the cloudflared docker. No error 6 without that docker active.

This is 100% tied to the cloudflared docker. Logs do not fill, no signal 6 errors when its off. I would like to understand why at a more strategic level, Unraid bricks itself when the logs partitition is full, rather than logrotate or something kicking in. Thats very poor behavior to brick the OS when logs fill up.

Im also experiencing this with no resolution

Any reason why the Unraid-Cloudflared-Tunnel docker would be spitting thousands of NGINX errors into syslog? Ive been chasing the problem here, and the only thing that makes it reliably go away is turning off Unraid-Cloudflared-Tunnel docker. Im testing now with just the raw docker command straight from the zero-trust dashboard.