Nedio Server Unraid

issue resolved - it was the slow network (going via power line). replaced that with LAN - all good now. 🙈

Hi, I am running immich 2.x on my unraid v7.1.4. The UI of Unraid as well as writing to the cache pools (ZFS) are ultra slow (2-7MB/s). Please advise where to start troubleshooting.. This was working well before, and is unbearable now. Immich backup uploads assets in the range of KB/s, hangs. Attaching the diagnostics. Thanks nedio-server-diagnostics-20251009-1551.zip

ok, my disk 2 was rebuilt. no errors so far 🤞. Now I want to eliminate the potential reason of the cache-b holding back from array stop/clean reboot.. I have changed the docker settings to be overlay2. Did not enable docker yet. Also I have moved everything from cache-b to array (using the mover). Now, since there is nothing on the cache-b, I would expect to see 0 on the sizes in Main tab: But I see some misterious 35 GB on it. Then I noticed that there are 344 snapshots (legacy) of the system share folder in the cache-b pool. (those grey/yellow/blue photo camera icons are confusing 😕) I want to clean the cache-b. Can I just delete these snapshots? I need to get rid of those legacy snapshots and stop them being taken.

Thank you @JorgeB! I think the answers to both parts of the condition you mentioned are "Yes". But just to clarify, how/where do I see if the emulated disk 2 is mounting? It shows as disabled in the array with a ❌ mark.

Ok, hard restarted 🥶, started the array in maintenance mode (docker and VM are still off). Did the XFS check on disk 2 - no errors detected 🥳 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. No file system corruption detected. The disc 2 still shows disabled. What are the next steps? I can't find the rebuilding-a-drive-onto-itself in the unraid documentation.. Did the procedure of restoring the disabled disc change? How should I proceed to fix the array?

I had switched to docker folder as my docker image file would get corrupt several times (due to not being able to shutdown the system cleanly and having to hard restart).. I will look into re-creating it using overlay2 driver, thanks! For now - any idea how do I proceed? Is it that the only option I have a hard restart again? Since my array is not ok, do I risk to get in trouble even more if I hard restart?

Here's the outcome: SRV:~# lsof | grep /mnt SRV:~# ps -f | grep "bash" | grep -v "grep" root 3081899 3081748 0 23:54 pts/0 00:00:00 /bin/bash --login SRV:~#

Yeah, docker was already stopped at the time of the diags taking (from the settings > docker > enable docker=No). The web-console was also closed (after diag extract). Still I have run the command you suggested. SRV:~# /etc/rc.d/rc.docker status Docker daemon is not running. SRV:~# /etc/rc.d/rc.docker stop rc.docker: Stopping containers... rc.docker: Stopping network... rc.docker: Stopping Docker daemon... rc.docker: Docker daemon... Already stopped. SRV:~# tail -f /var/log/syslog Sep 20 21:22:35 nedio-server emhttpd: Unmounting disks... Sep 20 21:22:35 nedio-server emhttpd: shcmd (39782356): /usr/sbin/zpool export -f cache-b Sep 20 21:22:36 nedio-server root: cannot export 'cache-b': pool is busy Sep 20 21:22:36 nedio-server emhttpd: shcmd (39782356): exit status: 1 Sep 20 21:22:36 nedio-server emhttpd: Retry unmounting disk share(s)... Sep 20 21:22:41 nedio-server emhttpd: Unmounting disks... Sep 20 21:22:41 nedio-server emhttpd: shcmd (39782357): /usr/sbin/zpool export -f cache-b Sep 20 21:22:41 nedio-server root: cannot export 'cache-b': pool is busy Sep 20 21:22:41 nedio-server emhttpd: shcmd (39782357): exit status: 1 Sep 20 21:22:41 nedio-server emhttpd: Retry unmounting disk share(s)... SRV:~#Isn't there a way to figure out what could be hugging the pool cache-b ?

Hi, I needed to stop my array. For that I stopped all the docker containers and the docker service itself. Same with the VMs. Then hit the stop button. The array still did not go down. Syslog shows: Sep 20 16:06:04 nedio-server emhttpd: Unmounting disks... Sep 20 16:06:04 nedio-server emhttpd: shcmd (39778678): /usr/sbin/zpool export -f cache-b Sep 20 16:06:04 nedio-server root: cannot export 'cache-b': pool is busy Sep 20 16:06:04 nedio-server emhttpd: shcmd (39778678): exit status: 1 Sep 20 16:06:04 nedio-server emhttpd: Retry unmounting disk share(s)... I tried to check if there's anything holding the pool. Can't see anything. On the Main tab - the array and the cache pools show online. Note: there is a small write (400-700 KB/s) that happens every few seconds on the cache-b. Please advise what should I do. I want to stop the array in order to start it in maintenance mode, so that I can check the array xfs for errors and then rebuild one of the disks which is showing as x Disabled. Please see attached the diagnostics. nedio-server-diagnostics-20250920-1610.zip

rebuilding onto inself sorted out the issue. back to green again. thanks!

Thank you for the swift reply @JorgeB Hmmm, last time I had the issue with disk 2 (back then I ended up replacing the HDD). See the thread here: https://forums.unraid.net/topic/186498-700-disk-error-xfs-dm-2-metadata-io-error Now before rebuilding it onto itself, I will indeed have to ckeck the cable. What should I be looking for with the cables replacement?

Hello, I am still having problems shutting down my server cleanly... Every update is a nightmare for me. I often end up with hard shutdown :( Like this time, where the UI stopped responding at some time and I could not ssh into the instance too. After the startup I see one of my disks is disabled. I did an XFS check xfs_repair -n /dev/mapper/md2p1 | tee /tmp/disk2_xfs_dryrun.log and then an XFS repair xfs_repair /dev/mapper/md2p1 | tee /tmp/disk2_xfs_repair.log of the disk2 which completed successfully While srray was started in Maintenance mode). The I stopped the array and started in normal mode hoping the disk will be back to normal, but it is stil shown as disabled and the contents is emulated. The disk seems ok. SMART test results are ok. The parity disk and the rest of the 3 disks of the array are shown as ok. Please see attached the diagnostics. Before I mess up irreversibly, wanted to consult with you guys, what are my next steps? nedio-server-diagnostics-20250526-1523.zip

managed to copy all my data from the old disk 🍀. have setup the backup ⛑️. a bit bitter after taste - why the hell my XFS file system got corrupted - still no answer 🤷‍♂️.

My array was messed up: Duration: 12 hours, 49 minutes, 35 seconds. Average speed: 86.6 MB/s Finding 487755460 errors All of that is supposed to be corrected now. Will start copying the data from the original disk to array.

I am considering to run a full parity check first on my Array. I somehow lost faith in it.. With writing corrections. Once I know my array is in a good shape - I will try to read from the original disk and write to the array. Please let me know if any part of this plan is a bad idea? 🙏 Then I will setup a proper 3-2-1 backup urgently...

nedio-server-diagnostics-20250209-1456.zip

I have started the array in maintenance mode (so that I can pass the encryption passphrase to unraid). Then I pressed the [mount] button next to the original disk in unassigned devices. Here is the log I see: Feb 9 14:46:56 nedio-server unassigned.devices: Mounting partition 'sdb1' at mountpoint '/mnt/disks/WD-WMC4N0F887V2'... Feb 9 14:46:56 nedio-server emhttpd: shcmd (4857): /usr/sbin/cryptsetup luksOpen '/dev/sdb1' 'WD-WMC4N0F887V2' Feb 9 14:46:58 nedio-server nginx: 2025/02/09 14:46:58 [alert] 11265#11265: worker process 90202 exited on signal 6 Feb 9 14:46:58 nedio-server unassigned.devices: Mount cmd: /sbin/mount -t 'xfs' -o rw,relatime '/dev/mapper/WD-WMC4N0F887V2' '/mnt/disks/WD-WMC4N0F887V2' Feb 9 14:46:58 nedio-server kernel: XFS (dm-8): Mounting V5 Filesystem f185aae9-b6c9-4eb8-b7dc-23ae79ee4411 Feb 9 14:46:58 nedio-server kernel: XFS (dm-8): Starting recovery (logdev: internal) Feb 9 14:46:59 nedio-server kernel: XFS (dm-8): Ending recovery (logdev: internal) Feb 9 14:47:00 nedio-server unassigned.devices: Successfully mounted '/dev/mapper/WD-WMC4N0F887V2' on '/mnt/disks/WD-WMC4N0F887V2'. Feb 9 14:47:00 nedio-server unassigned.devices: Device '/dev/sdb1' is not set to be shared. Feb 9 14:47:00 nedio-server nginx: 2025/02/09 14:47:00 [alert] 11265#11265: worker process 90521 exited on signal 6

ok, I have plugged in the original (3TB) disk. It shows up in the unassigned devices. Should I start the array (with the new 4TB disk) before posting the diagnostics? Should I try to mount the original disk outside of the array?

It mounted. The array has started. But there is a lot in the lost+found directory on disk3... 😨 Also the used and free size of the disk 3 shows only 49 GB used.. whereas I remember having 2+TB there... BTW: is there a way to know from the diagnostic how much data was on that HDD before XFS restore? Now, this disk (4TB) is the one that was rebuilt from the parity (when I replaced the originally failing HDD which was 3TB). On that 3TB disk I did not try to check/restore the xfs.. Do you think it is safe to plug the HDD-3TB back, start the array in maintenance mode and try to check the XFS there? Could it restore more data?

Done. Here are the last lines of the console output: Format log to cycle 2069032622. cache_purge: shake on cache 0x4d67c0 left 1 nodes!? cache_purge: shake on cache 0x4d67c0 left 1 nodes!? cache_zero_check: refcount is 1, not zero (node=0x14bf180fb010) XFS_REPAIR Summary Thu Feb 6 16:03:17 2025 Phase Start End Duration Phase 1: 02/06 16:00:31 02/06 16:00:31 Phase 2: 02/06 16:00:31 02/06 16:01:00 29 seconds Phase 3: 02/06 16:01:00 02/06 16:01:36 36 seconds Phase 4: 02/06 16:01:36 02/06 16:01:36 Phase 5: 02/06 16:01:36 02/06 16:01:37 1 second Phase 6: 02/06 16:01:37 02/06 16:01:37 Phase 7: 02/06 16:01:37 02/06 16:01:37 Total run time: 1 minute, 6 seconds done user@nedio-server:~# Thanks for helping me out JorgeB. I am a bit nervous, hoping I did not lose my data.. What's next?

I've run it. Got this: user@nedio-server:~# xfs_repair -v /dev/mapper/md3p1 Phase 1 - find and verify superblock... - block cache size set to 4590072 entries Phase 2 - using internal log - zero log... Log inconsistent (didn't find previous header) failed to find log head zero_log: cannot find log head/tail (xlog_find_tail=5) ERROR: The log head and/or tail cannot be discovered. Attempt to mount the filesystem to replay the log or use the -L option to destroy the log and attempt a repair.

ok, after rebooting the server I was able to start the array in Maintenance mode and clicked the Check File System button on disk 3. I have the unraid spinner running since 25 mins.. Here is the log output: Feb 6 14:28:53 nedio-server rc.nginx: Reloading Nginx server daemon configuration... Feb 6 14:28:54 nedio-server nginx: 2025/02/06 14:28:54 [alert] 11134#11134: worker process 25079 exited on signal 6 Feb 6 14:28:56 nedio-server nginx: 2025/02/06 14:28:56 [alert] 11134#11134: worker process 25363 exited on signal 6 Feb 6 14:28:57 nedio-server nginx: 2025/02/06 14:28:57 [alert] 11134#11134: worker process 25511 exited on signal 6 Feb 6 14:29:06 nedio-server nginx: 2025/02/06 14:29:06 [alert] 11134#11134: worker process 25549 exited on signal 6 Feb 6 14:29:07 nedio-server ool www[15128]: /usr/local/emhttp/plugins/dynamix/scripts/xfs_check 'start' '/dev/mapper/md3p1' 'ST4000NE001-2MA101_WS25BT0Z' '-n' Feb 6 14:29:07 nedio-server nginx: 2025/02/06 14:29:07 [alert] 11134#11134: worker process 26094 exited on signal 6 Feb 6 14:29:08 nedio-server nginx: 2025/02/06 14:29:08 [alert] 11134#11134: worker process 26284 exited on signal 6 ... Feb 6 14:31:02 nedio-server nginx: 2025/02/06 14:31:02 [alert] 11134#11134: worker process 33870 exited on signal 6 Feb 6 14:31:04 nedio-server nginx: 2025/02/06 14:31:04 [alert] 11134#11134: worker process 33950 exited on signal 6 Feb 6 14:31:06 nedio-server nginx: 2025/02/06 14:31:06 [alert] 11134#11134: worker process 34135 exited on signal 6 Feb 6 14:31:06 nedio-server publish: curl to arraymonitor failed Feb 6 14:31:08 nedio-server nginx: 2025/02/06 14:31:08 [alert] 11134#11134: worker process 34242 exited on signal 6 Feb 6 14:31:10 nedio-server nginx: 2025/02/06 14:31:10 [alert] 11134#11134: worker process 34321 exited on signal 6 ... Feb 6 14:34:48 nedio-server nginx: 2025/02/06 14:34:48 [alert] 11134#11134: worker process 48053 exited on signal 6 Feb 6 14:34:49 nedio-server nginx: 2025/02/06 14:34:49 [alert] 11134#11134: worker process 48151 exited on signal 6 Feb 6 14:39:14 nedio-server emhttpd: spinning down /dev/sdf Feb 6 14:39:14 nedio-server emhttpd: spinning down /dev/nvme0n1 Feb 6 14:39:15 nedio-server emhttpd: sdspin /dev/nvme0n1 down: 25 Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sdj Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sdh Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sdg Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sdd Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sde Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sdb Feb 6 14:43:48 nedio-server emhttpd: spinning down /dev/sdi Feb 6 14:44:17 nedio-server emhttpd: spinning down /dev/sdc Should I wait more? I do not see the xfs_check in htop.

The log that I am getting is: Feb 6 13:31:19 nedio-server emhttpd: cmdStart: already started Although on the main page the array looks like not started.. With dropdown boxes for disk selection. Should I restart the server maybe?

I can't start the array in maintenance mode... So currently I still have the previous HDD (3TB) which was mounted as Disk 3 - unplugged from the unraid server. And the new HDD (4TB) which I used trying to replace the HDD at Disk 3 of the array. I have precleared the 4TB HDD and then mounted in into the Disk3 slot. The array rebuild completed without errors. But now the array does not start (even if I check the maintenance mode). Please advise what should I do.

I got an error on one of my disks. I replaced it with a new one and rebuilt the array successfully. After one day I started getting this: Feb 6 00:00:33 nedio-server Docker Auto Update: Installing Updates for borgmatic immich mariadb Minio netdata nextcloud Feb 6 00:00:33 nedio-server kernel: XFS (dm-2): Metadata corruption detected at xfs_buf_ioend+0xa9/0x384, xfs_inode block 0xe4343410 xfs_inode_buf_verify Feb 6 00:00:33 nedio-server kernel: XFS (dm-2): Unmount and run xfs_repair Feb 6 00:00:33 nedio-server kernel: XFS (dm-2): First 128 bytes of corrupted metadata buffer: Feb 6 00:00:33 nedio-server kernel: 00000000: b4 cb fc 73 d1 4d 06 7b e3 4c 72 74 f9 72 b9 42 ...s.M.{.Lrt.r.B Feb 6 00:00:33 nedio-server kernel: 00000010: f1 19 35 c4 a7 ce b4 66 36 dc 1f f6 88 e1 ea ad ..5....f6....... Feb 6 00:00:33 nedio-server kernel: 00000020: 13 28 1e 6a a1 1c e5 46 6c 9f ff 94 7a 5d e1 12 .(.j...Fl...z].. Feb 6 00:00:33 nedio-server kernel: 00000030: 9b 97 0b 2c 47 92 6d 19 f3 84 fa d6 22 0b a3 3e ...,G.m....."..> Feb 6 00:00:33 nedio-server kernel: 00000040: 4b 51 0d 41 a3 8e 59 f9 09 13 ce 3f c1 a2 57 87 KQ.A..Y....?..W. Feb 6 00:00:33 nedio-server kernel: 00000050: f9 c6 30 b4 bc 30 65 b1 0f 86 aa 06 6f 35 57 cc ..0..0e.....o5W. Feb 6 00:00:33 nedio-server kernel: 00000060: c6 33 3c 33 9c 11 cc 2c d3 fc 97 10 61 ea 1f 7c .3<3...,....a..| Feb 6 00:00:33 nedio-server kernel: 00000070: bc 50 a2 fe bf b9 23 36 b3 60 bc 01 60 c0 c3 d7 .P....#6.`..`... Feb 6 00:00:33 nedio-server kernel: XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x52/0x74" at daddr 0xe4343410 len 32 error 117 Feb 6 00:00:34 nedio-server emhttpd: read SMART /dev/sdd Feb 6 00:00:41 nedio-server emhttpd: read SMART /dev/sde Feb 6 00:00:45 nedio-server kernel: XFS (dm-2): Metadata corruption detected at xfs_buf_ioend+0xa9/0x384, xfs_inode block 0x1467bfac8 xfs_inode_buf_verify Feb 6 00:00:45 nedio-server kernel: XFS (dm-2): Unmount and run xfs_repair I have stopped the array. Now I can't start it. Please help. My data is only partially backed up.. 😱 nedio-server-diagnostics-20250206-0928.zip