Apeiron

I think I found what I was looking for, and updates were now being reflected in the OS, but still not in real time. In the XML for the mounted filesystem, I changed to <cache mode='none'. ``` <filesystem type='mount' accessmode='passthrough'> <driver type='virtiofs' queue='1024'/> <binary path='/usr/libexec/virtiofsd' xattr='on'> <cache mode='none'/> <sandbox mode='chroot'/> </binary> <source dir='/mnt/user/development/'/> <target dir='development'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </filesystem> ``` Feels like my idea isn't going to work as I expected, so I just installed docker directly to my VM.

This is starting to become a real problem since I can't see the file changes being performed by Docker containers in real time. Most recently I did some npm package updates, and my package.json file was updated, but I can't review the changes. I realize my initial post was a lot of information, which isn't always the best for receiving help. I'll try to simplify this the best I can. I have an Unraid share that is mounted to a VM using virtiofs. I've also used a bind mount to mount that same share into a Docker container. When a file changes, such as Docker writing to a file, I cannot see these changes in the VM unless I reboot the VM. Docker containers do not have this issue and see any file changes immediately. My best guess is that this is just a quirk of virtiofs, and I need to find the appropriate configuration. Thank you for any advice. I'm going to have to switch back to local development until I resolve this issue.

I'm having an issue with data written by a docker container not being visible from a guest VM that has the data storage mounted via virtiofs until the guest is rebooted. Essentially, when docker writes data, I can't see that data in my guest OS, unless I reboot the guest. My goal is to be able to use Unraid's docker daemon to run my containers for development, with the Debian VM between my dev workstation and the Unraid server. My setup: I created a share called development. I created a new Debian VM that mounts this share using virtiofs to /development. I added this fstab entry to Debian: development /development virtiofs rw 0 0. I installed the docker cli to the Debian VM. I added the following entry to /boot/config/docker.cfg in Unraid: DOCKER_OPTS="--storage-driver=btrfs -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375" I use Remote-SSH in VS Code to connect to Debian from my development workstation. With this setup, I can: Manage Unraid docker containers from Debian and use compose Store repositories in the /development directory on Debian Connect to Debian from any workstation, and work off of the same dev environment This all works great, except for one issue: If I mount a docker volume into the development directory for persistent storage, the changes are not reflected in the Debian VM, unless I reboot (or umount and mount, I presume). This appears to be related to the virtiofs caching. For example, if I start up an nginx container, and mount a volume from the Unraid share directory to the nginx log directory, the log files are stored, but I can't see them from Debian at the time the files are written. Here is an example volume mount: volumes: - /mnt/user/development/compose-test/data:/var/log/nginxAfter nginx writes the log data to this directory/volume, you can see the Debian thinks /data is empty, while Unraid shows this directory containing files: Debian: ishmael@devserver:/development/compose-test$ ls -al drwxrwxr-x 1 ishmael ishmael 0 Sep 19 20:42 data -rw-rw-r-- 1 ishmael ishmael 242 Sep 19 20:46 docker-compose.yamlUnraid: root@solidsnake:/mnt/user/development/compose-test# ls -al drwxrwxr-x 1 ishmael 1000 38 Sep 19 20:46 data/ -rw-rw-r-- 1 ishmael 1000 242 Sep 19 20:46 docker-compose.yaml After rebooting Debian, it picks up the files in the data directory without issue. But, it doesn't catch any new writes until the next reboot. I thought this might be a permissions issue at first, but the files written by Docker into the data directory are set with the read flag for everyone. root@solidsnake:/mnt/user/development/compose-test/data# ls -al -rw-r--r-- 1 root root 2932 Sep 19 21:16 access.log -rw-r--r-- 1 root root 17879 Sep 19 21:31 error.log I've tried to modify the fstab entry in Debian to mount the directory without caching cache=none, but Debian failed to boot. I also tried adding additional flags for nofail and _netdev, but the mount failed. Example fstab troubleshooting: development /development virtiofs rw,cache=none,nofail,_netdev 0 0

I did the PCI/AER change first, and happy to report no errors after a couple weeks. I upgraded directly to 7.0.1 this evening. Everything (shares, dockers, VMs) seems to be working fine. No errors in the logs. Thanks for the help!

Working on it. Thanks for the response. I will update again as solved once the upgrade is complete, or if there were any issues.

I'm currently running 6.12.6, and am interested in upgrading to the latest version, 7.0.1. Namely for security updates and to ensure my plugins stay updated. I've read through all release notes from previous versions, and I have a few concerns with the upgrade that I'm hoping to get some guidance on. My system is pretty stable as-is, although every six months or so, unraid will become unresponsive and I have to force a reboot. My syslog server has been down for awhile, so I don't have logs from the crashes. I haven't had any issues other than that, and the system always comes back online, and parity checks are clean. I run a parity check every 30 to 60 days, and after a forced reboot. I have 7 disks using encrypted xfs, 1 parity drive, and 1 ssd cache using btrfs. I'm using Docker, and VMs, along with several plugins: Fix Common Problems, Nerd Tools, Open Files, SNMP, Unassigned Devices and User Scripts. Concerns/Questions: 1) The `macvlan` issue with Docker that came up in the 6.12 upgrade path. I am running the `macvlan` custom network type in my docker settings. If I remember correctly, I had issues trying to change off of `macvlan`, so I ended up changing it back and leaving it alone. I don't have errors related to this in my logs. Not sure if I should try again to change this setting or not. I'm currently using two network configurations for Docker, `bridge` and a custom `br0`. 2) I do get PCI / AER errors logged pretty consistently. The `severity` is always marked as `Corrected`. I researched this in the past and I believe I eventually came to conclusion that it can be ignored. ``` Feb 12 00:03:17 solidsnake kernel: pcieport 0000:00:01.3: AER: Corrected error received: 0000:22:00.0 Feb 12 00:03:17 solidsnake kernel: atlantic 0000:22:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID) Feb 12 00:03:17 solidsnake kernel: atlantic 0000:22:00.0: device [1d6a:07b1] error status/mask=00000001/0000a000 ``` 3) As with any upgrade, it seems the forum gets a lot of posts with people having problems. I've read through a number of these, and figured I probably don't have anything specific to worry about. However, what should I try to change or avoid to ensure the upgrade is successful? 4) Can I upgrade straight to 7.0.1, or should I upgrade to any intermediate versions in a specific order? Thank you. solidsnake-diagnostics-20250303-1520.zip

Reran the check with no option, which would not proceed. Did as suggested, ran it with -L. The repair appears successful, I stopped the array and started it back in normal mode. Disk 3 mounted successfully. Everything looks normal. Thank you both for your responses and help. Unless further checks are necessary, I'll marked this as solved.

I started the array in maintenance mode per the XFS instructions. Here is the output from the filesystem check -n on Disk 3. If I'm reading this correctly, it looks like a single file has an issue, and would be rebuilt if modify was allowed? Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being ignored because the -n option was used. Expect spurious inconsistencies which may be resolved by first mounting the filesystem to replay the log. - scan filesystem freespace and inode maps... sb_fdblocks 1489953900, counted 1503578523 - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 inode 15068100473 - bad extent starting block number 4503567550935200, offset 0 correcting nextents for inode 15068100473 bad data fork in inode 15068100473 would have cleared inode 15068100473 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 7 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 6 - agno = 5 entry "Phil Hine - Tantrum Magick.pdf" at block 0 offset 2672 in directory inode 15066855983 references free inode 15068100473 would clear inode number in entry at offset 2672... inode 15068100473 - bad extent starting block number 4503567550935200, offset 0 correcting nextents for inode 15068100473 bad data fork in inode 15068100473 would have cleared inode 15068100473 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "Phil Hine - Tantrum Magick.pdf" in directory inode 15066855983 points to free inode 15068100473, would junk entry bad hash table for directory inode 15066855983 (no data entry): would rebuild would rebuild directory inode 15066855983 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting.

The Problem: I experienced a lockup during an array stop procedure, unclean shutdown, and now an unmountable disk. There were no known issues prior to this error. What Happened: I was preparing for a manual parity check. I went through my normal procedure: shutdown running services, stop array, reboot and parity check. I shutdown all VMs and Containers, and clicked Stop Array, the system hung while unmounting the disks. I left the system overnight, and came back in the morning to no progress. I did a hard shutdown via power button, unplugged the server power, then booted back up. I started the array in maintenance mode, and issued a parity check. There was a notification for the unclean shutdown. The parity check completed ~24 hours later (normal) with no issues. I stopped the array, and started it back in normal mode. Disk 3 in my array was detected as being unmountable. I stopped the array, shutdown, unplugged power, and booted back up. Disk 3 was still unmountable. Stopped the array again. Pulled the diagnostic log, and now I'm here. syslog entry for disk 3: Jul 4 11:49:06 solidsnake emhttpd: mounting /mnt/disk3 Jul 4 11:49:06 solidsnake emhttpd: shcmd (115): mkdir -p /mnt/disk3 Jul 4 11:49:06 solidsnake emhttpd: shcmd (116): mount -t xfs -o noatime,nouuid /dev/mapper/md3p1 /mnt/disk3 Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Mounting V5 Filesystem Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Starting recovery (logdev: internal) Jul 4 11:49:06 solidsnake kernel: 00000000: 36 12 01 00 01 00 00 00 40 d4 b7 3c 84 88 ff ff 6.......@..<.... Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Internal error xfs_efi_item_recover at line 614 of file fs/xfs/xfs_extfree_item.c. Caller xlog_recover_process_intents+0x9c/0x25e [xfs] Jul 4 11:49:06 solidsnake kernel: CPU: 12 PID: 14282 Comm: mount Tainted: P O 6.1.64-Unraid #1 Jul 4 11:49:06 solidsnake kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi Ultimate, BIOS P3.10 04/25/2019 Jul 4 11:49:06 solidsnake kernel: Call Trace: Jul 4 11:49:06 solidsnake kernel: <TASK> Jul 4 11:49:06 solidsnake kernel: dump_stack_lvl+0x44/0x5c Jul 4 11:49:06 solidsnake kernel: xfs_corruption_error+0x63/0x83 [xfs] Jul 4 11:49:06 solidsnake kernel: ? xlog_recover_process_intents+0x9c/0x25e [xfs] Jul 4 11:49:06 solidsnake kernel: xfs_efi_item_recover+0x92/0x1a8 [xfs] Jul 4 11:49:06 solidsnake kernel: ? xlog_recover_process_intents+0x9c/0x25e [xfs] Jul 4 11:49:06 solidsnake kernel: xlog_recover_process_intents+0x9c/0x25e [xfs] Jul 4 11:49:06 solidsnake kernel: ? preempt_latency_start+0x2b/0x46 ### [PREVIOUS LINE REPEATED 1 TIMES] ### Jul 4 11:49:06 solidsnake kernel: xlog_recover_finish+0x2b/0x290 [xfs] Jul 4 11:49:06 solidsnake kernel: ? xfs_ag_resv_init+0x164/0x1af [xfs] Jul 4 11:49:06 solidsnake kernel: xfs_log_mount_finish+0x5a/0x111 [xfs] Jul 4 11:49:06 solidsnake kernel: xfs_mountfs+0x5c6/0x73b [xfs] Jul 4 11:49:06 solidsnake kernel: xfs_fs_fill_super+0x683/0x761 [xfs] Jul 4 11:49:06 solidsnake kernel: ? xfs_open_devices+0x184/0x184 [xfs] Jul 4 11:49:06 solidsnake kernel: get_tree_bdev+0x1d5/0x229 Jul 4 11:49:06 solidsnake kernel: vfs_get_tree+0x1c/0x8a Jul 4 11:49:06 solidsnake kernel: path_mount+0x62f/0x70d Jul 4 11:49:06 solidsnake kernel: do_mount+0x5c/0x8d Jul 4 11:49:06 solidsnake kernel: __do_sys_mount+0x100/0x12e Jul 4 11:49:06 solidsnake kernel: do_syscall_64+0x6b/0x81 Jul 4 11:49:06 solidsnake kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce Jul 4 11:49:06 solidsnake kernel: RIP: 0033:0x14780a0c9eea Jul 4 11:49:06 solidsnake kernel: Code: 48 8b 0d 31 1f 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fe 1e 0d 00 f7 d8 64 89 01 48 Jul 4 11:49:06 solidsnake kernel: RSP: 002b:00007ffcbd4cdd18 EFLAGS: 00000206 ORIG_RAX: 00000000000000a5 Jul 4 11:49:06 solidsnake kernel: RAX: ffffffffffffffda RBX: 000000000040f380 RCX: 000014780a0c9eea Jul 4 11:49:06 solidsnake kernel: RDX: 000000000040f5d0 RSI: 000000000040f650 RDI: 000000000040f5b0 Jul 4 11:49:06 solidsnake kernel: RBP: 0000000000000000 R08: 000000000040f610 R09: 0000000000000060 Jul 4 11:49:06 solidsnake kernel: R10: 0000000000000400 R11: 0000000000000206 R12: 000000000040f5b0 Jul 4 11:49:06 solidsnake kernel: R13: 000000000040f5d0 R14: 000014780a25efa4 R15: 000000000040f498 Jul 4 11:49:06 solidsnake kernel: </TASK> Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Corruption detected. Unmount and run xfs_repair Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Failed to recover intents Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Filesystem has been shut down due to log error (0x2). Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Please unmount the filesystem and rectify the problem(s). Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): Ending recovery (logdev: internal) Jul 4 11:49:06 solidsnake kernel: XFS (dm-2): log mount finish failed Jul 4 11:49:06 solidsnake root: mount: /mnt/disk3: mount(2) system call failed: Structure needs cleaning. Jul 4 11:49:06 solidsnake root: dmesg(1) may have more information after failed mount system call. Jul 4 11:49:06 solidsnake emhttpd: shcmd (116): exit status: 32 Jul 4 11:49:06 solidsnake emhttpd: /mnt/disk3 mount error: Unsupported or no file system Jul 4 11:49:06 solidsnake emhttpd: shcmd (117): rmdir /mnt/disk3 Next steps: Looking through some of the other posts, it would appear that I need to run a file system check. The logs mention running xfs_repair. Before I proceed with anything else, I wanted to confirm what exactly my next steps should be to maximize chances of recovering the disk intact. Thank you for any help you can provide. solidsnake-diagnostics-20240704-1153.zip

Thank you for the advice. Makes good sense. I always laughed jokingly at people who ran rm -rf / as root, and now I'm practically guilty of it. Lesson learned. Thanks.

I was performing data copy and cleanup using disk to disk transfers. I scrolled to the wrong command and mistakenly ran rm -rf /mnt/disk4 I immediately saw the error and killed the process. Using a disk usage comparison, and modified directory times, it looks like I deleted about 100GB of data from one of the folders on disk4. Reviewing the forums and FAQ, I saw info for ReiserFS and the "general help" link is broken on the FAQ. My data drive is using encrypted XFS. I fortunately did not delete anything critical and I have a backup. No other changes to the data filesystem have taken place. I'm looking for advice on next steps. Should I attempt a recovery operation, or restore from backup? I think restoring would guarantee 100% recovery, but a recovery is probably easier and faster. I'd be ok with a <100% recovery operation if most of the data could be recovered. I have experience recovering NTFS drives for windows using testdisk on linux, but never with XFS, or an encrypted drive. solidsnake-diagnostics-20220802-1026.zip

I finally received my second Sata controller. I used the same one I had previously, SI-PEX40139, so now I have 2 of these cards installed. I have 8 total hard drives, and split them to 4 drives on each card. Nothing is connected via the mobo Sata ports. I have (1) 1TB M.2 drive installed as cache. Lastly, the Unraid USB. After getting the server booted up, I added the new drive as a 7th data drive, and started the array in maintenance mode. I immediately started getting I/O buffer errors again. These started popping up without any load on the system. I have not started the formatting of the new drive yet. I've now replaced the PSU, all Sata cables, added an additional sata pcie card, added cooling fans and cleaned everything. The drive I am adding is new, and no errors were detected during the preclear. Jul 24 19:09:26 solidsnake emhttpd: Opening encrypted volumes... Jul 24 19:09:26 solidsnake kernel: Buffer I/O error on dev md7, logical block 0, async page read Suggestions for next steps or isolating the issue? Thank you. solidsnake-diagnostics-20220724-1921.zip

It seems I was premature. I'm not getting a CRC error, but a different error now. I had gotten full parity back with no apparent issues. So I swapped out a drive, and started the data rebuild process again. Several hours in, I received the following errors and then it stabilized and continued. The rebuild is still running right now. I've been looking into this error using keywords AMD-Vi and IO_PAGE_FAULT, and there are a lot of interesting results. I've seen some comments around IOMMU settings, PCIexpress lanes, BIOS upgrades and kernel bugs. Notably, several comments were found related to my system, Ryzen 2700X and ASMedia X470, having SATA issues with some configurations. Jul 16 01:00:04 solidsnake kernel: ahci 0000:03:00.1: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0x407d4db010608040 flags=0x0030] Jul 16 01:00:35 solidsnake kernel: ata2.00: exception Emask 0x0 SAct 0x80fe7fc0 SErr 0x0 action 0x6 frozen Followed by: Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/20:30:38:49:7d/02:00:f9:01:00/40 tag 6 ncq dma 278528 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/58:38:58:4b:7d/02:00:f9:01:00/40 tag 7 ncq dma 307200 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/10:40:b0:4d:7d/02:00:f9:01:00/40 tag 8 ncq dma 270336 in Jul 16 01:00:35 solidsnake kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/10:48:c0:4f:7d/02:00:f9:01:00/40 tag 9 ncq dma 270336 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/30:50:d0:51:7d/01:00:f9:01:00/40 tag 10 ncq dma 155648 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/f8:58:00:53:7d/01:00:f9:01:00/40 tag 11 ncq dma 258048 in Jul 16 01:00:35 solidsnake kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/08:60:f8:54:7d/01:00:f9:01:00/40 tag 12 ncq dma 135168 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/c0:68:00:56:7d/01:00:f9:01:00/40 tag 13 ncq dma 229376 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/40:70:c0:57:7d/01:00:f9:01:00/40 tag 14 ncq dma 163840 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/d8:88:00:59:7d/01:00:f9:01:00/40 tag 17 ncq dma 241664 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/a0:90:d8:5a:7d/01:00:f9:01:00/40 tag 18 ncq dma 212992 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/20:98:78:5c:7d/01:00:f9:01:00/40 tag 19 ncq dma 147456 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/e0:a0:98:5d:7d/00:00:f9:01:00/40 tag 20 ncq dma 114688 in Jul 16 01:00:35 solidsnake kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/d8:a8:78:5e:7d/02:00:f9:01:00/40 tag 21 ncq dma 372736 in Jul 16 01:00:35 solidsnake kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/68:b0:50:61:7d/02:00:f9:01:00/40 tag 22 ncq dma 315392 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/60:b8:b8:63:7d/02:00:f9:01:00/40 tag 23 ncq dma 311296 in Jul 16 01:00:35 solidsnake kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2.00: failed command: READ FPDMA QUEUED Jul 16 01:00:35 solidsnake kernel: ata2.00: cmd 60/20:f8:18:66:7d/03:00:f9:01:00/40 tag 31 ncq dma 409600 in Jul 16 01:00:35 solidsnake kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Jul 16 01:00:35 solidsnake kernel: ata2.00: status: { DRDY } Jul 16 01:00:35 solidsnake kernel: ata2: hard resetting link Jul 16 01:00:40 solidsnake kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Jul 16 01:00:40 solidsnake kernel: ata2.00: configured for UDMA/133 Jul 16 01:00:40 solidsnake kernel: ata2.00: device reported invalid CHS sector 0 Jul 16 01:00:40 solidsnake kernel: ata2.00: device reported invalid CHS sector 0 Jul 16 01:00:40 solidsnake kernel: ata2.00: device reported invalid CHS sector 0 Jul 16 01:00:40 solidsnake kernel: ata2.00: device reported invalid CHS sector 0 Jul 16 01:00:40 solidsnake kernel: ata2.00: device reported invalid CHS sector 0 Jul 16 01:00:40 solidsnake kernel: ata2: EH complete My next NAS build is definitely going to be a real server. Thoughts on these errors? Note: can't seem to unmark this as solved, so I can start a new thread if necessary. Thank you. solidsnake-diagnostics-20220716-0137.zip

The results won't be too scientific, as I made multiple changes at once. I have parity back, with the new 16TB drive. No other disk changes have been made yet. I ran SMART short self-tests on all disks which passed. It does not appear that the CRC error rate increased since the first instance of the problem. Looking at some historical logs, I noticed my UPS has been engaging for a second or two somewhat regularly (once every few days to once daily). I attribute the fluctuations to the local power service constantly adjusting loads during this summer heat. I don't have a correlating UPS log at the time of the read errors, so probably just an anecdote, but maybe not outside of realm of possibility that I had a power issue. Added several more cooling fans I had laying around and cleaned the case. The case wasn't very dirty, but hard disk temps would go ~41+ during parity checks. Temps dropped to 36 max during latest parity check. Split 2 SATA cables from the motherboard to PCI Syba SI-PEX40139 SATA card. Currently 4x SATA, 3x PCI, previously 6/1. Matched all SATA cables by manufacturer and connector Arranged cables to reduce cable touching on long parallel SATA runs Updated PSU - HX750 On boot up, a read check was executed on all data disks, while the parity was still disabled. After the read check succeeded, I started the parity rebuild, which was also successful. Next step is to replace one of the data drives with the old parity drive, and then add a new drive to the array. Finally, posting the diagnostics again, with new smart reports, in case anything may look awry. Thanks for the help. solidsnake-diagnostics-20220715-1541.zip

Ordered some stuff, should be here in a couple days. Will table this until the next update. Thank you for your response.