kstrike155

Here's the copy: I read that as: "if there's this much free space available on disks belonging to the share, allow writing". Looks like that was the problem as now it's fixed, thanks for the tip!

It's set to 390.5 GB. EDIT: OK I think what's happening here is that I don't have more than that amount of space available on any of my disks. The language for free space makes it seem like it's at the share level, but because Unraid writes entire files to disk it needs to ensure that there is that much free space available on *some* disk in the array. I reduced this to 10 GB for all shares and will try again.

I have been getting errors for days/weeks with my backups saying that there's no space left on device, when I have over 1 TB free. Debug key 372c1d4e-122a-4194-bf1c-a9189c65da2e Example: [02.01.2024 09:50:25][ℹ️][traefik] Stopping traefik... done! (took 13 seconds) [02.01.2024 09:50:38][ℹ️][traefik] Should NOT backup external volumes, sanitizing them... [02.01.2024 09:50:38][ℹ️][traefik] Calculated volumes to back up: /mnt/user/appdata/homelab/traefik-docker/certs, /mnt/user/appdata/homelab/traefik-docker/rules.yaml, /mnt/user/appdata/homelab/traefik-docker/letsencrypt [02.01.2024 09:50:38][ℹ️][traefik] Backing up traefik... [02.01.2024 09:50:38][❌][traefik] tar creation failed! Tar said: tar (child): /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/traefik.tar.gz: Cannot open: No space left on device; tar (child): Error is not recoverable: exiting now; tar: /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/traefik.tar.gz: Wrote only 4096 of 10240 bytes; tar: Child returned status 2; tar: Error is not recoverable: exiting now [02.01.2024 09:50:40][ℹ️][traefik] Starting traefik... (try #1) done! [02.01.2024 09:50:45][ℹ️][unifi] Stopping unifi... done! (took 6 seconds) [02.01.2024 09:50:51][ℹ️][unifi] Should NOT backup external volumes, sanitizing them... [02.01.2024 09:50:51][ℹ️][unifi] Calculated volumes to back up: /mnt/user/appdata/unifi, /mnt/user/appdata/homelab/unifi-docker/config.gateway.json [02.01.2024 09:50:51][ℹ️][unifi] Backing up unifi... [02.01.2024 09:50:51][❌][unifi] tar creation failed! Tar said: tar (child): /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/unifi.tar.gz: Cannot open: No space left on device; tar (child): Error is not recoverable: exiting now; tar: /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/unifi.tar.gz: Wrote only 4096 of 10240 bytes; tar: Child returned status 2; tar: Error is not recoverable: exiting now [02.01.2024 09:50:53][ℹ️][unifi] Starting unifi... (try #1) done! But looking at free space: root@homer:~# df -h /mnt/user/backups/homer/mnt/user/appdata/ Filesystem Size Used Avail Use% Mounted on shfs 11T 10T 1002G 92% /mnt/user

Confirmed, power supply was the issue!

OK, I tried YET ANOTHER drive (because the one I used to replace was an older drive). I purchased something brand new, and still got similar errors. Now I've replaced the power supply (with an old one but much higher quality...) and don't seem to be encountering these errors anymore, at least not in the last 30 minutes! Fingers crossed.

No splitters and I've got brand new SATA cables. I did have a random blip on one of my other disks where it disappeared from a slot after replacing the parity drive. Reboot solved it. Maybe power supply failure?

OK the slow parity rebuild was because a time machine backup had started. Back up to speed now at 175 MB/s. However, the errors do still concern me...

I added the new drive and seem to still be having some weirdness. I'm rebuilding parity right now and it was running pretty quickly at around 130 MB/s, but it has since slowed to around 20 MB/s and I see errors in the new disk log (below). New diags attached. Any ideas? I find it hard to believe that BOTH of the SATA controllers would be going bad (I'm running Ryzen 1600 on an ASRock B450 Pro4, so it has 4 direct CPU-connected SATA ports and then there's a separate ASMedia ASM1061 controller with 2 SATA ports). Nov 3 18:35:46 homer kernel: mdcmd (1): import 0 sdb 64 5860522532 0 HGST_HUS726060ALE610_NCG18YDL Nov 3 18:35:46 homer kernel: md: import disk0: (sdb) HGST_HUS726060ALE610_NCG18YDL size: 5860522532 Nov 3 18:35:46 homer emhttpd: read SMART /dev/sdb Nov 3 18:35:52 homer emhttpd: shcmd (397): echo 128 > /sys/block/sdb/queue/nr_requests Nov 3 18:36:33 homer kernel: ata2.00: exception Emask 0x50 SAct 0x7 SErr 0xb0802 action 0xe frozen Nov 3 18:36:33 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 18:36:33 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 18:36:33 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 18:36:33 homer kernel: ata2.00: cmd 61/40:00:68:49:09/05:00:00:00:00/40 tag 0 ncq dma 688128 out Nov 3 18:36:33 homer kernel: ata2.00: status: { DRDY } Nov 3 18:36:33 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 18:36:33 homer kernel: ata2.00: cmd 61/40:08:a8:4e:09/05:00:00:00:00/40 tag 1 ncq dma 688128 out Nov 3 18:36:33 homer kernel: ata2.00: status: { DRDY } Nov 3 18:36:33 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 18:36:33 homer kernel: ata2.00: cmd 61/40:10:e8:53:09/05:00:00:00:00/40 tag 2 ncq dma 688128 out Nov 3 18:36:33 homer kernel: ata2.00: status: { DRDY } Nov 3 18:36:33 homer kernel: ata2: hard resetting link Nov 3 18:36:39 homer kernel: ata2: found unknown device (class 0) Nov 3 18:36:43 homer kernel: ata2: softreset failed (device not ready) Nov 3 18:36:43 homer kernel: ata2: hard resetting link Nov 3 18:36:44 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 18:36:45 homer kernel: ata2.00: configured for UDMA/133 Nov 3 18:36:45 homer kernel: ata2: EH complete Nov 3 19:46:29 homer kernel: ata2.00: exception Emask 0x50 SAct 0x0 SErr 0xb0802 action 0xe frozen Nov 3 19:46:29 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 19:46:29 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 19:46:29 homer kernel: ata2.00: failed command: FLUSH CACHE EXT Nov 3 19:46:29 homer kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 27 Nov 3 19:46:29 homer kernel: ata2.00: status: { DRDY } Nov 3 19:46:29 homer kernel: ata2: hard resetting link Nov 3 19:46:35 homer kernel: ata2: found unknown device (class 0) Nov 3 19:46:39 homer kernel: ata2: softreset failed (device not ready) Nov 3 19:46:39 homer kernel: ata2: hard resetting link Nov 3 19:46:41 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 19:46:42 homer kernel: ata2.00: configured for UDMA/133 Nov 3 19:46:42 homer kernel: ata2.00: retrying FLUSH 0xea Emask 0x50 Nov 3 19:46:42 homer kernel: ata2: EH complete Nov 3 20:07:36 homer kernel: ata2.00: exception Emask 0x50 SAct 0x3c000000 SErr 0xb0802 action 0xe frozen Nov 3 20:07:36 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 20:07:36 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:d0:c8:ed:1d/05:00:2b:00:00/40 tag 26 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:d8:08:f3:1d/05:00:2b:00:00/40 tag 27 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:e0:48:f8:1d/05:00:2b:00:00/40 tag 28 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:e8:88:fd:1d/05:00:2b:00:00/40 tag 29 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2: hard resetting link Nov 3 20:07:42 homer kernel: ata2: found unknown device (class 0) Nov 3 20:07:46 homer kernel: ata2: softreset failed (device not ready) Nov 3 20:07:46 homer kernel: ata2: hard resetting link Nov 3 20:07:47 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 20:07:47 homer kernel: ata2.00: configured for UDMA/133 Nov 3 20:07:47 homer kernel: ata2: EH complete Nov 3 20:24:41 homer kernel: ata2.00: exception Emask 0x50 SAct 0x80800001 SErr 0xb0802 action 0xe frozen Nov 3 20:24:41 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 20:24:41 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 20:24:41 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:24:41 homer kernel: ata2.00: cmd 61/40:00:d0:74:08/05:00:31:00:00/40 tag 0 ncq dma 688128 out Nov 3 20:24:41 homer kernel: ata2.00: status: { DRDY } Nov 3 20:24:41 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:24:41 homer kernel: ata2.00: cmd 61/40:b8:50:6a:08/05:00:31:00:00/40 tag 23 ncq dma 688128 out Nov 3 20:24:41 homer kernel: ata2.00: status: { DRDY } Nov 3 20:24:41 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:24:41 homer kernel: ata2.00: cmd 61/40:f8:90:6f:08/05:00:31:00:00/40 tag 31 ncq dma 688128 out Nov 3 20:24:41 homer kernel: ata2.00: status: { DRDY } Nov 3 20:24:41 homer kernel: ata2: hard resetting link Nov 3 20:24:47 homer kernel: ata2: found unknown device (class 0) Nov 3 20:24:51 homer kernel: ata2: softreset failed (device not ready) Nov 3 20:24:51 homer kernel: ata2: hard resetting link Nov 3 20:24:53 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 20:24:53 homer kernel: ata2.00: configured for UDMA/133 Nov 3 20:24:53 homer kernel: ata2: EH complete homer-diagnostics-20231103-2045.zip

Yes, the errors are related to the errors I added in my original post. I'm trying to run an extended SMART test and I can't. Sometimes if I unplug and replug the drive I can get it to do something. Other times no. Right now I try looking at drive attributes and they are blank, and try running a test and it doesn't seem to be running. Given I have swapped cables and ports, I'm calling this one basically dead. New drive comes today so we'll see if that has any similar SATA errors.

Attached homer-diagnostics-20231103-1029.zip

My parity drive keeps moving into failed state. I replaced the cable and also moved to a different SATA port on a different controller. I also get failures when trying to format the drive using Unassigned Devices. SMART is OK, but looking at the disk logs, I see the following which indicate to me that maybe the SATA controller on the drive itself has failed? Nov 2 19:33:44 homer kernel: ata2.00: failed command: READ DMA EXT Nov 2 19:33:44 homer kernel: ata2.00: cmd 25/00:08:00:be:c0/00:00:d1:01:00/e0 tag 20 dma 4096 in Nov 2 19:33:44 homer kernel: ata2.00: status: { DRDY DF ERR } Nov 2 19:33:44 homer kernel: ata2.00: error: { ABRT } Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: configured for UDMA/133 (device error ignored) Nov 2 19:33:44 homer kernel: ata2: EH complete Nov 2 19:33:44 homer kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Nov 2 19:33:44 homer kernel: ata2.00: irq_stat 0x40000001 Nov 2 19:33:44 homer kernel: ata2.00: failed command: READ DMA EXT Nov 2 19:33:44 homer kernel: ata2.00: cmd 25/00:08:00:be:c0/00:00:d1:01:00/e0 tag 5 dma 4096 in Nov 2 19:33:44 homer kernel: ata2.00: status: { DRDY DF ERR } Nov 2 19:33:44 homer kernel: ata2.00: error: { ABRT } Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: configured for UDMA/133 (device error ignored) Nov 2 19:33:44 homer kernel: ata2: EH complete Nov 2 19:33:44 homer kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Nov 2 19:33:44 homer kernel: ata2.00: irq_stat 0x40000001 Nov 2 19:33:44 homer kernel: ata2.00: failed command: READ DMA EXT Nov 2 19:33:44 homer kernel: ata2.00: cmd 25/00:08:00:be:c0/00:00:d1:01:00/e0 tag 12 dma 4096 in Nov 2 19:33:44 homer kernel: ata2.00: status: { DRDY DF ERR } Nov 2 19:33:44 homer kernel: ata2.00: error: { ABRT } Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: configured for UDMA/133 (device error ignored) Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 Sense Key : 0x5 [current] Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 ASC=0x21 ASCQ=0x4 Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00 Nov 2 19:33:44 homer kernel: I/O error, dev sde, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Nov 2 19:33:44 homer kernel: Buffer I/O error on dev sde, logical block 976754624, async page read Nov 2 19:33:44 homer kernel: ata2: EH complete

My syslog is in the first post of the thread and doesn't show much. I switched to the IPvlan network driver now so hopefully that resolves things.

Looks like it is hanging again. It's not a TOTAL hard lock, I am able to ping the machine. I was also able to (very slowly) SSH into the machine but I can't run anything. After a few minutes I can't SSH to it at all now. It's almost as if the system is TOTALLY overloaded. Unfortunately I am out of town and am not able to reset the machine remotely now. Can still ping, though! Can also run nc commands to check open ports and they seem to be responding as expected, so I don't think the machine is in kernel panic.

Interesting you mention that. I am running the @mbentley docker-timemachine image with a custom MAC address and IP so that it can show up on the network using a dedicated IP. I just happened to disable that container on Saturday because I'm running array-only with spinning disks right now (while my SSD is being replaced) and need all of the drive speed I can get for my Docker containers. Will let it run for a while and see if it's stable (no other containers are using the macvlan driver).

That didn't seem to resolve the problem either. I've been running without the SSD for a few days now (running everything off the array, no cache: woof). System just came to a halt not too long ago. I didn't do a full memtest yet, only 20%, but it came back clean thus far. It seems to happen when under load, maybe a power supply issue? I read a pretty interesting thread on Reddit, it looks like this is a common issue with 6.12. I uninstalled the unassigned devices plugin as mentioned in that thread just for giggles.