kstrike155

I put this in my /boot/config/go file to download and install the newest version at every reboot: # Docker buildx mkdir -p /usr/libexec/docker/cli-plugins/ BUILDX_VERSION=$(curl -s https://api.github.com/repos/docker/buildx/releases/latest | grep 'tag_name' | cut -d\" -f4) curl -L https://github.com/docker/buildx/releases/download/${BUILDX_VERSION}/buildx-${BUILDX_VERSION}.linux-amd64 -o /usr/libexec/docker/cli-plugins/docker-buildx chmod +x /usr/libexec/docker/cli-plugins/docker-buildx

Here's the copy: I read that as: "if there's this much free space available on disks belonging to the share, allow writing". Looks like that was the problem as now it's fixed, thanks for the tip!

It's set to 390.5 GB. EDIT: OK I think what's happening here is that I don't have more than that amount of space available on any of my disks. The language for free space makes it seem like it's at the share level, but because Unraid writes entire files to disk it needs to ensure that there is that much free space available on *some* disk in the array. I reduced this to 10 GB for all shares and will try again.

I have been getting errors for days/weeks with my backups saying that there's no space left on device, when I have over 1 TB free. Debug key 372c1d4e-122a-4194-bf1c-a9189c65da2e Example: [02.01.2024 09:50:25][ℹ️][traefik] Stopping traefik... done! (took 13 seconds) [02.01.2024 09:50:38][ℹ️][traefik] Should NOT backup external volumes, sanitizing them... [02.01.2024 09:50:38][ℹ️][traefik] Calculated volumes to back up: /mnt/user/appdata/homelab/traefik-docker/certs, /mnt/user/appdata/homelab/traefik-docker/rules.yaml, /mnt/user/appdata/homelab/traefik-docker/letsencrypt [02.01.2024 09:50:38][ℹ️][traefik] Backing up traefik... [02.01.2024 09:50:38][❌][traefik] tar creation failed! Tar said: tar (child): /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/traefik.tar.gz: Cannot open: No space left on device; tar (child): Error is not recoverable: exiting now; tar: /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/traefik.tar.gz: Wrote only 4096 of 10240 bytes; tar: Child returned status 2; tar: Error is not recoverable: exiting now [02.01.2024 09:50:40][ℹ️][traefik] Starting traefik... (try #1) done! [02.01.2024 09:50:45][ℹ️][unifi] Stopping unifi... done! (took 6 seconds) [02.01.2024 09:50:51][ℹ️][unifi] Should NOT backup external volumes, sanitizing them... [02.01.2024 09:50:51][ℹ️][unifi] Calculated volumes to back up: /mnt/user/appdata/unifi, /mnt/user/appdata/homelab/unifi-docker/config.gateway.json [02.01.2024 09:50:51][ℹ️][unifi] Backing up unifi... [02.01.2024 09:50:51][❌][unifi] tar creation failed! Tar said: tar (child): /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/unifi.tar.gz: Cannot open: No space left on device; tar (child): Error is not recoverable: exiting now; tar: /mnt/user/backups/homer/mnt/user/appdata/ab_20240102_094256/unifi.tar.gz: Wrote only 4096 of 10240 bytes; tar: Child returned status 2; tar: Error is not recoverable: exiting now [02.01.2024 09:50:53][ℹ️][unifi] Starting unifi... (try #1) done! But looking at free space: root@homer:~# df -h /mnt/user/backups/homer/mnt/user/appdata/ Filesystem Size Used Avail Use% Mounted on shfs 11T 10T 1002G 92% /mnt/user

Confirmed, power supply was the issue!

OK, I tried YET ANOTHER drive (because the one I used to replace was an older drive). I purchased something brand new, and still got similar errors. Now I've replaced the power supply (with an old one but much higher quality...) and don't seem to be encountering these errors anymore, at least not in the last 30 minutes! Fingers crossed.

No splitters and I've got brand new SATA cables. I did have a random blip on one of my other disks where it disappeared from a slot after replacing the parity drive. Reboot solved it. Maybe power supply failure?

OK the slow parity rebuild was because a time machine backup had started. Back up to speed now at 175 MB/s. However, the errors do still concern me...

I added the new drive and seem to still be having some weirdness. I'm rebuilding parity right now and it was running pretty quickly at around 130 MB/s, but it has since slowed to around 20 MB/s and I see errors in the new disk log (below). New diags attached. Any ideas? I find it hard to believe that BOTH of the SATA controllers would be going bad (I'm running Ryzen 1600 on an ASRock B450 Pro4, so it has 4 direct CPU-connected SATA ports and then there's a separate ASMedia ASM1061 controller with 2 SATA ports). Nov 3 18:35:46 homer kernel: mdcmd (1): import 0 sdb 64 5860522532 0 HGST_HUS726060ALE610_NCG18YDL Nov 3 18:35:46 homer kernel: md: import disk0: (sdb) HGST_HUS726060ALE610_NCG18YDL size: 5860522532 Nov 3 18:35:46 homer emhttpd: read SMART /dev/sdb Nov 3 18:35:52 homer emhttpd: shcmd (397): echo 128 > /sys/block/sdb/queue/nr_requests Nov 3 18:36:33 homer kernel: ata2.00: exception Emask 0x50 SAct 0x7 SErr 0xb0802 action 0xe frozen Nov 3 18:36:33 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 18:36:33 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 18:36:33 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 18:36:33 homer kernel: ata2.00: cmd 61/40:00:68:49:09/05:00:00:00:00/40 tag 0 ncq dma 688128 out Nov 3 18:36:33 homer kernel: ata2.00: status: { DRDY } Nov 3 18:36:33 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 18:36:33 homer kernel: ata2.00: cmd 61/40:08:a8:4e:09/05:00:00:00:00/40 tag 1 ncq dma 688128 out Nov 3 18:36:33 homer kernel: ata2.00: status: { DRDY } Nov 3 18:36:33 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 18:36:33 homer kernel: ata2.00: cmd 61/40:10:e8:53:09/05:00:00:00:00/40 tag 2 ncq dma 688128 out Nov 3 18:36:33 homer kernel: ata2.00: status: { DRDY } Nov 3 18:36:33 homer kernel: ata2: hard resetting link Nov 3 18:36:39 homer kernel: ata2: found unknown device (class 0) Nov 3 18:36:43 homer kernel: ata2: softreset failed (device not ready) Nov 3 18:36:43 homer kernel: ata2: hard resetting link Nov 3 18:36:44 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 18:36:45 homer kernel: ata2.00: configured for UDMA/133 Nov 3 18:36:45 homer kernel: ata2: EH complete Nov 3 19:46:29 homer kernel: ata2.00: exception Emask 0x50 SAct 0x0 SErr 0xb0802 action 0xe frozen Nov 3 19:46:29 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 19:46:29 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 19:46:29 homer kernel: ata2.00: failed command: FLUSH CACHE EXT Nov 3 19:46:29 homer kernel: ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 27 Nov 3 19:46:29 homer kernel: ata2.00: status: { DRDY } Nov 3 19:46:29 homer kernel: ata2: hard resetting link Nov 3 19:46:35 homer kernel: ata2: found unknown device (class 0) Nov 3 19:46:39 homer kernel: ata2: softreset failed (device not ready) Nov 3 19:46:39 homer kernel: ata2: hard resetting link Nov 3 19:46:41 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 19:46:42 homer kernel: ata2.00: configured for UDMA/133 Nov 3 19:46:42 homer kernel: ata2.00: retrying FLUSH 0xea Emask 0x50 Nov 3 19:46:42 homer kernel: ata2: EH complete Nov 3 20:07:36 homer kernel: ata2.00: exception Emask 0x50 SAct 0x3c000000 SErr 0xb0802 action 0xe frozen Nov 3 20:07:36 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 20:07:36 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:d0:c8:ed:1d/05:00:2b:00:00/40 tag 26 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:d8:08:f3:1d/05:00:2b:00:00/40 tag 27 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:e0:48:f8:1d/05:00:2b:00:00/40 tag 28 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:07:36 homer kernel: ata2.00: cmd 61/40:e8:88:fd:1d/05:00:2b:00:00/40 tag 29 ncq dma 688128 out Nov 3 20:07:36 homer kernel: ata2.00: status: { DRDY } Nov 3 20:07:36 homer kernel: ata2: hard resetting link Nov 3 20:07:42 homer kernel: ata2: found unknown device (class 0) Nov 3 20:07:46 homer kernel: ata2: softreset failed (device not ready) Nov 3 20:07:46 homer kernel: ata2: hard resetting link Nov 3 20:07:47 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 20:07:47 homer kernel: ata2.00: configured for UDMA/133 Nov 3 20:07:47 homer kernel: ata2: EH complete Nov 3 20:24:41 homer kernel: ata2.00: exception Emask 0x50 SAct 0x80800001 SErr 0xb0802 action 0xe frozen Nov 3 20:24:41 homer kernel: ata2.00: irq_stat 0x00400000, PHY RDY changed Nov 3 20:24:41 homer kernel: ata2: SError: { RecovComm HostInt PHYRdyChg PHYInt 10B8B } Nov 3 20:24:41 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:24:41 homer kernel: ata2.00: cmd 61/40:00:d0:74:08/05:00:31:00:00/40 tag 0 ncq dma 688128 out Nov 3 20:24:41 homer kernel: ata2.00: status: { DRDY } Nov 3 20:24:41 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:24:41 homer kernel: ata2.00: cmd 61/40:b8:50:6a:08/05:00:31:00:00/40 tag 23 ncq dma 688128 out Nov 3 20:24:41 homer kernel: ata2.00: status: { DRDY } Nov 3 20:24:41 homer kernel: ata2.00: failed command: WRITE FPDMA QUEUED Nov 3 20:24:41 homer kernel: ata2.00: cmd 61/40:f8:90:6f:08/05:00:31:00:00/40 tag 31 ncq dma 688128 out Nov 3 20:24:41 homer kernel: ata2.00: status: { DRDY } Nov 3 20:24:41 homer kernel: ata2: hard resetting link Nov 3 20:24:47 homer kernel: ata2: found unknown device (class 0) Nov 3 20:24:51 homer kernel: ata2: softreset failed (device not ready) Nov 3 20:24:51 homer kernel: ata2: hard resetting link Nov 3 20:24:53 homer kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Nov 3 20:24:53 homer kernel: ata2.00: configured for UDMA/133 Nov 3 20:24:53 homer kernel: ata2: EH complete homer-diagnostics-20231103-2045.zip

Yes, the errors are related to the errors I added in my original post. I'm trying to run an extended SMART test and I can't. Sometimes if I unplug and replug the drive I can get it to do something. Other times no. Right now I try looking at drive attributes and they are blank, and try running a test and it doesn't seem to be running. Given I have swapped cables and ports, I'm calling this one basically dead. New drive comes today so we'll see if that has any similar SATA errors.

Attached homer-diagnostics-20231103-1029.zip

My parity drive keeps moving into failed state. I replaced the cable and also moved to a different SATA port on a different controller. I also get failures when trying to format the drive using Unassigned Devices. SMART is OK, but looking at the disk logs, I see the following which indicate to me that maybe the SATA controller on the drive itself has failed? Nov 2 19:33:44 homer kernel: ata2.00: failed command: READ DMA EXT Nov 2 19:33:44 homer kernel: ata2.00: cmd 25/00:08:00:be:c0/00:00:d1:01:00/e0 tag 20 dma 4096 in Nov 2 19:33:44 homer kernel: ata2.00: status: { DRDY DF ERR } Nov 2 19:33:44 homer kernel: ata2.00: error: { ABRT } Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: configured for UDMA/133 (device error ignored) Nov 2 19:33:44 homer kernel: ata2: EH complete Nov 2 19:33:44 homer kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Nov 2 19:33:44 homer kernel: ata2.00: irq_stat 0x40000001 Nov 2 19:33:44 homer kernel: ata2.00: failed command: READ DMA EXT Nov 2 19:33:44 homer kernel: ata2.00: cmd 25/00:08:00:be:c0/00:00:d1:01:00/e0 tag 5 dma 4096 in Nov 2 19:33:44 homer kernel: ata2.00: status: { DRDY DF ERR } Nov 2 19:33:44 homer kernel: ata2.00: error: { ABRT } Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: configured for UDMA/133 (device error ignored) Nov 2 19:33:44 homer kernel: ata2: EH complete Nov 2 19:33:44 homer kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Nov 2 19:33:44 homer kernel: ata2.00: irq_stat 0x40000001 Nov 2 19:33:44 homer kernel: ata2.00: failed command: READ DMA EXT Nov 2 19:33:44 homer kernel: ata2.00: cmd 25/00:08:00:be:c0/00:00:d1:01:00/e0 tag 12 dma 4096 in Nov 2 19:33:44 homer kernel: ata2.00: status: { DRDY DF ERR } Nov 2 19:33:44 homer kernel: ata2.00: error: { ABRT } Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: failed to enable AA (error_mask=0x1) Nov 2 19:33:44 homer kernel: ata2.00: configured for UDMA/133 (device error ignored) Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 Sense Key : 0x5 [current] Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 ASC=0x21 ASCQ=0x4 Nov 2 19:33:44 homer kernel: sd 2:0:0:0: [sde] tag#12 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00 Nov 2 19:33:44 homer kernel: I/O error, dev sde, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Nov 2 19:33:44 homer kernel: Buffer I/O error on dev sde, logical block 976754624, async page read Nov 2 19:33:44 homer kernel: ata2: EH complete

My syslog is in the first post of the thread and doesn't show much. I switched to the IPvlan network driver now so hopefully that resolves things.

Looks like it is hanging again. It's not a TOTAL hard lock, I am able to ping the machine. I was also able to (very slowly) SSH into the machine but I can't run anything. After a few minutes I can't SSH to it at all now. It's almost as if the system is TOTALLY overloaded. Unfortunately I am out of town and am not able to reset the machine remotely now. Can still ping, though! Can also run nc commands to check open ports and they seem to be responding as expected, so I don't think the machine is in kernel panic.

Interesting you mention that. I am running the @mbentley docker-timemachine image with a custom MAC address and IP so that it can show up on the network using a dedicated IP. I just happened to disable that container on Saturday because I'm running array-only with spinning disks right now (while my SSD is being replaced) and need all of the drive speed I can get for my Docker containers. Will let it run for a while and see if it's stable (no other containers are using the macvlan driver).

That didn't seem to resolve the problem either. I've been running without the SSD for a few days now (running everything off the array, no cache: woof). System just came to a halt not too long ago. I didn't do a full memtest yet, only 20%, but it came back clean thus far. It seems to happen when under load, maybe a power supply issue? I read a pretty interesting thread on Reddit, it looks like this is a common issue with 6.12. I uninstalled the unassigned devices plugin as mentioned in that thread just for giggles.

Welp, it happened again. This time I noticed that the HDD activity light was blinking when Unraid started to hang. But it wasn't blinking like normal activity; it was a steady/consistent blink on/off about 2-3 times per second. I then looked inside my case and saw the LED blinking on my cache drive which is an HP EX920 M.2 NVMe SSD (see here for a similar video from someone else). Surely this must mean something is wrong with my cache drive and Unraid isn't handling it well?

I already had the C6 update in place from previously, so really the only change I made was removing the GPU and setting Power Supply Idle Control to typical. That BIOS option is specific to AMD, so won't help you, sorry. With that said, so far so good, time will tell if this remains stable.

I went to the BIOS and Global C-State was set to Auto, but I actually still have my c6-disable flag on during boot: /usr/local/sbin/zenstates --c6-disable What I did notice is that Power Supply Idle Control was set to Auto as well. In my research it sounds like that may actually be what's causing system hangs, so I set that to Typical and left the C-State setting enabled. I also pulled my GPU. Hopefully these two things resolve the issue!

Actually, I reset the BIOS during my troubleshooting in booting off the USB after the upgrade. I bet this is a C-State thing, will check and reply with findings.

General system specs are: AMD Ryzen 5 1600 ASRock B450 Pro4 2x8 GB DDR4-2400 ECC RAM 3x + 1x parity drives (all 4TB, mix of WD Red and Blue) and a single NVMe cache disk Been running Unraid for years without much issue, but recently my system has been becoming completely unresponsive: no console, no SSH, no nothing. It happens probably about once a week on average. I have to hard reboot using the power button. I've attached diagnostics from shortly after a reboot. I did have a heck of a time upgrading to 6.12.4 and has to rebuild my USB stick. Then the My Servers recovery didn't image the stick properly, so I had to start over with my own backup. Not sure if it's related, though it seems like it might have started happening after the upgrade. I added an old video card to the machine during my debugging of the upgrade (so I could use a monitor), and left that in the machine. I am going to try removing it to see if it fixes anything, but wondering if there are any other ideas? I also noticed I've been receiving some errors on my parity disk which I will replace, but I would hope disk failure wouldn't bring my whole server down. Here's syslog output from before and after the reboot (full syslog after reboot in the diagnostics); nothing of note really: Sep 23 23:27:48 homer emhttpd: read SMART /dev/sdd Sep 23 23:27:57 homer emhttpd: read SMART /dev/sdc Sep 24 00:01:33 homer root: /mnt/cache: 858.9 GiB (922254512128 bytes) trimmed on /dev/nvme0n1p1 Sep 24 00:28:06 homer emhttpd: spinning down /dev/sdc Sep 24 00:39:56 homer sshd[16907]: Received disconnect from 192.168.1.228 port 55478:11: Normal Shutdown Sep 24 00:39:56 homer sshd[16907]: Disconnected from user root 192.168.1.228 port 55478 Sep 24 00:55:24 homer emhttpd: spinning down /dev/sdd Sep 24 02:02:56 homer emhttpd: read SMART /dev/sdc =================================== REBOOT ======================================== Sep 24 07:42:55 homer kernel: Linux version 6.1.49-Unraid (root@Develop-612) (gcc (GCC) 12.2.0, GNU ld version 2.40-slack151) #1 SMP PREEMPT_DYNAMIC Wed Aug 30 09:42:35 PDT 2023 Sep 24 07:42:55 homer kernel: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot Sep 24 07:42:55 homer kernel: BIOS-provided physical RAM map: Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009d3ff] usable Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x000000000009d400-0x000000000009ffff] reserved Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000009d01fff] usable Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x0000000009d02000-0x0000000009ffffff] reserved Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x000000000a000000-0x000000000a1fffff] usable Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x000000000a200000-0x000000000a20afff] ACPI NVS Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x000000000a20b000-0x000000000affffff] usable Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x000000000b000000-0x000000000b01ffff] reserved Sep 24 07:42:55 homer kernel: BIOS-e820: [mem 0x000000000b020000-0x00000000db1c8fff] usable homer-diagnostics-20230924-0800.zip

I think Jorgen's approach is probably much better if you are running a Mac VM, with the caveat that you wouldn't be able to mount that photo library remotely to use on a physical Mac (though iCloud Photo Library would take care of this; you could also potentially create a share in the Mac VM to then open the library on another mac). While my approach is "working", I can't really recommend it as it is SLOOOOOOW.

I'm in the process of doing this right now. The steps that I took were: Create a share dedicated to my Photos library. Mount the share using your Mac. Using the Mac Disk Utility, create a disk image stored in the share. I chose to create a 1TB sparse bundle. Mount the sparsebundle on your Mac as a volume. Hold down the option key and open Photos. Choose Create New... to create a new Photos library, and choose your sparsbundle mounted volume. (optional) Head to Photos preferences and choose "Use as System Photo Library" in order to enable iCloud Photos on this library.