May 2, 20242 yr Community Expert TLDR; Having a lot of docker issues after updating beyond 6.12.6. Is it safe to downgrade to unraid 6.12.6 with the recent docker vulnerability that was patched after 6.12.6? I have multiple containers exposed to the internet through a reverse proxy. Thinking about downgrading from 6.12.10 to 6.12.6 as docker and the unraid webui has become almost unusable after updating beyond 6.12.6. It's like docker itself has become very slow as updating or restarting a container now takes about 15-30 minutes. And the containers seems to run more slowly as well. Now I'm having problems with emby loading slow an movies stops to buffer as well and I have never had playback issues before. Emby is on the same version as it was on 6.12.6. My Audiobookshelf container was litteraly unusable on their latest version on 6.12.10 and I had to downgrade that to an older version which works. Everything was super slow, and wasn't able to load most times. But the newst version of audiobookshelf shoud run just fine though slower than version 2.3.3 which I had to downgrade to and even that is slower then it was before. And updating or restarting a container takes up to 15-30 min as I said and while doing that the unraid webui is unreachable, it's just loading. More often than not when restarting a container it comes back with an exceution error "server error", but it does restart anyway when it finanlly decides to. Uploading my diags here from a couple hours ago when my emby and audiobookshelf container was unracahble again, it was just loading. Tried to restart emby but i gave up after about 30 min and decided to stop the array and start it again which also took about 15-20 more min. I guess restarting the docker service would have done the same thing getting my containers up again. And when starting the arrray wasn't all containers started simultaneously before? Now they start one by one from the top and takes ablout 30 min that too before all are started. Before everything was up in about 5 min after starting the array. Can this has something to do with the appdata backup plugin? I have that set to stop, backup and start for each container. I can also add that I was using docker directory, but tried to switch to image again just to see if that would solve the issues, but it didn't. I've also changed all the hardware. But that didn't help. (i have two servers, but one isn't in use right now so I just swapped the all the drives to the other server.) My next step before downgrading is to reboot in safemode to rule out any plugins. Docker runs in safe mode right? Hoping to get some help diagnosing this and to tell me it safe to downgrade to 6.12.6 again if I can't figure this out. tower-diagnostics-20240502-1116.zip Edited May 2, 20242 yr by strike
May 14, 20242 yr Author Community Expert Well it's not caused by any plugins. The issue still remains when running in safe mode. After the array starts it takes almost exactly an hour from when the first container is starting until the last container is started. AN HOUR to start 22 containers. This used to take about 5 min.. Everything docker related is running ridiculously slow. I stopped the appdata backup yesterday as it was not finished after 5 HOURS.. Is it safe to downgrade to 6.12.6? If I were to start from scratch and redo the flash drive, which files do I need to copy over to keep all my settings? Edited May 14, 20242 yr by strike
May 15, 20242 yr Community Expert If you create a new docker image and add a couple of new containers, differentfrom the ones you are using, and without restoring the old ones initially, do you see the same?
May 16, 20242 yr Author Community Expert On 5/15/2024 at 9:23 AM, JorgeB said: If you create a new docker image and add a couple of new containers, differentfrom the ones you are using, and without restoring the old ones initially, do you see the same? So I tested this. Nuked my docker image and installed a few containers I have never installed before. Worked like normal. Restart/stop of a container took about 30 seconds and no execution error either. Nuked my docker image again and started installing all my previous containers from CA. The first 10-12 installed like normal, then it started acting up again. Not finishing pulling all the layers and complaining about not finding the image locally so it started pulling all the layers again, See attached images. Took me about 2 hours to install all the containers. Normally it would take less then half of that time IIRC. It downloads fast but the rest takes more time then it used to especially when it can't seem to finish pulling all the layers the first time. Then when everything was up an running again I restared one container just to check and yup, same behavior as I have struggled with ever since updating from 6.12.6. Took about 5 min to restart and I was greeted with an nice execution error in the end. I just love my life... Waiting for some new hardware which will arrive next week. Almost up to date CPU and MB. Hope that solves it. If not I'm downgrading or staring over with a new flash drive. Edited May 17, 20242 yr by strike
May 17, 20242 yr Author Community Expert It's on an ssd pool. And this is just getting worse. Now mover is running very slowly as well. Tried to move some stuff last night, gave up after 2 hours as it just had moved about 100MB. Tried to stop the array, it wouidn't stop so I had to restart in stead. Booted up again and started all containers, started the mover again. Now after 8 hours is has moved another 100MB and it's still running apparantly. Starting to think it must be something wrong with my HBA or something. Or it could be that windows rebooted and aborted my manually started mover script. I have a script that pauses all my downloads, starts the mover and starts all my downloads again when it's finished. Looks like everythings is working as normal when I move stuff manually. Nothing wrong with the speed there. No error is the syslog either. Will try to move some stuff off my cache pool manually as well just to check that too. The test earlier was from array disk to array disk. If my test goes well, I have no clue what the issue with the mover is. Maybe start it manually without my script and see how it behaves.
May 17, 20242 yr Author Community Expert Solution I think I figured it out. There must be something wrong with my cache pool. I deleted my docker image again and changed it's location to another ssd pool. 23 containers installed in under 20 min compared to 2 hours on my cache pool. I'll have to do some speedtest on my cache pool as all my appdata and downloads go there. Either it's the drives themself or my HBA. One of the pools is on my MB SATA connections I just don't remember which one. So it could be my HBA since the mover acted up as well. I ran the mover again manually earlier today and then it was fine. This was before I moved the dockerimage.
March 17, 20251 yr On 5/17/2024 at 6:55 PM, strike said: I think I figured it out. There must be something wrong with my cache pool. I deleted my docker image again and changed it's location to another ssd pool. 23 containers installed in under 20 min compared to 2 hours on my cache pool. I'll have to do some speedtest on my cache pool as all my appdata and downloads go there. Either it's the drives themself or my HBA. One of the pools is on my MB SATA connections I just don't remember which one. So it could be my HBA since the mover acted up as well. I ran the mover again manually earlier today and then it was fine. This was before I moved the dockerimage. Did you ever find anything more on this? I'm experiencing similar slowness. Wondering what tests you ran on your cache pool and what you found?
March 17, 20251 yr Author Community Expert I just ran some write tests to the cache pool which confirmed the drives was the culprit. They were probably thrashed due to trim not working. Tried to secure erase them and put them back in a new pool which sometimes can help,but it didn't this time.
March 19, 20251 yr On 3/17/2025 at 5:24 PM, strike said: I just ran some write tests to the cache pool which confirmed the drives was the culprit. They were probably thrashed due to trim not working. Tried to secure erase them and put them back in a new pool which sometimes can help,but it didn't this time. Were you able to eventually recover this or were the drives just shot? Were you just doing dd -> cache disks and testing speed or checking errors in logs? Did you pull them and test them outside of unraid? Thankfully I have a spare set of 2.5" SSDs that are nearly the same size....I could move my cache to these for now if I need to....just curious what you did to determine how bad they were and what your thresholds were. I think this is the first time I've truly faced potential SSD death. Any other SSD I've outgrown and handed down to be a USB drive or...actually, these I think were hand-me-down drives years ago.... One was...the other was bought to match so I'd get RAID 1. Uh...this don't look great... repeated cycles of `exception Emask...`, `failed command`, and `hard resetting link` Mar 15 01:10:45 THECONSTRUCT emhttpd: #011 /dev/sdg1 ONLINE 0 0 0 Mar 15 01:15:12 THECONSTRUCT kernel: ata14.00: exception Emask 0x0 SAct 0x80000 SErr 0x0 action 0x6 frozen Mar 15 01:15:12 THECONSTRUCT kernel: ata14.00: failed command: READ FPDMA QUEUED Mar 15 01:15:12 THECONSTRUCT kernel: ata14.00: cmd 60/08:98:80:55:2f/00:00:3b:00:00/40 tag 19 ncq dma 4096 in Mar 15 01:15:12 THECONSTRUCT kernel: ata14.00: status: { DRDY } Mar 15 01:15:12 THECONSTRUCT kernel: ata14: hard resetting link Mar 15 01:15:13 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 01:15:13 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 01:15:13 THECONSTRUCT kernel: ata14: EH complete Mar 15 02:44:07 THECONSTRUCT kernel: ata14.00: exception Emask 0x0 SAct 0x80000000 SErr 0x0 action 0x6 frozen Mar 15 02:44:07 THECONSTRUCT kernel: ata14.00: failed command: WRITE FPDMA QUEUED Mar 15 02:44:07 THECONSTRUCT kernel: ata14.00: cmd 61/10:f8:e8:4e:c0/00:00:1c:00:00/40 tag 31 ncq dma 8192 out Mar 15 02:44:07 THECONSTRUCT kernel: ata14.00: status: { DRDY } Mar 15 02:44:07 THECONSTRUCT kernel: ata14: hard resetting link Mar 15 02:44:08 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 02:44:08 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 02:44:08 THECONSTRUCT kernel: ata14: EH complete Mar 15 02:44:20 THECONSTRUCT root: /mnt/cache: 191 GiB (205107937280 bytes) trimmed on /dev/sdg1, /dev/sdf1 Mar 15 17:26:01 THECONSTRUCT kernel: ata14.00: exception Emask 0x0 SAct 0x8 SErr 0x0 action 0x6 frozen Mar 15 17:26:01 THECONSTRUCT kernel: ata14.00: failed command: WRITE FPDMA QUEUED Mar 15 17:26:01 THECONSTRUCT kernel: ata14.00: cmd 61/30:18:98:79:b0/00:00:0c:00:00/40 tag 3 ncq dma 24576 out Mar 15 17:26:01 THECONSTRUCT kernel: ata14.00: status: { DRDY } Mar 15 17:26:01 THECONSTRUCT kernel: ata14: hard resetting link Mar 15 17:26:01 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 17:26:01 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 17:26:01 THECONSTRUCT kernel: ata14: EH complete Mar 15 17:30:14 THECONSTRUCT kernel: ata14.00: exception Emask 0x0 SAct 0x20000000 SErr 0x0 action 0x6 frozen Mar 15 17:30:14 THECONSTRUCT kernel: ata14.00: failed command: WRITE FPDMA QUEUED Mar 15 17:30:14 THECONSTRUCT kernel: ata14.00: cmd 61/38:e8:d8:9c:b6/00:00:0c:00:00/40 tag 29 ncq dma 28672 out Mar 15 17:30:14 THECONSTRUCT kernel: ata14.00: status: { DRDY } Mar 15 17:30:14 THECONSTRUCT kernel: ata14: hard resetting link Mar 15 17:30:15 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 17:30:15 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 17:30:16 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 17:30:16 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 17:30:16 THECONSTRUCT kernel: ata14: EH complete Mar 15 17:43:23 THECONSTRUCT kernel: ata14.00: exception Emask 0x0 SAct 0x2000000 SErr 0x0 action 0x6 frozen Mar 15 17:43:23 THECONSTRUCT kernel: ata14.00: failed command: WRITE FPDMA QUEUED Mar 15 17:43:23 THECONSTRUCT kernel: ata14.00: cmd 61/48:c8:c8:80:02/00:00:35:00:00/40 tag 25 ncq dma 36864 out Mar 15 17:43:23 THECONSTRUCT kernel: ata14.00: status: { DRDY } Mar 15 17:43:23 THECONSTRUCT kernel: ata14: hard resetting link Mar 15 17:43:23 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 17:43:23 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 17:43:23 THECONSTRUCT kernel: ata14: EH complete Mar 15 18:20:55 THECONSTRUCT kernel: ata14.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen Mar 15 18:20:55 THECONSTRUCT kernel: ata14.00: failed command: WRITE FPDMA QUEUED Mar 15 18:20:55 THECONSTRUCT kernel: ata14.00: cmd 61/20:00:20:df:00/00:00:22:00:00/40 tag 0 ncq dma 16384 out Mar 15 18:20:55 THECONSTRUCT kernel: ata14.00: status: { DRDY } Mar 15 18:20:55 THECONSTRUCT kernel: ata14: hard resetting link Mar 15 18:20:56 THECONSTRUCT kernel: ata14: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Mar 15 18:20:56 THECONSTRUCT kernel: ata14.00: configured for UDMA/133 Mar 15 18:20:56 THECONSTRUCT kernel: ata14: EH complete Mar 16 02:41:34 THECONSTRUCT root: /mnt/cache: 189 GiB (203010412544 bytes) trimmed on /dev/sdg1, /dev/sdf1 Edited March 19, 20251 yr by NeoMatrixJR
March 19, 20251 yr 1 hour ago, NeoMatrixJR said: repeated cycles of `exception Emask.. Usually caused by poor connection of the Sata cabling. Reseat (both ends) or replace
March 21, 20251 yr Author Community Expert On 3/19/2025 at 2:46 PM, NeoMatrixJR said: Were you able to eventually recover this or were the drives just shot? They were just shot. I was thinking about using them in a pool for backup. But after testing them both in my main server and moving them to my backup server with the same issue, I gave up. It was weird tho because the speed was normal for a few minutes (probably because it was still in RAM), but then it just got slower and slower and I realized I couldn't use them. One thing I didn't test tho was just having them in a single 1 drive pool. It could be that just one of them is shot and the other one is fine.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.