I'm running 6.10 rc4.

My USB drive died.

My latest backup was from a month ago before I had upgraded from 6.10 rc2 to rc3 then rc4.

I followed the instructions to restore the backup and boot up with the new thumb drive.  

I then had to upgrade before I could transition the license.

I then saw a bunch of plugins and docker containers needed upgrades.

I upgraded the plugins, no problem.

I tried mass upgrading the containers but they seem to have gotten stuck.

I rebooted the server and tried upgrading one container, it downloaded the layers fine but got stuck at extracting them.  At this point the Web UI became unresponsive, so I rebooted the server again.
I figured maybe Docker needed to be nuked and I needed to start over.

I stopped the docker service and deleted the Docker data directory at /mnt/user/system/docker/, restarted then started the service again.

I then used Apps to reinstall the first container, it did the same thing, extraction took forever and the UI become unresponsive.

I realized it was trying to rebuild parity on the main array so I let it do that overnight and just checked again.  That first app installed but the next Docker container I tried did the same thing.  container layer extraction took forever for a 5MB layer and the UI become unresponsive (new tab)

 

This was not the case before the restore of the backup, so something is hosed somewhere.

 

All of my Docker data and the app data is stored on cache, which is a RAID 1 BTRFS of two 2 TB NVME drives that read/write at 5GB a second.
I ran a scrub on the BTRFS drives and nothing came back amiss.

I just noticed a BTRFS scrub is currently running on the cache drives.

    `Balance on '/mnt/cache' is running 32 out of about 255 chunks balanced (33 considered), 87% left`

So maybe that is it.  I'm going to go ahead and post this and respond back if things get better.

 

This server has been heavily tested with Memtest, prime95 and each disk deep scanned.

 

Specs: 

Dell r720XD with dual Intel® Xeon® CPU E5-2697 v2 @ 2.70GHz

128 GB RAM

Dual Samsung SSD 970 EVO Plus 2TB in RAID 1 with BTRFS

 

Attaching diagnostics below.

 

thor-diagnostics-20220426-1411.zip

I stopped the Balance to test as it was going to take forever.
I then added another container back, it still locked up the interface and is taking forever for a simple container.

I see there is a new version out, I'll try upgrading to that when the container finally finishes.
 

The interface never came back from the test above and the docker container never got installed (after over an hour)

I upgraded to RC5 and rebooted.  

Everything seems fine until I try to install another docker container using apps.

The instance the install starts the UI is unavailable in another tab of chrome., even while the docker image downloads and the interface is updated in the first tab.  Eventually the interface in the first tab also stops responding, maybe when I try to reload the UI in the second tab?

 

I'm super confused as to what could be causing this and am getting really frustrated. 

 

I should also note that SSH access is fine.  I can interact with the shell without issue while the UI is locked up.

I'll give this a bit then try again without trying to load the UI in another tab, see if the docker image download will complete then.

 

Nope, does not complete, at least not in any kind of normal time.  Takes freaking forever and the UI becomes unresponsive.

OK, found something out.

Doing a manual docker image pull from the ssh shows the same basic results, it starts out "ok" but then slows down to a crawl.

So I started looking at the cache drives and something is seriously wrong.  It is SLOWWWWW

I ran a basic test we use at work:
  dd if=/dev/zero of=/mnt/user/system/test1.img bs=1G count=1 oflag=dsync

On my local box which is Windows with Windows Subsystem for Linux installed with Ubuntu I get 
   1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.12554 s, 954 MB/s

That's on an NVME drive, but it's also on an emulated file system running in a VM, so . . 

On Unraid?

Well, I don't know, it's still running about fifteen minutes later.

ctrl+c no worky, one does not simply interrupt dd

And it's finished

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 543.663 s, 2.0 MB/s

Well.  There's the problem.  To megabytes a second.  Now . .why?!?!

 

In my first post I had gone to the Unraid UI --> Main --> Cache to look at the cache drive and everything looked fine, now when I go there after upgrading to RC5 the page is refresh looping REALLY fast (weird)  If I hit Escape and just the right time I can see filled out results.  Things still look normal there.

I can go to the Cache 2 page just fin and any other disk page, but the main cache page is borked.

 

Disk log for Cache looks fine

Apr 26 19:26:50 Thor kernel: nvme1n1: p1
Apr 26 19:26:50 Thor kernel: BTRFS: device fsid bca1d14f-b1a9-4986-bb92-c7c687acbe70 devid 2 transid 284634 /dev/nvme1n1p1 scanned by udevd (1385)
Apr 26 19:28:58 Thor emhttpd: Samsung_SSD_970_EVO_Plus_2TB_S59CNM0R925667W (nvme1n1) 512 3907029168
Apr 26 19:28:58 Thor emhttpd: import 30 cache device: (nvme1n1) Samsung_SSD_970_EVO_Plus_2TB_S59CNM0R925667W
Apr 26 19:28:58 Thor emhttpd: read SMART /dev/nvme1n1
Apr 26 19:29:45 Thor emhttpd: /mnt/cache devid 2 size 1.82TiB used 452.03GiB path /dev/nvme1n1p1

 

Someone help please?

Note: Using oflag=direct is fast, but using dsync is very very slow
dsync is
 

Quote

‘dsync’

Use synchronized I/O for data. For the output file, this forces a physical write of output data on each write. For the input file, this flag can matter when reading from a remote file that has been written to synchronously by some other process. Metadata (e.g., last-access and last-modified time) is not necessarily synchronized.

 

For those that care (seems to be approximately zero of you) I figured it out, sort of.

Finally tracked it down to network.  Transfers from Unraid to my PC were fine but transfers from my PC to Unraid or Unraid curling a file were abysmal. < 1 MB a second.

I uninstalled most of the plugins I had installed and shut down docker, restarted and now network is fine.

Turning Docker back on and now I can install Docker apps just fine.

So.  all those plugins and tweaking are gone.  Maybe in the end that is best. 

Also, Unraid folks, when installing docker containers, the UI in any other tab does still become unresponsive.  Just FYI.

Hi Sarge,

 

Sorry to come that late but i am experiencing the exact same thing these days.

 

It takes ages to pull and extract a single docker container.
During all that process, the rest of the interface is simply unresponsive.

 

I understand you uninstalled the plugins but it sounds unrelated to the network.

Any further information about that?

I am also having this exact same issue. Interface freezing up/unresponsive dockers tab. Dockers seems to work fine when they eventually turn on but installing a new docker takes ~30 min

Attach Diagnostics to your NEXT post in this thread 

Glad I'm not alone. Also having the exact same experience. @Sarge what were the steps you took to fix this?

 

ark-diagnostics-20230602-2121.zip

One Year later with same Problem.

In Dashboard the CPU Loads goes all up, while cpu in htop stays low as 1-2%

cache drive write with 1-5mb/s while installing docker container.

Already tried to delete the docker image file and recreate it - same problem.

I have ASUS H610I Mini-ITX with i5 12500T

but installing and start/stop docker container is slower than my raspberry pi zero 2w