UnRAID slow after update from 6.8 to 6.10

[arough]

Hey,

 

I updated my server from 6.8 to 6.10 and now pretty much everything runs with high CPU load and/or slower then before the update.

 

iperf is somewhat slow.
 

Spoiler

 

 

 

Before the update I was able to move files from a local machine to the server with the full 1Gbit/s and now im only getting 70-80MB/s.

In the GUI it looks like there isn't even any activity on the cache drives until the transfer is nearly done and then it goes up to the ~500Mbit/s of my MX500s for the last few seconds.

When moving files on the server the speed normally went up to ~400MB/s and now hovers around 140MB/s.

 

One measurable slowdown is with my duplicati backup. It normally took 1-2 minutes when nothing has changes and only the checks were running. It now takes 5-7 minutes.

 

Spoiler

 

A minecraft server I set up a few days ago ramps up my CPU to ~90% with one player connected.
Before the update four players were able to play without that much load on the CPU.

 

Plex also uses nearly 100% when transcoding for one user and ~50% when direct playing.

That are the things I noticed so far, I think.

 

I don't understand what is happening here. The network isn't up to snuff and the CPU is overloading constantly.
The drives seem to be fine, when checking the speed via the DiskSpeed docker.
(One of my array drives is collecting `Reallocated sector count` errors, but I'm already in contact with Seagate for advanced RMA. And as this seems to be a problem when using the cache drives, I don't see how that could be related)

 

UnRAID forgot the settings for the SMART notifications after the update so I was getting the "normal" `current pending sector count` errors every MX500 gets. I didn't find any other settings UnRAID "lost" after the update so far.

 

I attached the diag archive.

 

Can someone make any sense of all this?
 

Thanks in advance.

tower-diagnostics-20220912-2100.zip

Edited September 12, 2022 by arough
forgot diag files

[JorgeB]

Try disabling Spectre and Meltdown mitigations to see if you notice a difference, you can do that by adding mitigations=off to syslinux.cfg tne reboot, like so:

 

 

 

[arough]

Sadly that didn't help.

 

Spoiler

 

iperf was even slower for a bit, but I don't think that's because of the disabled mitigations.
At least I couldn't explain how it should have an effect on it.

 

Spoiler

 

While copying a 2.1GB file this is what htop looks like:

 

Spoiler

And the UnRAID GUI: 

 

Spoiler

 

As you can see there is some activity after the file transfer is complete.
The progress bar for the transfer is at 100% and my MacBook isn't sending data anymore.
Only when it's at 100% the cache is written to at it's normal speed of ~500MB/s.

Same behaviour with my Windows PC.
It feels like there is some kind of slower cache in between. It definitely isn't the RAM as it would be faster and the RAM usage isn't going up significantly while transferring.

 

And of course I'm not using WiFi for all of this.

[JorgeB]

Please go back to v6.8 and post the iperf results.

[arough]

Can I just use the backup of my Flash Drive to go back or is there another way?

[JorgeB]

If you updated using the GUI you can downgrade by going to Tools -> Update OS

[arough]

Well...

I restored to 6.8.3 via the GUI and now I don't have an `Apps` tab an my docker tab says `No Docker containers installed`.

I had to install iperf via Nerd Pack again as it was not installed after the downgrade.

It doesn't look so different from 6.10.3

 

Spoiler

 

But there is something fishy going on now with the docker stuff.

 

Here is the diagnostics.

 

tower-diagnostics-20220913-1330.zip

[arough]

oh I think I see what happened.
My cache drives aren't assigned.
If I try to assign them they are marked as `new device`.
I didn't start the array with them assigned yet, because I fear UnRAID will try to erase them, as they are detected as a new device.
How to proceed from here?

[trurl]

Go to Settings and disable Docker and VM Manager. Since you started the array without cache all the related shares were recreated from scratch on the array.

 

[JorgeB]

6 minutes ago, arough said:

If I try to assign them they are marked as `new device`.

That's OK, as long as you assign all the pool devices the existing pool will be imported, there can't be a "data on this device will be deleted" warning in front of any pool device.

 

18 minutes ago, arough said:

It doesn't look so different from 6.10.3

That suggests the update isn't the problem.

[arough]

God damn it...
forget every iperf log I posted before.
For some reason it is way slower over my Mac then with my PC.
The two top ones are from my windows PC.

 

Spoiler

 

But UnRAID is nevertheless faster with 6.8.3.

The duplicati backups take 1-2 minutes again.
The minecraft server for some reason will not start so I can't test it right now.
And because CA is missing (the newest version isn't compatible with 6.8 anymore and I can't find a way to install an old one) I can't install another minecraft app/docker.

Is there any way to do this?

 

Plex seems to behave better.
It is now transcoding for ~30 secs at 100% to fill the buffer and then only spiking every few seconds for like a second to fill the buffer again.

 

While moving the same file (on my Mac again to keep it comparable) htop now looks like this.
 

Spoiler

 

Way lower load than with 6.10.3.

 

Moving files via FileBot is now back to ~450 MB/s again, opposed to ~140MB/s with 6.10.3.
I use FileBot on Windows, but it is transferring directly on the cache drives as source and target are both on it.

 

Should I update to 6.10.3 again and see how iperf performs with my PC so the data I present to you is actually useful? 😅

 

EDIT: Just assigning the cache drives again worked btw.

Edited September 13, 2022 by arough

[trurl]

1 hour ago, trurl said:

Since you started the array without cache all the related shares were recreated from scratch on the array.

 

5 minutes ago, arough said:

assigning the cache drives again worked

But those shares will still have files on the array.

[arough]

I don't know what you mean.

 

When I started without the cache drives UnRAID created every share again?
 

After assigning the cache and starting the array every share looks fine to me.

All cache settings are there and all the files seem to be there.

What do you suppose I should do?

[trurl]

https://wiki.unraid.net/Manual/Shares#Default_Shares

 

If Docker / VM Manager is enabled, and these shares don't exist, they get created. Since you didn't have cache they didn't exist and so were created on the array. Now that you have cache again, they are also on cache with their original contents, but the folders/files that were created on the array still exist.

 

Go to User Shares and click Compute... for each of those shares to see what I mean.

[arough]

As far as I can tell this is only relevant for `prefer/only` right?

 

Spoiler

 

The other ones already had folders on array disks and just used them.

As there was no docker containers running and nothing moved in the `running without cache` state only empty folders were created and the complete `system` share was copied somehow.

How should I proceed from here?
Delete the empty folders manually via the console and delete the `system` folder on disk4?

 

Spoiler

 

[trurl]

11 minutes ago, arough said:

only relevant for `prefer/only`

Only relevant for those specific shares mentioned in the link, since those are the only shares that are created automatically by having Docker/VM Manager enabled, if they don't already exist. You can see these paths in the Settings for Docker and VM Manager.

 

12 minutes ago, arough said:

no docker containers running

If any containers were run they would have created their appdata on the array. The shares mentioned are created by simply having Docker and VM Manager enabled.

 

14 minutes ago, arough said:

complete `system` share was copied somehow

Not copied, recreated. The system files on the array are not the same as those on cache, since those on the array didn't actually have any VMs or containers created in them. They are new.

 

16 minutes ago, arough said:

Delete the empty folders manually via the console and delete the `system` folder on disk4?

Since the files on cache have precedence, those on the array are only taking up space. You can delete them.

[arough]

Thanks @trurl.
Will delete the shares.

 

 

I forked CA and hacked together an old version that still runs on 6.8.3 to potentially fix the Minecraft docker.
In the end CA wasn't even needed to fix it

I got Minecraft running again (was an issue with an old version of `runc` that is already updated on UnRAID 6.9.2+)

The Minecraft server is running great again. Peaks to 90% when someone is joining or exploring new areas, but when playing in already generated chunks UnRAID hovers around 20%.

 

It looks to me as if there is a direct link between my problems and the upgrade.
Is there anything I should test while on 6.8.3 to gather data and compare it to 6.10.3?
 

Here is another diagnostics with docker running and everything as fast as it was before.

tower-diagnostics-20220913-1630.zip

Are there maybe some other settings that were introduced in 6.9 or 6.10 that could mess with my setup?
Should I maybe upgrade to 6.9 first to see if the slowdowns happen there too?

Is there any way to check if the Spectre and Meltdown mitigations were correctly disabled on 6.10.3 after I edited the `syslinux.cfg`?

I would of course check this after upgrading to 6.10.3 again.

 

 

I wanted to upgrade to a Ryzen 3600 in the coming weeks and thought it would be good to upgrade UnRAID before that.
Maybe that wasn't the best idea