[6.8.3-6.9.2] Huge CPU Load and IO-Wait when copying to Array

Are you copying from a VM to the array or from the array to a VM? Or does a VM write to an passed thru virtual disk?

 

I ask because I do see high QEMU utilization in your top window.

 

3 hours ago, hawihoney said:

Are you copying from a VM to the array or from the array to a VM? Or does a VM write to an passed thru virtual disk?

 

I ask because I do see high QEMU utilization in your top window.

 

 

No VMs are involved in this case. Just copying from my PC to a SMB share. 

Also, the problem consists regardless of VMs or Docker Containers. If I switch them all off, I still get the same high CPU utilization and iowait while copying to the array.

Edited May 27, 2021 by windowslucker

16 minutes ago, windowslucker said:

No VMs are involved in this case. Just copying from my PC to a SMB share.

 

Ok, but qemu-system-86 is the PC System Emulator. It shows that huge CPU load in the screenshot.

 

QEMU with high CPU load, and with high IO wait values, points directly to data written thru this emulator. QEMU has to emulate virtual disks and it's reads/writes. If there are no VMs envolved, I'm out because I don't understand running QEMU processes without active VMs.

 

I do see the shfs process as well, but both can escalate each other. High IO-wait because of QEMU emulator with parallel access to SMB can bring a system nearly to halt.

 

This is my own experience, but if there are no VMs running I can't help you any further because I do not understand it then.

 

25 minutes ago, hawihoney said:

I'm out because I don't understand running QEMU processes without active VMs

 

Well I may have been unclear. The VMs are active in the screenshot above but weren't actively used or copied from. They were more or less sitting idle at that time.

48 minutes ago, windowslucker said:

The VMs are active in the screenshot above but weren't actively used or copied from.

 

Please have a look at the screenshot below. This is my running system with two active VMs. There's a 20 GB copy going on from bare metal from the host to one of the VMs.

 

Now compare the CPU usage of my system with your screenshot. My VMs are heavily in use and your VMs are not actively used? I guess that the VM's are 

Don't know what happened. Without further activity my post has been posted and I cann't edit it any longer.

 

---

 

Now compare the CPU usage of my system with your screenshot. My VMs are heavily in use and your VMs are not actively used? I guess that the VM's are busy with itself.

 

I would check what the VMs are doing, what and were they writing to. Is there enough RAM for the VMs, etc.

 

This may block SMB out.

 

21 hours ago, hawihoney said:

My VMs are heavily in use and your VMs are not actively used? I guess that the VM's are busy with itself.

 

I just ran a test with VM Manager switched off under Settings -> VM Manager and another test with VM-Manager switched on, but without starting any VMs. So no VMs turned on in any of those two tests.

The results between both of those tests stayed the same.

The only difference to the problem I described in my post above was, that the CPU load didn't skyrocket like it did earlier, but rather increased slowly and gradually over the time of 3 minutes or so. The outcome, however is the same. Dockers unresponding and huge CPU load.

 

This test was done by coping three 10 Gb files from my desktop computer to an unraid share. The share's configuration can be seen in the screenshot below.

 

 

I've also recorded the main part of the copying stuff where the unresponsive dockers, switched off VMs and huge CPU load can be observed.

 

The Video can be found here: Unraid Bug Report CPU/IOWAIT

 

Edited May 28, 2021 by windowslucker

Try copying straight to disk shares to confirm the issue is not present. This seems related to all other shfs tickets logged on this very forum. I've been using disk shares due to this issue.

4 hours ago, xxxliqu1dxxx said:

Try copying straight to disk shares to confirm the issue is not present

 

I'm not sure if I understand you correctly. Do you mean I should try copying straight to e.g. /mnt/disk1/folderxy instead of /mnt/user/folderxy?

11 minutes ago, windowslucker said:

 

I'm not sure if I understand you correctly. Do you mean I should try copying straight to e.g. /mnt/disk1/folderxy instead of /mnt/user/folderxy?

Exactly. You will see the CPU usage will be lower. I just want you to test this to confirm it's the same problem that was mentioned in several other posts - copying to the array causes a lot of shfs problems, which involves unusually high CPU loads. Copying straight to the disk share does not exhibit the same behavior which further reinforces the problem in shfs which I hope limetech will fix one day.

Edited May 29, 2021 by xxxliqu1dxxx
typo

1 hour ago, xxxliqu1dxxx said:

Exactly. You will see the CPU usage will be lower. I just want you to test this to confirm it's the same problem that was mentioned in several other posts - copying to the array causes a lot of shfs problems, which involves unusually high CPU loads. Copying straight to the disk share does not exhibit the same behavior which further reinforces the problem in shfs which I hope limetech will fix one day.

Hi - sorry if I was not clear, but I would ask if you could test and post your results, this way you can be confident the problem is with uploading to the array, vs uploading to the disk shares. Plus, additional information will be documented for this SHFS issue.

Quote

I would ask if you could test and post your results

 

I just did another test copying directly to the disk share, as you suggested. VMs were switched off, again. As I can't think of any way to copy to a disk share using SMB, the only difference to my prior tests is that I was using WinSCP to copy the files to /mnt/disk1/copyTestShare.

As far as I can tell, the issue stayed the same. Unaccessible dockers and high CPU load. The Unraid Webinterface stayed accessible, though.

 

 

 

Also, WinSCP got some weird disconnects while copying. The text on the top right window says: "The remote computer didn't send any data for more than 15 seconds." I think it has something to do with unraid freezing, but of course I'm not completely sure if that's really the case.

 

Again, I recorded everything and uploaded it to Youtube for you to review.

 

 

You can use disk shares via SMB. You just need to enable that.

Make sure you have yes configured there. You may have to stop the array first. And then, try again with SMB.

1 minute ago, xxxliqu1dxxx said:

You can use disk shares via SMB. You just need to enable that.

Make sure you have yes configured there. You may have to stop the array first. And then, try again with SMB.

This is Settings \ Global Share Settings in case you are wondering.

i love it when threads end in nothing. so helpful.