Libvirt Error: virNetSocketReadWire:1791 : End of file while reading data: Input/output error after upgrading to 6.11.5

[Green Dragon]

Hiya!

 

After upgrading to 6.11.5 I am seeing the following errors in the libvirt log, about every 10 minutes:

 

2022-11-21 02:38:20.190+0000: 13082: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2022-11-21 02:48:23.307+0000: 13082: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2022-11-21 02:58:26.536+0000: 13082: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2022-11-21 03:08:29.686+0000: 13082: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2022-11-21 03:18:32.955+0000: 13082: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error

 

Thoughts?

[baboolian]

Same here (although it started in 6.11.4), and when I launch my previously working windows VM with gfx card passed through it locks up Unraid and I have to do a force reboot

[Green Dragon]

Also found that If I stop all VMs, disable VMs, then enable VMs the libvirt service fails to start.  A complete reboot of UnRaid is required to get VMs working again.

[exus]

I installed unraid on my new server for the first time two days ago and I've the same error every 10 minutes

[Gonzo-the-great]

Will jump in here and say "Me too". From my research, it's possible that this is a known issue with libvirt library, but nothing definitive to support that.

[bosswaz]

Im living the same issues you are describing...however my VM will not shutdown clean...runs fine during the day.

If I try a grace full shutdown the system locks up and I need to power down to restart.

Also something occurs overnight as in the morning I find its locked up...again I need to power down to restart as a Grace full shutdown locks up unraid.

[SimonF]

There messages are likely due to the virsh client disconnecting from libvirtd are you running VMBackup or similar plugins?

[Green Dragon]

13 minutes ago, SimonF said:

There messages are likely due to the virsh client disconnecting from libvirtd are you running VMBackup or similar plugins?

I am running VMBackup.  No other VM related plugins though.  Is this typical behavior for a plugin such as VMBackup to cause a connect/disconnect which generates a libvert I/O error every ten minutes?  Is this a reportable bug?  VMBackup?  Libvert? UnRaid?

[SimonF]

34 minutes ago, Green Dragon said:

I am running VMBackup.  No other VM related plugins though.  Is this typical behavior for a plugin such as VMBackup to cause a connect/disconnect which generates a libvert I/O error every ten minutes?  Is this a reportable bug?  VMBackup?  Libvert? UnRaid?

Not sure dont use VMBackup. Do you want to post diagnostics for review to see if I can see anything. Libvirt has been the same since 6.11.1 did you have the issue pre 6.11.5? Maybe related to something else.

[Green Dragon]

3 hours ago, SimonF said:

Not sure dont use VMBackup. Do you want to post diagnostics for review to see if I can see anything. Libvirt has been the same since 6.11.1 did you have the issue pre 6.11.5? Maybe related to something else.

Here ya go...  Enjoy!

gothamcity-diagnostics-20221127-1426.zip

[SimonF]

12 minutes ago, Green Dragon said:

Here ya go...  Enjoy!

gothamcity-diagnostics-20221127-1426.zip 237.21 kB · 1 download

I think you where trying to install the guest agent for 2003 have you tried removing it? or did it not install?

[Green Dragon]

3 minutes ago, SimonF said:

I think you where trying to install the guest agent for 2003 have you tried removing it? or did it not install?

Attempts have been made.  Unfortunately, as you can see from this thread, I am unable to get it to successfully install.  As it stands today I do not have it successfully installed.

[SimonF]

1 hour ago, Green Dragon said:

Attempts have been made.  Unfortunately, as you can see from this thread, I am unable to get it to successfully install.  As it stands today I do not have it successfully installed.

Cannot see any thing obvious at present.

Edited November 27, 2022 by SimonF

[bosswaz]

Solved my problem by isolating iommu for my radeon video card and then binding to vfio...in the tools system devices tab...took a long while to find the solution but im happier now...

[Green Dragon]

4 hours ago, bosswaz said:

Solved my problem by isolating iommu for my radeon video card and then binding to vfio...in the tools system devices tab...took a long while to find the solution but im happier now...

Awesome!  Could it be as simple as me needing to bind the 'Serial bus controller: Intel Corporation Comet Lake' to iommu for my 2003 server qemu issue referenced above?

[chrisjenx]

On 11/20/2022 at 9:01 PM, Green Dragon said:

Also found that If I stop all VMs, disable VMs, then enable VMs the libvirt service fails to start.  A complete reboot of UnRaid is required to get VMs working again.

I have the exact same issue, I am running portainer (docker images) inside a VM, wondering if some weird VTx issue, I only started noticing on the last 2/3 versions of unraid.

[Sascha_B]

On 11/21/2022 at 5:01 AM, Green Dragon said:

Also found that If I stop all VMs, disable VMs, then enable VMs the libvirt service fails to start.  A complete reboot of UnRaid is required to get VMs working again.

 

I also have these errors in the libvirt log every 10 minutes. My system behaves exactly the same way. After a reboot, the VM run again, the log entries come back.

 

My behavior is in this thread

 

[bobobeastie]

I just started having this same issue after changing all of my cache mounts from referencing user to the cache drives name because I saw a youtube video about IO Wait.  Changed the 2 path settings in the VM config back and still have the issue. it's a home assistant VM if anyone cares.  I was having rebooting issues before today, but I found a message regarding insufficient memory, which I think I solved by changing the memory settings for the VM, so probably unrelated.

 

Reboot didn't do anything.  Tried creating a new debian VM and that wont start either.  Diagnostics attached are from after reboot and Debian attempt/test.

nastheripper-diagnostics-20230119-1642.zip

[nongfeng]

这个问题有什么进展吗？我也遇到了同样的问题，暂时不清楚会有什么影响
下面是我的诊断报告
nongfeng-diagnostics-20230301-1046.zip

[JimmyGerms]

I'm getting these errors too every 10 min. I rebuilt my unRAID flash from scratch and setup everything manually again one by one to track down another issue and noticed it happening. Not sure where to look for a further trace stack on the error.

 

I suspect this is happening to quite a few people?

[Lightsabata]

Yes i have the same issue

Edited March 6, 2023 by Lightsabata

[darxtorm]

+1 seeing the same issue, 10min like clockwork
 

2023-03-01 13:47:51.553+0000: 11837: info : libvirt version: 8.7.0
2023-03-01 13:47:51.553+0000: 11837: info : hostname: jdstorage5
2023-03-01 13:47:51.553+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 13:57:52.293+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 14:07:53.041+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 14:17:53.774+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 14:27:54.536+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 14:37:55.269+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 14:47:56.013+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 14:57:56.750+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 15:07:57.579+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 15:17:58.364+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 15:27:59.206+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 15:38:00.373+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 15:48:01.279+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 15:58:02.275+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 16:08:03.318+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 16:18:04.442+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error
2023-03-01 16:28:05.263+0000: 11837: error : virNetSocketReadWire:1791 : End of file while reading data: Input/output error

 

Edited March 7, 2023 by darxtorm

[SimonF]

This is the feedback from Libvirt developers.

 

The socket connection to libvirtd / other daemons was abruptly
closed. Essentially you can almost always ignore this error
message as being mostly harmless.

With regards,
Daniel

[reboot81]

Same issue here.

virNetSocketReadWire:1791 : End of file while reading data: Input/output error

 

VM (WS2019) no longer booting. The vdisk contains "C:" (verified by mounting it) but the VM doesn't see the drive at boot and fails.

 

 6.11.5

[SimonF]

2 hours ago, reboot81 said:

Same issue here.

virNetSocketReadWire:1791 : End of file while reading data: Input/output error

 

VM (WS2019) no longer booting. The vdisk contains "C:" (verified by mounting it) but the VM doesn't see the drive at boot and fails.

 

 6.11.5

Suggest you raise own support thread and post diagnostics