Been having this issue for a while now. Didnt push much as the server was virtualized and unsupported, but after dealing with the issue for a year I gave in and moved back to a physical server, but the issue is still there. it is:\

 

On transfers to the singular share I have setup it starts OK, and runs fine for a minute or so. Then network transfers stop, my disk IO times shoot up, and it sits there. If it's something like sabnzdb doing a download, it eventually restarts and completes, but something like a plain file copy/backup form my system to the share fails because the timeout is too long, sometimes I even lose t he network mount. Main work machine runs Arch/KDE, but other network devices such as Linux and Windows laptops show the same issue.

 

CPU usage during this is usually pretty much idle, except for 1 core pegging 100%. Sometimes it stays pegged, sometimes a different core gets pegged, before then settle down and transfers start again.

 

Reducing transfers speeds in sabnzbd can help improve stability, but then I am downloading at half or less speed.

 

Attached for reference is a diagnostics package created while downloading with sabnzbd about 5 seconds after a slowdown from 40MB/s to nothing. I can also do a large file copy and generate when it crashed if that would be more helpful. Any other info/tips/ideas would be appreciated. Its not a dealbreaker yet, but it is causing issues with my backups and work and I would like to fix it.

nas-diagnostics-20241017-0103.zip

28 minutes ago, Genom said:

Reducing transfers speeds in sabnzbd can help improve stability, but then I am downloading at half or less speed.

First of all: what you see is NORMAL behaviour.

If the disk system cannot store the incoming data fast enough, it fills up the RAM as a buffer, then, if this runs full too, it has to stop everything and slowly write out the buffers to free them up.

Afterwards, transfers continue.

And again, again and again...

 

If you pace (slow down) the incoming speed, things will run smoother, the stalls will happen less frequent,

The lower download speed is just an illusion. Use a stopwatch and see that it takes the same amount of time with fast (and stalling) speed or slow (and smooth) speed.

It is just an optical thing that stresses your nervs. Relax! (Of course, the stalls block the whole server, so they are really annoying)

 

"How to fix it" is a real big problem. In general you need faster disks or more (bigger) cache SSDs. (which also need to be the kind that can stand large amounts of sustained write speed, these are not found in the cheapest regions!). Also you can limit UNRAID to use less RAM buffers, this will make start writing out the data earlier,slowing down transfers once more but smoothing it too. You could end up with a "lower than expected" but "without stalls" speed. There is no real general advice which values to use, it depends on your specific hardware. You need to experiment with it and see what fits your demands / soul best.

 

For instance, I have lowered the parameters of my 10G link (possible transfer rate 1,1Gb / s) to about 520Mb/s. This assures me constant transfers without stalls.

 

BTW: if you transfer small (but many) files it is not possible to prevent stalls at all. Each file needs a directory update and the SMB protocol will not move on to the next file until the receiver signaled a full sync. So you have only influence on BIG transfers. I dont know your sabnzbd but from what I have heard it transfers small chunks from Usenet and reconstructs them locally to larger files? If this is true, there is little hope but lowering the incoming speed.

 

 

Edited October 17, 20241 yr by MAM59

I appreciate the feedback, but I dont think this is the issue, as the storage is much faster than my 1Gb ethernet. I can also create a share on pretty much anything in my network, and move a 10GB file with no start/stops. It's only the Unraid system that does it.

Mhmm, ive no clue about first gen ryzen and linux but this looks "not normal", could certainly have a hand in the issue i would assume.

 

Oct 16 21:37:01 NAS kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
Oct 16 21:37:01 NAS kernel: ACPI: \_PR_.C000: Found 2 idle states
Oct 16 21:37:01 NAS kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
Oct 16 21:37:01 NAS kernel: ACPI: \_PR_.C002: Found 2 idle states
Oct 16 21:37:01 NAS kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
Oct 16 21:37:01 NAS kernel: ACPI: \_PR_.C004: Found 2 idle states
Oct 16 21:37:01 NAS kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)


Oct 16 21:37:01 NAS kernel: mce: [Hardware Error]: Machine check events logged
Oct 16 21:37:01 NAS kernel: mce: [Hardware Error]: CPU 9: Machine Check: 0 Bank 5: bea0000000000108
Oct 16 21:37:01 NAS kernel: mce: [Hardware Error]: TSC 0 ADDR 1ffff8121de60 MISC d012000100000000 SYND 4d000000 IPID 500b000000000 
Oct 16 21:37:01 NAS kernel: mce: [Hardware Error]: PROCESSOR 2:800f11 TIME 1729128985 SOCKET 0 APIC 9 microcode 8001138

Thats just a guess on my end because the logfiles are basically empty.

Adding on that I have been doing some testing, and it may have been related to the 6.12 storage changes. I was running on 6.8 for a while since it was stable and I dont update stable things, but my flash drive died, and when I got a new one re-licensed, I put on the latest software package, that being 6.12, and it changes the concept of cache completely. So my config was primary storage cache, secondary array, and with only 2TB of cache SSD, it was constantly shuffling files around it looks like. Changing my share to Array only, things seem to be more stable, but needs more testing. At least I was able to finally copy a 42GB backup file that had been pending forever.

4 hours ago, Genom said:

I put on the latest software package, that being 6.12, and it changes the concept of cache completely

Actually the functionality remained identical.   What changed was the terminology used as mentioned here in the online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page.    It was thought this would be a clearer way of describing it to new users and prepared the way for changes coming in Unraid 7.

Than maybe on the new flash something got corrupted, because the same cache drives/array under 6.8 worked fine, and once I went in and removed my cache pool use a yesterday, the share has been working fine without the unstable start stops I had. I am too behind on stuff to fiddle with the cache settings for the next few weeks so I am going to mark this as solved, but I don't really feel it is and will likely have to revisit it.