Inconstant write speeds.

[kodyorris]

Hello Everyone.

Im have some very inconstant write speeds. When transferring via SMB to a cache drive I get anything from 110MB/s to 2MB/s and hover around 50MB/s. When reading from the array or cache I get about 107MB/s constantly. When I watch the stats it looks like it caches with ram, the once ram fills it basically pauses the transfer, dumps to my cache drives, then starts the transfer again. I have tried transferring without the cache drives and I have basically the same result but with it dumping to the array. 

 

Why isn’t it just writing to the cache drives? They have a 550MB/s read/write so caching to ram seems useless when I can only transfer at gigabyte speeds anyway. Is there a way for the system to not use my ram as data cache? 

 

CPU = 3600x

Cache = 2x 1TB SSD’s

Array = 10TB and 8TB SAS and Sata drives. 

VM’s = 1 windows VM

[kodyorris]

jarvis-diagnostics-20200706-1658.zip

[Jerky_san]

When you do the transfers on the main page does the write speeds on the cache reflect what you are seeing as well? How big of files are you testing with? The pictures a interesting.

[kodyorris]

hey thanks for the response. Small update, The problem isn't as bad if a transfer using a windows machine (the above screen shots are from a mac). However, even in windows, It will still do this "cache with ram, the once ram fills it basically pauses the transfer, dumps to my cache drives, then starts the transfer again". If i don't use the cache drives it will transfer at 108MB/s for hours if needed. So it only seems to be a problem if i use cache drives but this behavior makes the purpose of cache drives pointless. 

 

To answer your questions, yes I will see that nothing is writing to the drives in the main page, then it will spike for a few seconds at 130+MB/s while it writes to the cache drives. I have been testing with anything from 10GB to 200GB. I work in media production so i often have large files like this. 

[Jerky_san]

17 minutes ago, kodyorris said:

hey thanks for the response. Small update, The problem isn't as bad if a transfer using a windows machine (the above screen shots are from a mac). However, even in windows, It will still do this "cache with ram, the once ram fills it basically pauses the transfer, dumps to my cache drives, then starts the transfer again". If i don't use the cache drives it will transfer at 108MB/s for hours if needed. So it only seems to be a problem if i use cache drives but this behavior makes the purpose of cache drives pointless. 

 

To answer your questions, yes I will see that nothing is writing to the drives in the main page, then it will spike for a few seconds at 130+MB/s while it writes to the cache drives. I have been testing with anything from 10GB to 200GB. I work in media production so i often have large files like this. 

Hmm that is strange. I kind of wonder if it's not a btrfs issue but I know a few people run mirrored pair cache drives on btrfs without issue. The fact that it waits before dumping to the cache drive is pretty odd. Weird thing is to my knowledge there isn't a lot of tuning to the cache you can do as side from changing the format. I kind of wonder what would happen if you wrote a file to cache and copied it from within unraid to the cache again. This would rule out networking and a few other things at least. Maybe someone else can chime in as well.

[kodyorris]

Yeah ill give that a shot and post my results. Ill also do a seperate test with just one cache drive to see if that changes anything 

 

[Jerky_san]

2 minutes ago, kodyorris said:

Yeah ill give that a shot and post my results. Ill also do a seperate test with just one cache drive to see if that changes anything 

 

If your willing to break your cache up try running one formatted as xfs if a single btrfs doesn't work. Tbh I rarely if ever leave real data on the cache drive for long. I have a mover schedule that moves it every night. I used to also have btrfs but had hang ups and stuff that were all alleviated by going to xfs

[kodyorris]

If i start the mover it will copy from the cache drives to the array at a constant 130MB/s-160MB/s.

[Jerky_san]

25 minutes ago, kodyorris said:

If i start the mover it will copy from the cache drives to the array at a constant 130MB/s-160MB/s.

If it weren't for the ram spiking and dumping to cache after I'd want to say network issue but the ram spiking/dumping is throwing me..

[kodyorris]

I brought my cache drive down to a single drive formatted in XFS, Same results. Bust of writes to the cache drive causing slow and inconstant transfer speeds. 

[Jerky_san]

1 minute ago, kodyorris said:

I brought my cache drive down to a single drive formatted in XFS, Same results. Bust of writes to the cache drive causing slow and inconstant transfer speeds. 

Sigh, I feel like I am out of my depth on this one. The network graph seems to show good performance there though I would kind of expect close to line speed. But you also said using the mover causes it to write to the array very rapidly. I am hoping someone else will chime in that can better help you.

 

I can say looking at your sys log it's jammed packed full of these errors. It's interesting because I don't see any drives labeled disk29 or disk0.

 

Jul  6 02:49:28 Jarvis kernel: md: do_drive_cmd: disk29: ATA_OP e0 ioctl error: -5
Jul  6 02:49:29 Jarvis kernel: mdcmd (5077): spindown 0
Jul  6 02:49:29 Jarvis emhttpd: error: mdcmd, 2723: Input/output error (5): write
Jul  6 02:49:29 Jarvis kernel: md: do_drive_cmd: disk0: ATA_OP e0 ioctl error: -5
Jul  6 02:49:29 Jarvis kernel: mdcmd (5078): spindown 29
Jul  6 02:49:29 Jarvis emhttpd: error: mdcmd, 2723: Input/output error (5): write
Jul  6 02:49:29 Jarvis kernel: md: do_drive_cmd: disk29: ATA_OP e0 ioctl error: -5

 

[kodyorris]

Well i do appreciate your help. Hopefully someone would know whats going on. I unfortunately don't know what any of these errors mean ( i'm a noob)

[Jerky_san]

8 minutes ago, kodyorris said:

Well i do appreciate your help. Hopefully someone would know whats going on. I unfortunately don't know what any of these errors mean ( i'm a noob)

Just a quick question.. are your drives mounted via the Fujitsu raid card or via onboard? Also is there some sort of expander or something?

[kodyorris]

Im running a IBM 81Y4494 H1110 card flashed to "IT mode". Im not using any onboard controllers. The only other PCIE card I have is a usb 3.0 card that I haven't used during these test.

[nicksphone]

6 hours ago, kodyorris said:

Im running a IBM 81Y4494 H1110 card flashed to "IT mode". Im not using any onboard controllers. The only other PCIE card I have is a usb 3.0 card that I haven't used during these test.

if your cache drives are sata try to put them on the on board controller that way they have their own lane i do the same with the party drives. i see less slowdowns that way even though your suppsed to have tons of through put in the pci lanes i did notice a difference. This will probably have very little effect on your issue but will make moving and party checks quicker. there is an issue with hypervisor and smb transfers this maybe fixed in the next version as we will be on a newer hypervisor version when 6.9 is rolled out (you will have to update the driver iso as well) but its a common issue with windows vm's and not an unraid only thing. maybe they fixed it in hyper 5.x

Edited July 7, 2020 by nicksphone
added stuff

[kodyorris]

I tried running the cache drives off of the Mobo, same result. Do you have any idea what could be causing these errors? They seem basically constant. My full DiaLog is posted above.

 

[Vr2Io]

8 hours ago, kodyorris said:

I tried running the cache drives off of the Mobo, same result. Do you have any idea what could be causing these errors? They seem basically constant. My full DiaLog is posted above.

 

Disk 0 29 are parity disk, SAS disk not support spindown, you may try set those disks "individual" never spindown .

[kodyorris]

Thanks! ill change that setting now and see if that at least clears up those errors. 

[Jerky_san]

9 hours ago, kodyorris said:

Thanks! ill change that setting now and see if that at least clears up those errors. 

I tried copying about 500 gb of files to my cache drive and i started seeing something similar to what you are experiencing but my drops are much less because I have an NVME I am assuming. I'm going to keep digging to see if I can figure something out.

[kodyorris]

Interesting, I start experiencing this after just a few seconds. 

[Jerky_san]

23 minutes ago, kodyorris said:

Interesting, I start experiencing this after just a few seconds. 

I'm going to try something soon after I finish this mover.

[Jerky_san]

On 7/8/2020 at 10:28 AM, kodyorris said:

Interesting, I start experiencing this after just a few seconds. 

Alright so did my tests.. First question do you have the Dynamix Trim Plugin installed?

 

If you have it installed make sure its running regularly. If you don't make sure you get that. Second under "settings", "global share settings", "enable direct io". I would personally do the trim first test. Then do the direct io second & test. Below is my results.. The problem I ended up hitting is my nvme started to get to hot and thus throttling. It was at 550megs a second which is pretty good transfer speeds. It did crater twice but that was also when it started throttling down pretty hard.

Edited July 9, 2020 by Jerky_san

[kodyorris]

On 7/7/2020 at 9:21 PM, Benson said:

Disk 0 29 are parity disk, SAS disk not support spindown, you may try set those disks "individual" never spindown .

Thanks! I set those drives to never spin down and that seemed to stop those errors. I also moved my cache drives to the on board Sata ports so now Trim is working. However, I still have the same results. The most recent speeds I am seeing is it will start at 116MB/s for a while until ram is full, then sits around 60-70MB/s. 

[kodyorris]

On 7/9/2020 at 1:41 PM, Jerky_san said:

Alright so did my tests.. First question do you have the Dynamix Trim Plugin installed?

 

If you have it installed make sure its running regularly. If you don't make sure you get that. Second under "settings", "global share settings", "enable direct io". I would personally do the trim first test. Then do the direct io second & test. Below is my results.. The problem I ended up hitting is my nvme started to get to hot and thus throttling. It was at 550megs a second which is pretty good transfer speeds. It did crater twice but that was also when it started throttling down pretty hard.

Thanks for the response. I recently moved my cache drives to the MOBO sata ports and now trim is working with Dynamix Trim Plugin. I haven't seen any speed difference unfortunately.  I set Direct IO to Yes and there hasn't been a difference. The most recent speeds I am seeing is it will start at 116MB/s for a while until ram is full, then sits around 60-70MB/s. I attached my latest diagnosis file. 

jarvis-diagnostics-20200711-0016.zip

[kodyorris]

Also, im copying mostly large zipped files, so i dont think its slowing down because its copying lots of small files or anything like that.