August 19, 20241 yr Community Expert Try using a single disk as the array without parity, and retest, if the same, try a different model disk, ideally not SAS.
August 19, 20241 yr Author Alrighty, I'll use one of the laptop HDDs I have loaded in the front of the server later today. In the pool it was acting similarly, but I'll test as the only array device. I'm getting the feeling something got changed in some setting when the large pool was attached, or maybe a bad shutdown changed something, although that doesn't make a lot of sense since everything should be in RAM and the system resides in flash. Is there some way to verify the files on the flash? My next major step is to just redo the entire install. I've done it before in the past and it's not too hard to do. UNRAID makes managing the array, dockers, and VMs quite easy. I'll also try removing my backup USB drive pool. Maybe it's causing some oddity. Edited August 19, 20241 yr by supacupa
August 19, 20241 yr Author 12 hours ago, JorgeB said: Try using a single disk as the array without parity, and retest, if the same, try a different model disk, ideally not SAS. OK, so removing the USB drives did nothing, but making a new array without parity on one of the 2.5 inch HDDs is running at its full speed, something it was not able to do in its own pool when the old array was in. RSYNC from NVME drive to SAS array through FUSE after removing USB pool (Average was around 90 MB/s, under half what the drive is capable of at its current fill level): root@Blade:/mnt# rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/Backup/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 13,288,439,808 4% 90.00MB/s 0:51:57 ^C RSYNC from NVME to SAS array direct after removing USB pool (Average was around 90 MB/s, under half what the drive is capable of at its current fill level): root@Blade:/mnt# rsync -aP /mnt/nvme_drive/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/disk1/Backup/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 13,893,992,448 4% 85.04MB/s 0:54:52 ^C RSYNC from NVME to new single SATA disk (XFS) parityless array through FUSE (100 MB/s is pretty close to the maximum speed of this disk at this fill level, empty): root@Blade:~# rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/test/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 17,351,933,952 5% 100.75MB/s 0:45:45 ^C RSYNC from NVME to new single SATA disk (ZFS compressed) parityless array through FUSE (Less than half the speed of XFS): rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/test/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 9,871,884,288 3% 41.85MB/s 1:53:05 ^C RSYNC from NVME to new single SATA disk (ZFS) parityless array through FUSE (Less than half the speed of XFS, about the same with ZFS compression on): rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/test/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 9,905,471,488 3% 42.94MB/s 1:50:11 ^C RSYNC from NVME to new single SATA disk (BTRFS compressed) parityless array through FUSE (Near maximum speed of this HDD at this fill level, empty): rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/test/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 19,727,220,736 6% 104.44MB/s 0:43:46 ^C RSYNC from NVME to new single SATA disk (BTRFS) parityless array through FUSE (About the same as with BTRFS with compression on): rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/test/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 17,185,210,368 5% 106.21MB/s 0:43:26 ^C RSYNC from NVME drive to SAS array through FUSE after doing above tests. Originally I was getting 90 MB/s, now it's down to 22 MB/s. Same disk, but different directory. This doesn't make any sense at all!: rsync -aP /mnt/user/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/user/test/ sending incremental file list ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 5,268,406,272 1% 22.12MB/s 3:37:20 ^C RSYNC from NVME drive to SAS array direct to disk 3, which is at about the same fill level as disk1, about the same as above: rsync -aP /mnt/nvme_drive/FastMedia/____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar /mnt/disk3/test/ sending incremental file list created directory /mnt/disk3/test ____newbackup___dirbacks___desktop___2023_11_16_21_19_18_testvid.tar 5,454,987,264 1% 22.78MB/s 3:30:53 ^C Edited August 19, 20241 yr by supacupa
August 20, 20241 yr Community Expert The results with the notebook disk are normal, zfs is slower due to the known issue, so it suggests some compatibility issue with the SAS drives.
August 21, 20241 yr Author 22 hours ago, JorgeB said: The results with the notebook disk are normal, zfs is slower due to the known issue, so it suggests some compatibility issue with the SAS drives. They were working fine before, and the problem also showed itself on the XFS array, but I'm copying the entire array to the old XFS drives which have been reformatted to BTRFS to see if that helps now. It'll take a while before I'll know if that makes a difference.
August 23, 20241 yr Author On 8/20/2024 at 2:24 AM, JorgeB said: The results with the notebook disk are normal, zfs is slower due to the known issue, so it suggests some compatibility issue with the SAS drives. Same problem with BTRFS. These drives worked perfectly well before. They work perfectly fine outside of the array. I transferred from the XFS array to these three new disks and then let the parity sync which took a normal amount of time. Something is wrong when it comes to writing with parity enabled. What other things can I check, is reinstalling UNRAID the next step? Maybe the problem is with the beta. Maybe downgrading is what's needed.
August 23, 20241 yr Author Is this normal for a disk write? Seeing several shfs commands when using rsync from the NVME to one of the disks on the array through FUSE. I'm guessing it is, but I'm not sure where to look for the problem at this point. When not going through FUSE: iowait% is fairly high considering this is a 22 core hyperthread machine. avg-cpu: %user %nice %system %iowait %steal %idle 0.4% 0.0% 1.0% 2.9% 0.0% 95.6% Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util loop0 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% loop1 0.60 12.0k 0.00 0.0% 0.00 20.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% loop2 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.80 4.0k 0.00 0.0% 0.00 5.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% loop3 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% nvme0n1 276.00 46.0M 0.00 0.0% 0.35 170.7k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.10 6.6% sda 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdb 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdc 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdd 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sde 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdf 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdg 947.80 60.4M 14481.40 93.9% 17.96 65.2k 785.20 60.4M 14632.40 94.9% 21.34 78.8k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.20 40.00 33.79 96.0% sdh 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdi 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdj 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdk 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdl 944.00 60.3M 14484.20 93.9% 17.66 65.4k 784.60 60.2M 14626.20 94.9% 22.82 78.6k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.20 49.00 34.58 97.1% sdm 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdn 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.60 2.4k 0.00 0.0% 0.00 4.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdo 0.00 0.0k 0.00 0.0% 0.00 0.0k 1.00 4.8k 0.00 0.0% 0.20 4.8k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdp 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.60 3.2k 0.20 25.0% 4.67 5.3k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.1% sdq 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sdr 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% sds 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.80 4.0k 0.00 0.0% 4.75 5.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.2% sdt 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.0k 0.00 0.0% 0.00 0.0k 0.00 0.00 0.00 0.0% Edited August 23, 20241 yr by supacupa
August 24, 20241 yr Author This seems wrong. I'm going to try setting it to 8 in the GUI. root@Blade:~# cat /sys/block/sdl/queue/nr_requests 256
August 24, 20241 yr Author 2 minutes ago, supacupa said: This seems wrong. I'm going to try setting it to 8 in the GUI. root@Blade:~# cat /sys/block/sdl/queue/nr_requests 256 Now we're gettin' somewhere!!
August 24, 20241 yr Author Might not be the right setting, but it made a difference. Going through FUSE is now fast and going directly is slow, which is very strange to me. Any tips on what I should set the tunables to? Perhaps I need to write a script to try out several settings to get optimal performance for both use cases. Edit: Actually, cancel that. After trying again through FUSE it's back to running slow. This isn't making any sense at all. It was just writing at acceptable speeds. Edited August 24, 20241 yr by supacupa
August 24, 20241 yr Community Expert 9 hours ago, supacupa said: Same problem with BTRFS. That's expected, since I think the issue are the drives, or the controller/drives/kernel combination.
August 24, 20241 yr Author 3 hours ago, JorgeB said: That's expected, since I think the issue are the drives, or the controller/drives/kernel combination. How do I go about finding out where the bottleneck is? There has to be some method of debugging this.
August 24, 20241 yr Community Expert Main issue I see is that the problem appears to not be always the same, i.e., you sometimes get good performance with the SAS drives, other times not without anything changing.
August 24, 20241 yr Author Solution 5 hours ago, JorgeB said: Main issue I see is that the problem appears to not be always the same, i.e., you sometimes get good performance with the SAS drives, other times not without anything changing. Agreed, it doesn't make sense. It's not the drives since this problem shows itself on other SAS drives. It's also not likely to be the controller as the problems showed itself on two separate controllers and I've not seen others report this problem on either of them. After a lot of testing and changing things out it's clear the problem is following the server and the UNRAID install. The server has not changed, but the UNRAID install has, so that is the most likely culprit. This same drive and controller setup works find on my EndeavourOS PC which is currently using 6.10.4-zen2-1-zen kernel, and my Debian server which is currently using the 6.9.7+bpo-amd64 kernel. If the 6.8.12-Unraid kernel is to blame there's not much I can do about that other than rollback and stick with slowly aging software with the hope some future update fixes it. I do firmware development so I'm familiar with figuring this kind of problem on bare metal, but I don't really know where to start with Linux unless I can find information online on what component to debug, something I've had some trouble with doing on UNRAID. If something isn't doing what it should be doing it should be possible to trace it. Any ideas on where I can start? Any advice will be appreciated. I need to have somewhat reliable speed which I did have before all this started. It was working perfectly fine on the previous UNRAID version and had been for many years. I was using these exact same drives with this exact same controller about three years ago. I switched to the older LSI card since I had an expander and could run more HDDs, but as I no longer have a need for hundreds of TBs of storage I simplified it to 4 primary 12 TB disks using the rest as cold storage. I primarily use UNRAID for VMs and Dockers. The UNRAID method of data storage and backup was nice back when I had an assortment of disks, but now that I have dozens of the same types of disks with many spares, it may not be the best method of data storage for me now, at least not when its this inconsistent. My current options seem to be as follows: 1. Find and solve the problem so I can continue using UNRAID as I do. That may involve rolling back to an older UNRAID version, which may mean I no longer can update UNRAID which may or may not be a problem. 2. Perhaps attempt to pass the HBA through to a VM and use RAIDZ pools for data storage and let UNRAID focus on only VMs and Dockers. This is a somewhat silly option, but I could see it working. Probably not worth the headache, though. 3. Install another OS that works correctly and run Dockers and VMs the old fashioned way. When I started using UNRAID something like 4 or 5 years ago I didn't have the time to do that, but I've now have other servers I manage and have figured out better ways to do so, so this might be the best option. I've already spent way too much time on this problem and am worried more compatibility issues may crop up in future UNRAID builds. Edited August 24, 20241 yr by supacupa
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.