Currently I am changing the filesystems of the array drive to (single) zfs.

So I've bought a new drive (not enough room to free one existing disk, best combination yielded 101% usage), formatted it with zfs and used mc to move one other disk to this new one. (disabled parity before the copy, wanted a fast move)

This took 3 days and gave a copy speed of 33mb/s 😞

 

The now freed old drive got reformatted with zfs and I moved ("move", its still in progress) the next xfs drive to this. Now using the gui.

But again:

 

(besides the 98% and ETA are just joking, the speed is for real and makes me very sad)

 

So, before this is finished and there are still 2 drives to go afterwards, I better ask if somebody has an idea how i can kick UNRAID to write and not to sleep?

CPU Usage often goes to 100% for one core, so I guess, IO is satuarted and blocking. But with 30MB/s ? The disks should be yawning.

A parity check usually runs with >200MB/s average here.

 

Without parity should be much faster, you can post diags to see if there's something there.

No big deal, but I dont see anything in there to worry about and the box runs very well and servs users normally (full speed)

f-diagnostics-20230709-1327.zip

Nothing that I can see, what are you using to copy the files? Any faster with rsync? e.g.:

 

rsync -av --progress /mnt/diskx/share/ /mnt/disky/share/

 

As i said above
The firrst try was done with mc, now the 2nd run done from the gui (builtin filemanager)

I am open to try rsync on try 3 one day😁

I've had a similar issue, although I'm getting speeds of 60-70MB/s when the disks in DiskSpeed app are capable of 250-130MB/s reads/writes (start and end of disk speeds respectively).

 

I moved all my data onto one XFS disk, which freed up some disks to format as ZFS. I've been using ZFS single disks just purely for the dataset capabilities but when moving a share using unBalance from one ZFS disk to another, the speed is roughly 66MB/s, even with Parity disabled for speed.

 

I've formatted one of the ZFS disks that was "slow" to read and write from as XFS, and now copying or moving from another ZFS disk to the XFS one is giving me a boost startup speed of 250MB/s and it is currently at 140MB/s having copied about 200GB already.

 

Not sure what's going on but it seems single ZFS disks are super slow, which explains why my Parity rebuilt took almost 2-3 days going at 60-70MB/s at max.

 

Going to switch back to XFS at least for the array and do more testing with ZFS single vdevs.

Reading from ZFS (from an already copied disk) still gives me the usual 260-270MB/s, just the local copy is so slow...

Have to check if there is a write problem, but then... there is a cache in front of the disk, so it is hard to tell.

 

No cache in front of mine (ZFS disks). I was doing the copy from /mnt/diskN to /mnt/diskN+1 using the GUI

 

Will try a copy from XFS back to ZFS to see what the write speed is, but I'm sure it was slow. I didn't catch on to it thinking it was because of the parity drive was there, but as you have realised, the parity might slow things down, but not that slow.

 

Either way if anyone else is experiencing the slowness at least they know what to try, to see if the speed comes back.

1 minute ago, vayidm said:

No cache in front of mine (ZFS disks)

The cache is only there if you access the share from windows.

 

I copy /mnt/diskX to /mnt/diskY.

 

This does not invoke the cache (or?? am I wrong ?? some of the admins might enlight us. Hey, JorgeB 😁! )

 

Ah yes I see what you mean, the memory cache? I haven't tried copying data from a share via Windows. It'll add too many variables into the mix and the main issue is really from ZFS disk to ZFS disk, with and without parity on. I haven't tried copying from share to another share yet.

 

Most likely if the performance is that bad on the machine, it won't get better going over the network, but who knows. Either way I'm going to go back to XFS for now until i do more testing and ZFS at least performs the same when using it in a single disk pool.

1 hour ago, vayidm said:

Ah yes I see what you mean, the memory cache?

nono, the usual UNRAID "cache". I could turn it off for the shares and test then. but too lazy.

The thing is, that the internal copy from UNRAID (either shell or gui) is so slow whilest external access is normally fast. Looks like UNRAID has a builtin brake that is kicking in for some unknown reason.

 

I forgot about this issue:

 

 

you can confirm if it's this by using rsync and limiting speed to a little less than max disk speed.

Will give that a test. I've found it occurs from disk to disk with or without parity set up, when the disks are ZFS singles.

5 hours ago, JorgeB said:

you can confirm if it's this by using rsync and limiting speed to a little less than max disk speed.

How do I do this?

 

btw, the 3rd "session" is going on, now with rsync as you have said. It started quite fast (220 MB/s) but also dropped quite fast again too and now settles around 60-70MB.

 

So the ranking currently is:

MC move: 33MB/s

GUI move: 66MB/s

Rsync (copy?): 65MB/s

 

Working but still disappointing speed. The current disk is 97% (17,4TB) full...

 

I cannot find a good manual page for rsync, I notice that it only copies and does not skip existing files (I guess there should be a way to tell him to move and not touch existing files unless they differ in certain aspects). And, of course, I have no idea how to speed limit it.

 

Another thing I've noticed: with mc or gui one random and changeging VCPU was at 100% (io blocking?) now with rsync it is always the same CPU that is occupied by the job.

 

21 hours ago, MAM59 said:

I copy /mnt/diskX to /mnt/diskY.

 

This does not invoke the cache

No pools (such as cache) are involved when copying from one array disk to another.

 

21 hours ago, vayidm said:

the memory cache?

16 minutes ago, MAM59 said:

started quite fast (220 MB/s)

 

Linux buffers all writes to RAM so initial burst is expected until RAM buffer fills.

27 minutes ago, MAM59 said:

How do I do this?

See the linked thread above.

1 hour ago, JorgeB said:

See the linked thread above.

loool, I thought "nice graphics" and did not notice it was a link to something :-)))

 

Ok, I've killed the old rsync job, added -t and --bwlimit=170000 (some MB/s below the slowest speed parity check tells me all the time) and fired it off again.

It instantly jumped to the last file where I have interrupted the last job and continued from there on (nice 🙂 )

Speed was 160+MB/s and I went off for my daily bike trip.

Now back after an hour or so and guess what? speed is still at 160+ !!!

 

Obviously that "trick" is working!. (obviously too there is a serious bug that needs to be fixed by limetech soon)

 

tnx! you've saved me some boring days of waiting (still about 3 days to go, but better than 9)

 

31 minutes ago, MAM59 said:

(obviously too there is a serious bug that needs to be fixed by limetech soon)

Yep, LT is aware of this issue, though no idea for now when to expect the fix.

Cheered too early 😞

 

Although speed was still high when I went to bed yesterday, this morning it was back down to 60MB/s only.

 

Killed the rsync job and restarted it with 150000 limit, instantly went back up to 160MB/s.

 

Either my 160000 guess was too optimistic or there is still a bug lurking that only shows up after a lots of TB transferred...

 

Update: 😞

The longer it takes, the faster it drops and the slower it gets.

 

I'm now down to 50MB/s already after 30mins after a new restart 😞

 

Seems that the bandwith throttle is only part of the solution. I imagine, it takes forever now to look for free space. The disk is now about 50% full (still 47% to copy...)

 

3 hours ago, MAM59 said:

Although speed was still high when I went to bed yesterday, this morning it was back down to 60MB/s only.

Don't forget that disks slow down as they fill up, so you will likely need to adjust the speed limit.

1 minute ago, JorgeB said:

Don't forget that disks slow down as they fill up, so you will likely need to adjust the speed limit.

I'm already down to 100000, this is much less than the drive can handle

I was patient and waited for the copy to finish. It went down to 40MB/s at the end 😞

Now with ZFS in place, performance is even worse:

Before with XFS I got full 1G/s transfer to cache drive. Now with ZFS cache is ignored (the full speed at the beginning seems to be ARC getting filled) and write speed is depressingly low.

I will test for some more days, but I guess, ZFS won't last long here...

 

(Reading is not speed-limited, its a write-only problem)

 

Additional Info: ARC was almost empty before I started to copy the file. After the copy ARC grew almost exactly the amount of that file (15G). So the transfer was done to RAM, but slowed down like hell from something I don't know. Filling ARC seems to be not sufficient to say "packet accepted and done". (or is ARC only a read cache???) Reading back the file to a different machine runs at 600MB/s, not the optimum, but sufficient speed I would say. So ARC is functioning for reading.

 

 

Edited July 13, 20232 yr by MAM59

3 hours ago, MAM59 said:

Before with XFS I got full 1G/s transfer to cache drive. Now with ZFS cache is ignored

Don't follow, if the share is correctly configured it will still right to cache.

31 minutes ago, JorgeB said:

Don't follow, if the share is correctly configured it will still right to cache.

YES&NO 🙂

Its always throwing a dice if you overwrite an existing file. Sometime it deletes it before and the write goes to the cache, sometimes it overwrites the file in the array directly bypassing the cache. One could prevent it by explicitly deleting the file before, but beeing lazy I often forget and get punished...

 

So I am not alarmed THAT it happens, but I am really shocked about "how deep can one drop?" the ~30MB/s of the write. That is really unacceptable.

 

Edited July 13, 20232 yr by MAM59