Newly formatted xfs disk, terrible write performance

[fireplex]

Hoping someone can help here, tearing my hair out!!

 

I have been converting my existing disks from resier to xfs, and all has been well.

 

But I am having problems with just one disk (note this is NOT a new disk, it was just resierfs previously).

 

Problem: During rsync writing data back to the disk I noticed it was crawling at kb/s write rate.

 

What I have tried:

1. Removed parity disk to ensure not related to that

2. Tested copying same files using rsync to another drive, to rule out source drive <- get good data rate

3. Ran SMART extended diagnostics overnight <- no errors

4. Changed SATA cable and port being used on motherboard

5. Ran xfs_repair -n  <- nothing reported

 

Note, this disk, /dev/disk3, was fine previously, I have never noticed any issues.

 

Also, the rsync had copied over 1TB data to this disk at a reasonable speed of around 30MB/s prior to it slowing down.

 

During a reboot, while mounting disks, it will hang on this disk for a period of time (unlike my other disks which mount instantly):

Jan  6 09:59:31 Tower  emhttpd: shcmd (10190): mkdir -p /mnt/disk3
Jan  6 09:59:31 Tower  emhttpd: shcmd (10191): mount -t xfs -o noatime,nouuid /dev/md3 /mnt/disk3
Jan  6 09:59:31 Tower kernel: XFS (md3): Mounting V5 Filesystem
Jan  6 10:01:50 Tower kernel: XFS (md3): Ending clean mount
Jan  6 10:01:50 Tower  emhttpd: shcmd (10192): xfs_growfs /mnt/disk3
Jan  6 10:01:50 Tower root: meta-data=/dev/md3               isize=512    agcount=6, agsize=268435455 blks
Jan  6 10:01:50 Tower root:          =                       sectsz=512   attr=2, projid32bit=1
Jan  6 10:01:50 Tower root:          =                       crc=1        finobt=1, sparse=1, rmapbt=0
Jan  6 10:01:50 Tower root:          =                       reflink=1    bigtime=1 inobtcount=1
Jan  6 10:01:50 Tower root: data     =                       bsize=4096   blocks=1465130633, imaxpct=5
Jan  6 10:01:50 Tower root:          =                       sunit=0      swidth=0 blks
Jan  6 10:01:50 Tower root: naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
Jan  6 10:01:50 Tower root: log      =internal log           bsize=4096   blocks=521728, version=2
Jan  6 10:01:50 Tower root:          =                       sectsz=512   sunit=0 blks, lazy-count=1
Jan  6 10:01:50 Tower root: realtime =none                   extsz=4096   blocks=0, rtextents=0

 

 

At this stage, I can only think this is an xfs filesystem related issue....

 

Any ideas guys ?

 

 

 

tower-diagnostics-20230106-1035.zip

tower-smart-20230106-0957.zip

Edited January 6, 2023 by fireplex
Added SMART

[Kilrah]

13 minutes ago, fireplex said:

Also, the rsync had copied over 1TB data to this disk at a reasonable speed of around 30MB/s prior to it slowing down.

That's an SMR drive so there's your "problem", just going to have to be patient / pause the transfer for a couple of hours before resuming.

 

Issue isn't xfs, it's "writing a large amount of data to an SMR drive in one go".

 

  

12 minutes ago, fireplex said:

1. Removed parity disk to ensure not related to that

Note that this will have made your existing parity useless, so you'll have to rebuild it.

Edited January 6, 2023 by Kilrah

[fireplex]

11 minutes ago, Kilrah said:

That's an SMR drive so there's your "problem", just going to have to be patient / pause the transfer for a couple of hours before resuming.

 

I don't believe that is the problem.

 

Disk 7 is also an SMR drive and I had no issues writing data back to that.

 

And I can't believe even SMR should be this slow (as reported by rsync):

Films_My_Rips/HDTV/Interstellar/Interstellar (2014).mkv
  2,473,164,800   6%  117.22kB/s   85:13:41

 

 

[Kilrah]

Note that I added more info to my post.

 

6 minutes ago, fireplex said:

Disk 7 is also an SMR drive and I had no issues writing data back to that.

Will all depend on access patterns, small files vs big files may affect it, etc.

 

6 minutes ago, fireplex said:

And I can't believe even SMR should be this slow (as reported by rsync):

That is precisely typical behavior for an SMR drive that has its CMR cache clogged, it's trying to still accept data while reorganizing things internally and just ends up wasting all its time seeking instead of doing something useful.

If you stop the transfer, give it a few hours to take care of its internal shenanigans and then resume it'll likely be fine for the next few hundred gigs again.

Edited January 6, 2023 by Kilrah

[fireplex]

5 minutes ago, Kilrah said:

If you stop the transfer, give it a few hours to take care of its internal shenanigans and then resume it'll likely be fine for the next few hundred gigs again.

 

Well I've done several reboots and power cycles in trying to troubleshoot this, and whever I restart the transfer it just crawls along immediately.

 

I just cannot see how I can possibly get my TBs of data back onto this disk at this rate 😢

[Kilrah]

6 minutes ago, fireplex said:

Well I've done several reboots and power cycles

That's exactly the opposite of what's needed... The drive has to physically move data, so for it to be able to do so it needs to be powered on and idle, will never have the chance to do it if it's being powered down and again sent data to write after powering back up.

 

6 minutes ago, fireplex said:

I just cannot see how I can possibly get my TBs of data back onto this disk at this rate 😢

The "pause when it grinds to a halt then resume after an hour or 2" might get you there faster, a few hundred gigs at a time. But yeah that's why advice is usually to avoid SMR drives.

They're fine if you gradually add data to them over weeks/months, but a major pain if you have to do a full drive write at some point. 

Edited January 6, 2023 by Kilrah

[JorgeB]

This SMR disk model family from Seagate (all capacities) has been known to have inconsistent performance, some disks perform reasonably well while others can be super slow, but it won't be a filesystem problem.

[fireplex]

Holy crap, didn't realise they could be this bad!

 

Seriously, it's mad. Guess I got off luckily with my disk7 then.

 

So you all believe it's definitely an SMR issue, and my diagnostics look OK?

[JorgeB]

38 minutes ago, fireplex said:

and my diagnostics look OK?

Yep, diags look fine.