Large copy/write on btrfs cache pool locking up server temporarily


Recommended Posts

  • 1 month later...

So ...  Samsung devices have a different NAND Erase block size; which I think is causing us all some extra headaches.  The starting block 64 isn't right for them and thats how they get partitioned in unraid and unassigned devices.  Check my findings from that thread: 

 

 

The testings in the link above are to a single 500GB and a 250GB samsung 840.  Both cleaned up after changing the starting block.  Do keep in mind I have a unique setup with 256gb of ram; so my writes are in large blocks from the ram cache.

 

As for the BTRFS pool...

I've mixed in a non-samsung (SANDISK) SSD into my btrfs pool and it's really interesting to see how the two drives work differently.  The Samsung will start causing huge IO wait times and it'll continue to write to the samsung upwards of 10 minutes after the sandisk is finished.  I am going to try and use GPARTED to move the starting block over on the samsung and see if unraid will let it still work in the btrfs pool.  I'll report back hopefully later tonight on that.

 

 

 

 

 

 

 

Edited by dnoyeb
Link to comment
4 minutes ago, dnoyeb said:

I am going to try and use GPARTED to move the starting block over on the samsung and see if unraid will let it still work in the btrfs pool.

It won't, unRAID requires that the partition starts on sector 64, only way is to use them with a different starting sector is as unassigned devices.

Link to comment
1 hour ago, johnnie.black said:

It won't, unRAID requires that the partition starts on sector 64, only way is to use them with a different starting sector is as unassigned devices.

 

This isn't a requirement of BTRFS raid...  I'm seeing where people use other starting blocks; so I guess this is an issue with unraid honestly and it forcing us to use sector 64.

 

I'm still going to play around and see what happens; since EVERYONE was so positive that changing the starting block wouldn't have any impact on my other tests and I was able to reproduce it 4 times with 2 different samsung devices...

 

 

Link to comment
18 hours ago, thomast_88 said:

Good findings @dnoyeb this has been bugging me for a year now. Hoping for a fix to this issue :)

 

Yea, I filled a feature request on it.  Until then I am swapping over to a single SSD and just doing nightly backups of all of my docker data and will try and do a weekly backup of the VM's.  

 

One thing I noticed when moving down to the single SSD on XFS; load averages while moving say, two 70GB files back to back from another SSD to it; my load is hitting max of 5.  When it was a BTRFS pool of two SSD's it was hitting 15-16.  IOwait levels staying much lower as well.  

Edited by dnoyeb
  • Like 1
Link to comment
4 hours ago, thomast_88 said:

It's more a bug report than a feature request :D

 

Haha yea... I've got a great flow now; files get downloaded to a Samsung 840 evo xfs formatted with the proper alignment which is mounted via unassigned devices as the temporary location; then they're extracted to the cache 1tb sandisk xfs drive (which works with the default 64 sector offset).  I went from issues where extracting a 70gb file would spike load to 15+ down to around 4-5.  Amazing how much faster and more efficient back to back downloads are now too.  Pulled down 3 70GB files all back to back with repairs and unencryption; which normally would have thrashed my system.. It didn't break a sweat.

 

System is a 32 core dual xeon with 256gb of ram; so do keep that in mind too.  

Link to comment

I kicked it over to 2048 which seems to work well.  I've read that you should do it in multiples of 1536 however, so in Gparted if you kicked it to 3mb, you'd be set.  

 

If I pull the drive out again, I'll probably kick it to 3mb instead.  I've got some more details over in that other thread (unassigned devices) showing how much it impacted my copies.  I'm EXTREMELY happy now with the performance of doing large scale downloads...  I'll probably test with 4x70gb downloads today and see how it handles it....  I'm on gigabit, so i'll routinely pull down at full gigabit; which really is a great test.

Link to comment
  • 2 weeks later...
5 minutes ago, thomast_88 said:

But then you don't have the possiblity to do Raid1 or Raid0 ?

 

True. But my 750GB SSD is fast enough that I don't feel the need to tinker with this until BTRFS becomes more mainstream and irons out some things. Rest of my drives are all BTRFS which I will slowly convert to XFS, and add another parity drive. 

Link to comment
6 hours ago, daze said:

 

True. But my 750GB SSD is fast enough that I don't feel the need to tinker with this until BTRFS becomes more mainstream and irons out some things. Rest of my drives are all BTRFS which I will slowly convert to XFS, and add another parity drive. 

With the BTRFS fixes introduced about a year ago in the kernel, I find single-disk BTRFS work very well. I have a couple of thousand installations in automotive environment with BTRFS that have worked very well.


But before the kernel fixes, single-disk BTRFS was very unstable - after an unexpected power loss you really couldn't trust it to come up to a stable state. Given the time LT spends picking up kernel patches, I would assume current unRAID versions has all BTRFS fixes.

 

I'm not sure if multi-disk BTRFS has also reached a reasonably stable state. There are quite a lot of people who seems to have issues with their multidisk caches. The RAID5/RAID6 support in BTRFS is most definitely not commercial grade.

Link to comment
On 3/26/2018 at 6:11 AM, pwm said:

With the BTRFS fixes introduced about a year ago in the kernel, I find single-disk BTRFS work very well. I have a couple of thousand installations in automotive environment with BTRFS that have worked very well.


But before the kernel fixes, single-disk BTRFS was very unstable - after an unexpected power loss you really couldn't trust it to come up to a stable state. Given the time LT spends picking up kernel patches, I would assume current unRAID versions has all BTRFS fixes.

 

I'm not sure if multi-disk BTRFS has also reached a reasonably stable state. There are quite a lot of people who seems to have issues with their multidisk caches. The RAID5/RAID6 support in BTRFS is most definitely not commercial grade.

pwm -- 

Thank you for that perspective. I may have jumped the gun and converted back to XFS for little reason. And if you're curious, when I ran the balance command on the cache drive, I saw it churn a lot and saw Used space decrease. And another factor, albeit wrong, was that my Docker image was full which I wrongly attributed to BTRFS. 

 

Thank you for the learnings.

Link to comment
  • 3 months later...
  • 2 weeks later...

I have a couple non Samsung's btrfs raid 1 and it still bogs the system down a bit. Not unusable but noticeable when trying to do things. Netdata shows between 7=10% iowait with mover going and spiking to 25%. Typically don't even worry about it due to the run late at night, but now after reading about some of this makes me want to raid 0 it or go to xfs and add the additional to unassigned devices for a VM.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.