7 minutes ago, thomast_88 said:

@johnnie.black can you provide a status after running for a few months?  

 

The issue of the filesystem getting fully allocated on SSDs (resulting in trim not working on the drive) seems to be resolved, this was one of the contributing factors for the slowdown, but possibly not the only one, I myself never noticed that issue.

I haven't had the allocation SSD issue happen since the upgrade. I think its resolved. at least i haven't seen less than 112Mbps transfer speeds so i haven't had a need to manually run rebalance.

I'm having the same issues with 2x512GB Samsung 850 Pros with BTFRS.  I've changed cables and ports.  No luck.  Running 6.4.1.

On 2/9/2018 at 4:41 AM, defiant said:

I'm having the same issues with 2x512GB Samsung 850 Pros with BTFRS.  I've changed cables and ports.  No luck.  Running 6.4.1.

 

Bummer :-( Wonder if this issue caught the attention of Limetech. It really breaks the ability to run a BTRFS cache array.

 

Are you running raid 0 or 1 @defiant ?

i use to run this command when i had cache fs full issues, it was because of allocated sectors in btrfs.

It's not so much an Unraid thing its more btrfs itself.

 

btrfs balance start -dusage=75 /mnt/cache

 

If that fixes it run a cron every few days.

So ...  Samsung devices have a different NAND Erase block size; which I think is causing us all some extra headaches.  The starting block 64 isn't right for them and thats how they get partitioned in unraid and unassigned devices.  Check my findings from that thread: 

 

 

The testings in the link above are to a single 500GB and a 250GB samsung 840.  Both cleaned up after changing the starting block.  Do keep in mind I have a unique setup with 256gb of ram; so my writes are in large blocks from the ram cache.

 

As for the BTRFS pool...

I've mixed in a non-samsung (SANDISK) SSD into my btrfs pool and it's really interesting to see how the two drives work differently.  The Samsung will start causing huge IO wait times and it'll continue to write to the samsung upwards of 10 minutes after the sandisk is finished.  I am going to try and use GPARTED to move the starting block over on the samsung and see if unraid will let it still work in the btrfs pool.  I'll report back hopefully later tonight on that.

 

 

 

 

 

 

 

Edited March 12, 20188 yr by dnoyeb

4 minutes ago, dnoyeb said:

I am going to try and use GPARTED to move the starting block over on the samsung and see if unraid will let it still work in the btrfs pool.

It won't, unRAID requires that the partition starts on sector 64, only way is to use them with a different starting sector is as unassigned devices.

1 hour ago, johnnie.black said:

It won't, unRAID requires that the partition starts on sector 64, only way is to use them with a different starting sector is as unassigned devices.

 

This isn't a requirement of BTRFS raid...  I'm seeing where people use other starting blocks; so I guess this is an issue with unraid honestly and it forcing us to use sector 64.

 

I'm still going to play around and see what happens; since EVERYONE was so positive that changing the starting block wouldn't have any impact on my other tests and I was able to reproduce it 4 times with 2 different samsung devices...

 

 

1 minute ago, dnoyeb said:

This isn't a requirement of BTRFS raid.

Never said it was, it's an unRAID requirement (for data and cache drives)

3 minutes ago, johnnie.black said:

Never said it was, it's an unRAID requirement (for data and cache drives)

 

Seems like something that should be variable for the cache drives using BTRFS.

 

Anyone know where we file a feature request?  

1 minute ago, dnoyeb said:

 

Seems like something that should be variable for the cache drives using BTRFS.

 

Anyone know where we file a feature request?  

https://lime-technology.com/forums/forum/53-feature-requests/

Good findings @dnoyeb this has been bugging me for a year now. Hoping for a fix to this issue

18 hours ago, thomast_88 said:

Good findings @dnoyeb this has been bugging me for a year now. Hoping for a fix to this issue

 

Yea, I filled a feature request on it.  Until then I am swapping over to a single SSD and just doing nightly backups of all of my docker data and will try and do a weekly backup of the VM's.  

 

One thing I noticed when moving down to the single SSD on XFS; load averages while moving say, two 70GB files back to back from another SSD to it; my load is hitting max of 5.  When it was a BTRFS pool of two SSD's it was hitting 15-16.  IOwait levels staying much lower as well.  

Edited March 13, 20188 yr by dnoyeb

It's more a bug report than a feature request 

4 hours ago, thomast_88 said:

It's more a bug report than a feature request 

 

Haha yea... I've got a great flow now; files get downloaded to a Samsung 840 evo xfs formatted with the proper alignment which is mounted via unassigned devices as the temporary location; then they're extracted to the cache 1tb sandisk xfs drive (which works with the default 64 sector offset).  I went from issues where extracting a 70gb file would spike load to 15+ down to around 4-5.  Amazing how much faster and more efficient back to back downloads are now too.  Pulled down 3 70GB files all back to back with repairs and unencryption; which normally would have thrashed my system.. It didn't break a sweat.

 

System is a 32 core dual xeon with 256gb of ram; so do keep that in mind too.  

7 hours ago, dnoyeb said:

a Samsung 840 evo xfs formatted with the proper alignment

 

What did you decide was the best starting sector to use, if not 64?

I kicked it over to 2048 which seems to work well.  I've read that you should do it in multiples of 1536 however, so in Gparted if you kicked it to 3mb, you'd be set.  

 

If I pull the drive out again, I'll probably kick it to 3mb instead.  I've got some more details over in that other thread (unassigned devices) showing how much it impacted my copies.  I'm EXTREMELY happy now with the performance of doing large scale downloads...  I'll probably test with 4x70gb downloads today and see how it handles it....  I'm on gigabit, so i'll routinely pull down at full gigabit; which really is a great test.

I've reached the conclusion that formatting my drives BTRFS was a mistake.

But then you don't have the possiblity to do Raid1 or Raid0 ?

5 minutes ago, thomast_88 said:

But then you don't have the possiblity to do Raid1 or Raid0 ?

 

True. But my 750GB SSD is fast enough that I don't feel the need to tinker with this until BTRFS becomes more mainstream and irons out some things. Rest of my drives are all BTRFS which I will slowly convert to XFS, and add another parity drive. 

6 hours ago, daze said:

 

True. But my 750GB SSD is fast enough that I don't feel the need to tinker with this until BTRFS becomes more mainstream and irons out some things. Rest of my drives are all BTRFS which I will slowly convert to XFS, and add another parity drive. 

With the BTRFS fixes introduced about a year ago in the kernel, I find single-disk BTRFS work very well. I have a couple of thousand installations in automotive environment with BTRFS that have worked very well.

But before the kernel fixes, single-disk BTRFS was very unstable - after an unexpected power loss you really couldn't trust it to come up to a stable state. Given the time LT spends picking up kernel patches, I would assume current unRAID versions has all BTRFS fixes.

 

I'm not sure if multi-disk BTRFS has also reached a reasonably stable state. There are quite a lot of people who seems to have issues with their multidisk caches. The RAID5/RAID6 support in BTRFS is most definitely not commercial grade.

On 3/26/2018 at 6:11 AM, pwm said:

With the BTRFS fixes introduced about a year ago in the kernel, I find single-disk BTRFS work very well. I have a couple of thousand installations in automotive environment with BTRFS that have worked very well.

But before the kernel fixes, single-disk BTRFS was very unstable - after an unexpected power loss you really couldn't trust it to come up to a stable state. Given the time LT spends picking up kernel patches, I would assume current unRAID versions has all BTRFS fixes.

 

I'm not sure if multi-disk BTRFS has also reached a reasonably stable state. There are quite a lot of people who seems to have issues with their multidisk caches. The RAID5/RAID6 support in BTRFS is most definitely not commercial grade.

pwm -- 

Thank you for that perspective. I may have jumped the gun and converted back to XFS for little reason. And if you're curious, when I ran the balance command on the cache drive, I saw it churn a lot and saw Used space decrease. And another factor, albeit wrong, was that my Docker image was full which I wrongly attributed to BTRFS. 

 

Thank you for the learnings.

There's been a few releases since the last post, but I don't think anything has been fixed yet correct? Any idea if/when that will happen? Debating whether to keep chugging away with my BTRFS RAID1 cache pool or just split it and format as XFS.

@deusxanime no it has not been fixed, and I don't think it's on the roadmap either. There have been no input from limetech guys in this post, and it doesn't seem like there is a workaround. I'd stick with XFS if you have Samsung Evo discs.

Edited July 5, 20188 yr by thomast_88

I have a couple non Samsung's btrfs raid 1 and it still bogs the system down a bit. Not unusable but noticeable when trying to do things. Netdata shows between 7=10% iowait with mover going and spiking to 25%. Typically don't even worry about it due to the run late at night, but now after reading about some of this makes me want to raid 0 it or go to xfs and add the additional to unassigned devices for a VM.