johnodon Posted June 4, 2015 Share Posted June 4, 2015 I was going to post this in defect reports but figured I would start here since I don't know if there is an issue or not. unRAID v6rc4 I have 3x 128GB SSDs in a BTRFS cache pool that I use for Docker confguration and VM image storage: In a quick chat with jonp in this thread, we both assumed that I should have 256GB of usual storage space (1 SSD sacrificed for protection): http://lime-technology.com/forum/index.php?topic=40354.msg379861#msg379861 Then I started to doubt myself and did some quick checking. The simplest way I could think of to check used space would be to highlight all files/folders on my cache share and get the properties. I know that this is not exact science but I figured it would be close enough. Here is what it came back with: According to the above, I am using only 65GB of space but yet according to unRAID I have only 87.4GB of free space. Mucho confused! can someone shed some light? John Link to comment
RobJ Posted June 4, 2015 Share Posted June 4, 2015 Here's what I see, but I'm not an expert here. The first picture shows what appears to be a RAID1 array, with 3 128GB SSD's, so their total size would be 128GB, with triple redundancy. If 40.6GB is used, then 87.4GB free would be correct. The second picture shows Windows view of that RAID1 device, and since it does not know about BTRFS COW, it assumes a large used figure of 48.2GB, what would be used if it was on a Windows drive. The BTRFS Copy On Write deduplication has apparently saved you almost 8GB in duplicated storage. Then Windows tries to guess at the size on disk, and it uses what it *thinks* the cluster size is, which if wrong will produce wildly wrong figures. Since you have over 30,000 files, many of them are probably very small, which with a large cluster size, leaves a lot of unused space in the last or only cluster for each file. That explodes the 48.2GB to 65.0GB. Link to comment
garycase Posted June 4, 2015 Share Posted June 4, 2015 Agree with Rob => I think the array is set for triple redundancy, and the numbers shown on the UnRAID display are correct. The Windows display is incorrectly reporting a much higher "Size on Disk" number, since it doesn't know the actual allocation scheme used in the btrfs pool. I don't know if there's a way to configure the pool to have less redundancy but more space ... but if so, I'd still only expect to see 192GB of space, since can't replicate 256GB on a 128GB drive It COULD, if the configuration option is available, ensure all data was replicated on a different drive while still providing 192GB of storage with those 3 drives. Link to comment
trurl Posted June 4, 2015 Share Posted June 4, 2015 Maybe I don't really understand btrfs raid1 cache pools (even though I have one) but it seems to me using the calculator linked in that other linked thread, 192G is the right answer Link to comment
RobJ Posted June 4, 2015 Share Posted June 4, 2015 192GB is what I would have expected, based on what has been discussed with BTRFS usage, but it would not be a true RAID1. Perhaps it should be called RAID-B, because it's a special BTRFS variation. It's my sense that 'RAID-B' creates a RAID0 of all the physical volumes, then mirrors the top half with the bottom half. That would give you a redundant 192GB in this case. Link to comment
garycase Posted June 4, 2015 Share Posted June 4, 2015 192GB is what I would have expected, based on what has been discussed with BTRFS usage, but it would not be a true RAID1. Perhaps it should be called RAID-B, because it's a special BTRFS variation. It's my sense that 'RAID-B' creates a RAID0 of all the physical volumes, then mirrors the top half with the bottom half. That would give you a redundant 192GB in this case. Top half with bottom half wouldn't guarantee the mirrored data was on different disks. I believe what it actually does is effectively create 3 RAID-1 volumes that each contain 1/2 of one drive and 1/2 of another ... and then combines these into a single logical pool. This was all data is stored on two different drives. But regardless of the mechanics, I'd expect 192GB of protected pool storage with 3 128GB drives. Link to comment
johnodon Posted June 4, 2015 Author Share Posted June 4, 2015 So where does this leave us? Do we need official answers from LT? Link to comment
jonp Posted June 5, 2015 Share Posted June 5, 2015 So I was wrong on your free space. It should be only 192GB usable capacity. BTRFS RAID 1 needs to be able to write each block of data to two different devices so that in case any one single device were to fail, 100% of data is still accessible. To do this across three devices will require that each device carries 1/2 of it's own data, 1/2 data from the other two devices. So 3 x 128GB devices would equate to 192GB of usable capacity in a BTRFS RAID 1 group. Now as far as trying to manually calculate the capacity, you'll go insane trying to do that by hand. This is due to the fact that BTRFS doesn't simply store data, but also uses metadata and a global reserve. Your pool should be able to store up to 192GB of total data, but I think we could probably do a better job of showing that in the webGui. Will look into this. Link to comment
Datapotomus Posted June 5, 2015 Share Posted June 5, 2015 The pool information on the first cache link will give you a better idea of what is actually being used. Click your first cache link then scroll down to the pool information. It will look something like the following, but you will have three disks. Label: none uuid: a4a6e12a-79d2-4647-a001-6b3cb6d35389 Total devices 2 FS bytes used 166.25GiB devid 1 size 1.82TiB used 168.03GiB path /dev/sdj1 devid 2 size 1.82TiB used 168.03GiB path /dev/sdk1 btrfs-progs v4.0.1 If all your disks have the same amount used, it might be, because your btrfs pool hasn't been balanced yet. Have you attempted to balance your cache pool? https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices If you go down to the section "Adding new devices". It talks about it. At this point we have a filesystem with two devices, but all of the metadata and data are still stored on the original device(s). The filesystem must be balanced to spread the files across all of the devices. btrfs filesystem balance /mnt You can balance in the GUI. Click your first Cache drive. Go down to balance status. Then click the balance. After you balance it should allocate data correctly. Or, at least I am going under that assumption. You can also do the balance manually according to the docs. https://wiki.archlinux.org/index.php/Btrfs This is down further by RAID features The raid level can be changed while the disks are online using the btrfs balance command: # btrfs balance start -mconvert=RAIDLEVEL -dconvert=RAIDLEVEL /path/to/mount For 2 disk sets, this matches raid levels as defined in md-raid (mdadm). For 3+ disk-sets, the result is entirely different than md-raid. For example: Three 1TB disks in an md based raid1 yields a /dev/md0 with 1TB free space and the ability to safely lose 2 disks without losing data. Three 1TB disks in a Btrfs volume with data=raid1 will allow the storage of approximately 1.5TB of data before reporting full. So for unraid it would be: btrfs balance start -mconvert=raid1 -dconvert=raid1 /mnt/cache Like I said above though. Using the balance button in the webGUI should handle it. Link to comment
johnodon Posted June 5, 2015 Author Share Posted June 5, 2015 What do you make of this: Label: none uuid: f5497a6f-06d0-4702-b941-f18cd1acac53 Total devices 3 FS bytes used 37.79GiB devid 1 size 119.24GiB used 119.19GiB path /dev/sdb1 devid 2 size 3.97MiB used 0.00B path /dev/sdc1 devid 3 size 119.24GiB used 119.19GiB path /dev/sdd1 btrfs-progs v4.0.1 Honestly, the above makes no sense to me whatsoever. According to it, I have .05GB + 3.97MB +.05GB of free space. I'm starting to wonder if the RAID set was not properly created when I inially jumped into v6 (b10 maybe? I would need to check). Link to comment
Datapotomus Posted June 5, 2015 Share Posted June 5, 2015 Can you run this on your system and see what you get? btrfs filesystem df -H /mnt/cache Output should be something like the following. root@NAS:~# btrfs filesystem df -H /mnt/cache Data, RAID1: total=178.24GB, used=177.57GB System, RAID1: total=33.55MB, used=49.15kB Metadata, RAID1: total=2.15GB, used=936.03MB GlobalReserve, single: total=318.77MB, used=0.00B I'm wondering if they metadata might be eating up the additional space. Note, you won't see free space. It really just shows what is being used by data verse metadata. Also, yeah... Your second device doesn't look right at all. The issue might be with it. btrfs might be having issues with seeing the device as a device with the full amount of data available to it. So, it can't balance the data across three disks, because it thinks your second disk is 4 megs inside. Link to comment
johnodon Posted June 5, 2015 Author Share Posted June 5, 2015 root@unRAID:~# btrfs filesystem df -H /mnt/cache Data, RAID1: total=126.87GB, used=40.50GB System, RAID1: total=33.55MB, used=49.15kB Metadata, RAID1: total=1.07GB, used=77.82MB GlobalReserve, single: total=33.55MB, used=0.00B I always thought that 2nd disk looked funky but figured it must just be some artifact from the RAID. Check the below (partition format) for cache2 vs. cache3... Link to comment
Datapotomus Posted June 5, 2015 Share Posted June 5, 2015 I have the same thing with the partition formats. I don't think that is an issue. Ok, so going back to one of my questions. Did you attempt to do a balance (click the balance button on the cache page)? If you do a balance it should reread the metadata, and set things so they dedupe already written extra copies of the data. The balance rereads all data, and metadata, and writes it back out. Link to comment
johnodon Posted June 5, 2015 Author Share Posted June 5, 2015 Running now... Balance on '/mnt/cache' is running 0 out of about 121 chunks balanced (1 considered), 100% left Running I'll report back. Link to comment
johnodon Posted June 5, 2015 Author Share Posted June 5, 2015 Finished...although it says this once it finsihed: No balance found on '/mnt/cache' Here is the device list: Label: none uuid: f5497a6f-06d0-4702-b941-f18cd1acac53 Total devices 3 FS bytes used 37.82GiB devid 1 size 119.24GiB used 42.03GiB path /dev/sdb1 devid 2 size 3.97MiB used 0.00B path /dev/sdc1 devid 3 size 119.24GiB used 42.03GiB path /dev/sdd1 btrfs-progs v4.0.1 Link to comment
Datapotomus Posted June 5, 2015 Share Posted June 5, 2015 I'm really thinking it's coming down to issues with drive 2. Link to comment
johnodon Posted June 5, 2015 Author Share Posted June 5, 2015 Is a good strategy to stop all dockers and VMs and rebuild the pool with a new drive2? The move all of the data back? Link to comment
Datapotomus Posted June 5, 2015 Share Posted June 5, 2015 Yeah, I would run the balance as soon as you rebuild the pool before you put anything on it. Link to comment
johnodon Posted June 7, 2015 Author Share Posted June 7, 2015 I decided to burn my flash drive down and rebuild it since I had some other weird stuff going on. While doing that, I ripped out all 3 SSDs and diskparted (clean all) them in my win8 machine. This looks a hell of a lot better. Link to comment
garycase Posted June 7, 2015 Share Posted June 7, 2015 ... This looks a hell of a lot better. In fact it actually looks RIGHT ... not sure what was going on before, but obviously all's well now. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.