I have my 9 disks formatted as BTRFS and compression set on all of them in the Array. The more disks I add the longer it gets, it takes about 14 minutes to start the array.

 

Is it normal to have a long array mounting time with BTRFS? And if so did I make a mistake by choosing it over XFS?

 

I have the logs below and it looks like it mounts each BTRFS device one by one instead of all at the same time

 

Quote

Aug 28 22:38:25 Unraid unraid-api[9413]: ✔️ UNRAID API started successfully!
Aug 28 22:38:27 Unraid flash_backup: start watching for file changes
Aug 28 22:38:50 Unraid SysDrivers: SysDrivers Build Complete
Aug 28 22:39:02 Unraid webGUI: Successful login user root from 192.168.0.107
Aug 28 22:39:27 Unraid kernel: mdcmd (31): set md_num_stripes 1280
Aug 28 22:39:27 Unraid kernel: mdcmd (32): set md_queue_limit 80
Aug 28 22:39:27 Unraid kernel: mdcmd (33): set md_sync_limit 5
Aug 28 22:39:27 Unraid kernel: mdcmd (34): set md_write_method
Aug 28 22:39:27 Unraid kernel: mdcmd (35): start STOPPED
Aug 28 22:39:27 Unraid kernel: unraid: allocating 56710K for 1280 stripes (11 disks)
Aug 28 22:39:27 Unraid kernel: md1p1: running, size: 17578328012 blocks
Aug 28 22:39:27 Unraid kernel: md2p1: running, size: 3907018532 blocks
Aug 28 22:39:27 Unraid kernel: md3p1: running, size: 13672382412 blocks
Aug 28 22:39:27 Unraid kernel: md4p1: running, size: 13672382412 blocks
Aug 28 22:39:27 Unraid kernel: md5p1: running, size: 17578328012 blocks
Aug 28 22:39:27 Unraid kernel: md6p1: running, size: 19531825100 blocks
Aug 28 22:39:27 Unraid kernel: md7p1: running, size: 17578328012 blocks
Aug 28 22:39:27 Unraid kernel: md8p1: running, size: 19531825100 blocks
Aug 28 22:39:27 Unraid kernel: md9p1: running, size: 19531825100 blocks
Aug 28 22:39:27 Unraid emhttpd: shcmd (107): udevadm settle
Aug 28 22:39:28 Unraid emhttpd: Opening encrypted volumes...
Aug 28 22:39:28 Unraid emhttpd: shcmd (108): touch /boot/config/forcesync
Aug 28 22:39:28 Unraid emhttpd: Mounting disks...
Aug 28 22:39:28 Unraid emhttpd: mounting /mnt/disk1
Aug 28 22:39:28 Unraid emhttpd: shcmd (109): mkdir -p /mnt/disk1
Aug 28 22:39:28 Unraid emhttpd: shcmd (110): mount -t btrfs -o noatime,space_cache=v2 /dev/md1p1 /mnt/disk1
Aug 28 22:39:28 Unraid kernel: BTRFS info (device md1p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:39:28 Unraid kernel: BTRFS info (device md1p1): using free space tree
Aug 28 22:40:28 Unraid flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update
Aug 28 22:40:49 Unraid emhttpd: shcmd (111): btrfs filesystem resize 1:max /mnt/disk1
Aug 28 22:40:49 Unraid kernel: BTRFS info (device md1p1): resizing devid 1
Aug 28 22:40:49 Unraid root: Resize device id 1 (/dev/md1p1) from 16.37TiB to max
Aug 28 22:40:49 Unraid emhttpd: mounting /mnt/disk2
Aug 28 22:40:49 Unraid emhttpd: shcmd (112): mkdir -p /mnt/disk2
Aug 28 22:40:50 Unraid emhttpd: shcmd (113): mount -t btrfs -o noatime,space_cache=v2 /dev/md2p1 /mnt/disk2
Aug 28 22:40:50 Unraid kernel: BTRFS info (device md2p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:40:50 Unraid kernel: BTRFS info (device md2p1): using free space tree
Aug 28 22:41:12 Unraid emhttpd: shcmd (114): btrfs filesystem resize 1:max /mnt/disk2
Aug 28 22:41:12 Unraid kernel: BTRFS info (device md2p1): resizing devid 1
Aug 28 22:41:12 Unraid root: Resize device id 1 (/dev/md2p1) from 3.64TiB to max
Aug 28 22:41:12 Unraid emhttpd: mounting /mnt/disk3
Aug 28 22:41:12 Unraid emhttpd: shcmd (115): mkdir -p /mnt/disk3
Aug 28 22:41:13 Unraid emhttpd: shcmd (116): mount -t btrfs -o noatime,space_cache=v2 /dev/md3p1 /mnt/disk3
Aug 28 22:41:13 Unraid kernel: BTRFS info (device md3p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:41:13 Unraid kernel: BTRFS info (device md3p1): using free space tree
Aug 28 22:42:25 Unraid nginx: 2024/08/28 22:42:25 [error] 9266#9266: *76 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.0.107, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "192.168.0.110", referrer: "http://192.168.0.110/Main"
Aug 28 22:42:30 Unraid kernel: BTRFS info (device md3p1): start tree-log replay
Aug 28 22:42:32 Unraid emhttpd: shcmd (117): btrfs filesystem resize 1:max /mnt/disk3
Aug 28 22:42:32 Unraid root: Resize device id 1 (/dev/md3p1) from 12.73TiB to max
Aug 28 22:42:32 Unraid kernel: BTRFS info (device md3p1): resizing devid 1
Aug 28 22:42:32 Unraid emhttpd: mounting /mnt/disk4
Aug 28 22:42:32 Unraid emhttpd: shcmd (118): mkdir -p /mnt/disk4
Aug 28 22:42:32 Unraid emhttpd: shcmd (119): mount -t btrfs -o noatime,space_cache=v2 /dev/md4p1 /mnt/disk4
Aug 28 22:42:32 Unraid kernel: BTRFS info (device md4p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:42:32 Unraid kernel: BTRFS info (device md4p1): using free space tree
Aug 28 22:43:53 Unraid emhttpd: shcmd (120): btrfs filesystem resize 1:max /mnt/disk4
Aug 28 22:43:53 Unraid root: Resize device id 1 (/dev/md4p1) from 12.73TiB to max
Aug 28 22:43:53 Unraid kernel: BTRFS info (device md4p1): resizing devid 1
Aug 28 22:43:53 Unraid emhttpd: mounting /mnt/disk5
Aug 28 22:43:53 Unraid emhttpd: shcmd (121): mkdir -p /mnt/disk5
Aug 28 22:43:54 Unraid emhttpd: shcmd (122): mount -t btrfs -o noatime,space_cache=v2 /dev/md5p1 /mnt/disk5
Aug 28 22:43:54 Unraid kernel: BTRFS info (device md5p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:43:54 Unraid kernel: BTRFS info (device md5p1): using free space tree
Aug 28 22:44:58 Unraid kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Aug 28 22:44:58 Unraid kernel: nvidia-uvm: Loaded the UVM driver, major device number 240.
Aug 28 22:45:22 Unraid emhttpd: shcmd (123): btrfs filesystem resize 1:max /mnt/disk5
Aug 28 22:45:22 Unraid root: Resize device id 1 (/dev/md5p1) from 16.37TiB to max
Aug 28 22:45:22 Unraid kernel: BTRFS info (device md5p1): resizing devid 1
Aug 28 22:45:22 Unraid emhttpd: mounting /mnt/disk6
Aug 28 22:45:22 Unraid emhttpd: shcmd (124): mkdir -p /mnt/disk6
Aug 28 22:45:23 Unraid emhttpd: shcmd (125): mount -t btrfs -o noatime,space_cache=v2 /dev/md6p1 /mnt/disk6
Aug 28 22:45:23 Unraid kernel: BTRFS info (device md6p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:45:23 Unraid kernel: BTRFS info (device md6p1): using free space tree
Aug 28 22:47:10 Unraid emhttpd: shcmd (126): btrfs filesystem resize 1:max /mnt/disk6
Aug 28 22:47:10 Unraid root: Resize device id 1 (/dev/md6p1) from 18.19TiB to max
Aug 28 22:47:10 Unraid kernel: BTRFS info (device md6p1): resizing devid 1
Aug 28 22:47:10 Unraid emhttpd: mounting /mnt/disk7
Aug 28 22:47:10 Unraid emhttpd: shcmd (127): mkdir -p /mnt/disk7
Aug 28 22:47:11 Unraid emhttpd: shcmd (128): mount -t btrfs -o noatime,space_cache=v2 /dev/md7p1 /mnt/disk7
Aug 28 22:47:11 Unraid kernel: BTRFS info (device md7p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:47:11 Unraid kernel: BTRFS info (device md7p1): using free space tree
Aug 28 22:47:11 Unraid kernel: BTRFS info (device md7p1): bdev /dev/md7p1 errs: wr 0, rd 0, flush 0, corrupt 3498, gen 0
Aug 28 22:49:07 Unraid emhttpd: shcmd (130): btrfs filesystem resize 1:max /mnt/disk7
Aug 28 22:49:07 Unraid root: Resize device id 1 (/dev/md7p1) from 16.37TiB to max
Aug 28 22:49:07 Unraid kernel: BTRFS info (device md7p1): resizing devid 1
Aug 28 22:49:07 Unraid emhttpd: mounting /mnt/disk8
Aug 28 22:49:07 Unraid emhttpd: shcmd (131): mkdir -p /mnt/disk8
Aug 28 22:49:08 Unraid emhttpd: shcmd (132): mount -t btrfs -o noatime,space_cache=v2 /dev/md8p1 /mnt/disk8
Aug 28 22:49:08 Unraid kernel: BTRFS info (device md8p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:49:08 Unraid kernel: BTRFS info (device md8p1): using free space tree
Aug 28 22:51:02 Unraid emhttpd: shcmd (133): btrfs filesystem resize 1:max /mnt/disk8
Aug 28 22:51:02 Unraid root: Resize device id 1 (/dev/md8p1) from 18.19TiB to max
Aug 28 22:51:02 Unraid kernel: BTRFS info (device md8p1): resizing devid 1
Aug 28 22:51:02 Unraid emhttpd: mounting /mnt/disk9
Aug 28 22:51:02 Unraid emhttpd: shcmd (134): mkdir -p /mnt/disk9
Aug 28 22:51:03 Unraid emhttpd: shcmd (135): mount -t btrfs -o noatime,space_cache=v2 /dev/md9p1 /mnt/disk9
Aug 28 22:51:03 Unraid kernel: BTRFS info (device md9p1): using crc32c (crc32c-intel) checksum algorithm
Aug 28 22:51:03 Unraid kernel: BTRFS info (device md9p1): using free space tree
Aug 28 22:52:32 Unraid emhttpd: shcmd (136): btrfs filesystem resize 1:max /mnt/disk9
Aug 28 22:52:32 Unraid root: Resize device id 1 (/dev/md9p1) from 18.19TiB to max
Aug 28 22:52:32 Unraid kernel: BTRFS info (device md9p1): resizing devid 1
Aug 28 22:52:32 Unraid emhttpd: shcmd (137): /usr/sbin/zfs mount -a
Aug 28 22:52:32 Unraid emhttpd: shcmd (138): sync
Aug 28 22:52:52 Unraid emhttpd: shcmd (139): mkdir /mnt/user0
Aug 28 22:52:52 Unraid emhttpd: shcmd (140): /usr/local/bin/shfs /mnt/user0 -disks 1022 -o default_permissions,allow_other,noatime
Aug 28 22:52:52 Unraid shfs: FUSE library version 3.12.0
Aug 28 22:52:52 Unraid emhttpd: shcmd (141): mkdir /mnt/user
Aug 28 22:52:52 Unraid emhttpd: shcmd (142): /usr/local/bin/shfs /mnt/user -disks 1023 -o default_permissions,allow_other,noatime -o remember=330
Aug 28 22:52:52 Unraid shfs: FUSE library version 3.12.0
Aug 28 22:52:52 Unraid emhttpd: shcmd (145): /usr/local/sbin/update_cron
Aug 28 22:52:53 Unraid root: Starting CA Auto Turbo Mode

 

Btrfs takes longer than xfs to mount, but 14 minutes seems excessive, it takes less than a minute for me with 20 disks, unless you have a lot of metadata, are these used with media type files or something different?

1 hour ago, JorgeB said:

Btrfs takes longer than xfs to mount, but 14 minutes seems excessive, it takes less than a minute for me with 20 disks, unless you have a lot of metadata, are these used with media type files or something different?

I do have a lot of data on the disks, they are almost full over 95% each. In regards to the metadata this is the stats below for my disk7:

Data, single: total=16.06TiB, used=16.00TiB System, DUP: total=40.00MiB, used=1.88MiB Metadata, DUP: total=40.00GiB, used=38.01GiB GlobalReserve, single: total=512.00MiB, used=16.00KiB

 

Does this look normal compared to yours? My metadata is 38GB/40GB from what I see. If you are able to share one of your stats so I can maybe compare the differences

Edited August 29, 20241 yr by Hellomynameisleo

Seems a lot, but most of my disks are smaller, will need to check later to confirm, I'll get back to you.

This is for an almost full 8TB drive, and it takes about 8 secs to mount:

Data, single: total=7.22TiB, used=7.06TiB
Metadata, DUP: total=8.50GiB, used=7.51GiB

 

You have more metadata comparatively, still seems like a long time to mount the disks, I remember reading that btrfs was going to optimize that for an upcoming kernel, maybe 6.10

7 hours ago, JorgeB said:

This is for an almost full 8TB drive, and it takes about 8 secs to mount:

Data, single: total=7.22TiB, used=7.06TiB
Metadata, DUP: total=8.50GiB, used=7.51GiB

 

You have more metadata comparatively, still seems like a long time to mount the disks, I remember reading that btrfs was going to optimize that for an upcoming kernel, maybe 6.10

Do you have compression set on your one? I have mine set at zstd:3 for all disks and most of my disks are around 40GB for the metadata. I counted the metadata total of all the disks and it sits at 305GB. So I am guessing its 305GB of metadata having to be read first in total to start up the array, and at the speed of 280MB/s it lines up with the time it takes . But I'm not sure if compression is making it longer.

For the content on the disks I have millions of files mixed data types all over so I think that contributes to the metadata size

I don't use compression since they are mostly media files, but btrfs doesn't need to read all the metadata during mount, but the more files/metadata the longer it will take to mount, zfs is also similar, xfs is always much faster than both.

2 hours ago, JorgeB said:

I don't use compression since they are mostly media files, but btrfs doesn't need to read all the metadata during mount, but the more files/metadata the longer it will take to mount, zfs is also similar, xfs is always much faster than both.

I'll deal with the longer array startup time then not someimt I do often and its saving me 10TB (126/136TB) of space so far with its compression which is a decent amount.