XFS Extended Defragmentation

[mgutt]

55 minutes ago, shorshi said:

can i somehow see a progress

No

 

55 minutes ago, shorshi said:

default 2hrs seems very low?

It's meant to be executed every night by cron or similar. Of course you can change the timeout to whatever you prefer 

 

[domidomi]

So I'm using this script for the first time, and I don't know much about how xfs_fsr is supposed to work, but it seems to be really slow? It could be that I'm running it with parity enabled, but the write speed on the target HDD is around 13 MB/s while CPU is not even reaching 50%. Nothing else is accessing the shares, no Docker containers are running, no VMs installed at all.

 

I guess I just want to know: is this normal?

 

Edit: Could it be that I'm using this on encrypted XFS? Obviously I have modified the script to use /dev/mapper.

Edited July 14, 2022 by domidomi

[mgutt]

1 hour ago, domidomi said:

write speed on the target HDD is around 13 MB/s

Which model and how much space is free?

[domidomi]

16 minutes ago, mgutt said:

Which model and how much space is free?

They're five drives of the same model, Seagate IronWolf 8TB. This particular drive that is currently being defragged has 300+ GB free space.

[mgutt]

How fast is copying a huge file on the same disk?

[domidomi]

8 minutes ago, mgutt said:

How fast is copying a huge file on the same disk?

If you mean copying it from disk 1 to disk 1, then it's literally instant. Tried it with a 4.1 GB file.

 

I have noticed that xfs_fsr is running faster on disk 5, which has about 3.5 TB free space. There I get around 40 MB/s write speed.

 

Edit: A few minutes of observation shows that occasionally, on disk 5, it reaches 90 MB/s, then it'll jump down to 35-45 MB/s. It seems like it might depend on the which file it's currently working with?

Edited July 14, 2022 by domidomi

[mgutt]

1 hour ago, domidomi said:

If you mean copying it from disk 1 to disk 1, then it's literally instant. Tried it with a 4.1 GB file.

This is because you are copying the file into your RAM. Linux automatically uses free RAM as cache.

 

1 hour ago, domidomi said:

it reaches 90 MB/s, then it'll jump down to 35-45 MB/s. It seems like it might depend on the which file it's currently working with?

Don't forget: The file is fragmented. This means it's data is located on very different positions on the disk, needs to be collected and written to a new position. Then it's important to know, that an HDD is much slower at the last sectors compared to the first sectors:

 

Thats why I asked for the free space. More free space means more faster sectors.

 

I think this in combination with the parity impact and the encryption, is the reason, why it is so slow.

[domidomi]

@mgutt Thanks for the help, much appreciated!

 

If someone else reads this post in the future, I'll add that I balanced the disks so that they all have roughly 20% free space (1.5 TB free of 8 TB total), and it hasn't significantly changed the speed of the defragmentation in general. I will also add that I've tested the speed of all the disks and they all start at 250+ MB/s and end at around 130 MB/s, so that's not the issue either.

I'll just be writing this up as caused by the parity drive and disk encryption. I still don't quite understand how the slowest of the disk extents can reach only around 13-15 MB/s, that doesn't really make much sense to me personally. Some extents do perform better, around 70-90 MB/s.

Edited July 15, 2022 by domidomi

[domidomi]

I did some of investigation into xfs_fsr in order to determine if defragging XFS filesystems actually makes any significant difference. Below are my findings. Note that I am no expert on any of this, so take all of this with a grain of salt.

 

My idea was to take the most fragmented file that I could find on a drive and see how long it would take to read the entire file. Then I was going to defragment the file and perform the same read test again and compare the difference in speed. I chose disk1 for this, but I guess it shouldn't matter much.

 

Turns out that, by far, the most fragmented file is the Docker image file with a whopping 15,571 extents. My Docker image file is 20 GB in size. xfs_db gave me the inode number and then I just used "find" to figure out which file it corresponded to.

 

Before defragmenting the file, reading the entire file took 98.6 seconds. After having defragmented the file, it took 82.96 seconds, which is an improvement by about 15%. I performed the experiment on the second-to-most fragmented file, on a different drive, which spanned 97 extents. The speed improvement there was around 10%. I don't know enough about HDD's to know if this could be a result of the drive head having been moved to a better position after the defragmentation or not. It doesn't seem unlikely to me.

 

Some people in this forum have reported that defragmenting their drives improved performance tremendously, but based on what I've found here, I wouldn't put too much trust in those reports. Placebo is a hell of a drug!

 

root@tower:~# xfs_db -r /dev/mapper/md1 -c "frag -v" | sort -k4n | tail -n 2 | head -n 1
inode 2147483777 actual 15571 ideal 1

root@tower:~# find /mnt/disk1/ -inum 2147483777 -printf "%p (%s bytes)\n"
/mnt/disk1/system/docker/docker.img (21474836480 bytes)

root@tower:~# dd if=/mnt/disk1/system/docker/docker.img of=/dev/null status=progress
21196114432 bytes (21 GB, 20 GiB) copied, 98 s, 216 MB/s
41943040+0 records in
41943040+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 98.6201 s, 218 MB/s

root@tower:~# xfs_fsr -d -v /mnt/disk1/system/docker/docker.img
/mnt/disk1/system/docker/docker.img
/mnt/disk1/system/docker/docker.img extents=15571 can_save=15570 tmp=/mnt/disk1/system/docker/.fsr1141
DEBUG: fsize=21474836480 blsz_dio=16773120 d_min=512 d_max=2147483136 pgsz=4096
Temporary file has 3 extents (15571 in original)
extents before:15571 after:3      /mnt/disk1/system/docker/docker.img

root@tower:~# dd if=/mnt/disk1/system/docker/docker.img of=/dev/null status=progress
21222499840 bytes (21 GB, 20 GiB) copied, 82 s, 259 MB/s
41943040+0 records in
41943040+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 82.9575 s, 259 MB/s

 

[JonathanM]

37 minutes ago, domidomi said:

Some people in this forum have reported that defragmenting their drives improved performance tremendously, but based on what I've found here, I wouldn't put too much trust in those reports. Placebo is a hell of a drug!

 

Expect write performance to increase almost 4x read because the parity updates the same sectors, so updating a highly fragmented file could possibly see better improvements.

 

A well optimized Unraid system is mostly WORM on the parity array, so not much improvement to be had by defragging there. All random I/O should be happening on cache pools.

[shorshi]

is running this script for 2hrs every night the optimal way to go about it? will it reduce HDD life span wit that much disc activity?

[shorshi]

**push**

[ChatNoir]

On 9/12/2022 at 7:17 AM, shorshi said:

will it reduce HDD life span wit that much disc activity?

If there are defragmentation to be done, whether you are doing it 2h a day or for longer, it will probably produce the same wear on the drive in the end.

[Gorosch]

Is it possible, to abort an running defrag? If i kill the xfs_fsr process, another starts. The "abort Script" button seems not to work.

[KyleS]

Doesn't work at all with encryption (can't find the disks). - just needs a minor change to your "regex".

 

Do you have a layout defragger as well?

Edited November 1, 2022 by KyleS
fixed it.

[mgutt]

On 10/19/2022 at 11:08 AM, Gorosch said:

Is it possible, to abort an running defrag? If i kill the xfs_fsr process, another starts. The "abort Script" button seems not to work.

This is something related to the user scripts plugin which is a really old bug:

https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/?do=findComment&comment=881200

 

1 hour ago, KyleS said:

Doesn't work at all with encryption (can't find the disks).

Please post results of these commands:

 

lsblk

 

df -h

[KyleS]

23 hours ago, mgutt said:

This is something related to the user scripts plugin which is a really old bug:

https://forums.unraid.net/topic/48286-plugin-ca-user-scripts/?do=findComment&comment=881200

 

Please post results of these commands:

 

lsblk

 

df -h

/dev/mapper/md*

[Hexenhammer]

Op, thanks for your work and the script.

I have 2 questions.

1. what does defrag_only_sleep=1 mean?

2. When you say it will just defrag 2 hours, does it mean 2 hours each drive or 2 hours all of them in paralleled?

 

And how i can defrag in parallel like in windows? whats the command that defrays all the HDDs at once

 

Thanks

[mgutt]

2 hours ago, Hexenhammer said:

what does defrag_only_sleep=1 mean?

Defrags only if parity is not spinning (to avoid defrag while for example parity check is running).

 

2 hours ago, Hexenhammer said:

does it mean 2 hours each drive or 2 hours all of them

Single drive. Writing to all drives at the same time would be extremely slow because of the parity.

 

 

[Hexenhammer]

19 hours ago, mgutt said:

Defrags only if parity is not spinning (to avoid defrag while for example parity check is running).

 

Single drive. Writing to all drives at the same time would be extremely slow because of the parity.

 

 

 

Thanks, i don't have parity right now, is there a way [command line?] to defrag all in parallel?
Before i enable Parity i want all the drives defraged

Thanks

 

EDIT, OP i found a xfs defrag guide and it has a recommended settings, any opinions?

 

Quote

We can further help the XFS file system in reducing the level of fragmentation by mounting it with ‘allocsize’ specified, for example see the below configuration line from /etc/fstab.

When mounted this has speculatively preallocated us 512mb of contiguous sequential space, for most users the defaults that XFS provides will generally be fine.

/dev/xvda1 / xfs defaults,allocsize=512m 0 0

https://www.rootusers.com/how-to-defragment-an-xfs-file-system/

 

EDIT 2: script doesn't work on systems without Parity drives.[I have not set any parity HDDs, just set 2 empty slots for future use]

 

This is what i get:

/tmp/user.scripts/tmpScripts/Defrag v0.2/script: line 26: mdcmd: command not found
Parity has device id
Defragmentation has been skipped because of active parity disk
 

Edited December 28, 2022 by Hexenhammer

[DuzAwe]

This appears to have stopped working with Version: 6.12.0-rc1.

 

[Computron]

On 3/18/2023 at 9:10 AM, DuzAwe said:

This appears to have stopped working with Version: 6.12.0-rc1.

 

 

As of Unraid 6.12 the drive path is of the form mdXpY. I think for most cases Y=1 (i.e mdXp1) but in case it's not I've added a new regex match.

 

The modified HDD finding code looks like this:

 

# parse /etc/mtab and check rotational status
declare -a mtab
while IFS= read -r -d '' line; do
    disk_id_regex="^/dev/md([0-9]+)p([0-9]+)"
    if [[ "$line" =~ $disk_id_regex ]]; then
        disk_id=${BASH_REMATCH[1]}
        part_id=${BASH_REMATCH[2]}
        rotational=$(cat /sys/block/md${disk_id}p${part_id}/queue/rotational)
        if [[ "$rotational" == "1" ]]; then
            mtab+=("$line")
            echo "Found HDD with id md${disk_id}p${part_id} (added)"
            continue
        fi
        echo "Found SSD with id md${disk_id}p${part_id} (skipped)"
    fi
done < <(cat /etc/mtab | grep -E '^/dev/md' | tr '\n' '\0')

 

[Rolucious]

Is it possible to place the entire script that you're running?
Ive adjusted the old with the new code, but when I run it, I only see that its trying to identify the HDD's. 
I have my docker turned off and parity spun down, but I don't know how it 'should' look like.
I see not read/writes on the drives either.

[DoINeedATag]

Here is an updated version for Unraid Version 6.12.6. I used and modified both mgutt and Computron's code to get it working. Your mileage may very on other versions.

 

I called it version v0.3 just to keep things straight if others find this code and get it confused with the one on the main page.

 

#!/bin/bash
# #####################################
# Script:      XFS Extended Defragmentation v0.3
# Description: Defragmentate only HDDs (SSDs will be ignored) if requirements met for a selectable running time.
# Author:      Marc Gutt, Computron, DoINeedATag
# 
# Changelog:
# 0.3
# - 6.12.6+ fix for regex finding drive paths (Updated by Computron & DoINeedATag)
# 0.2
# - SSD recognition added
# 0.1
# - first release
# 
# ######### Settings ##################
# defrag_seconds=7200 : Defrag only for a specific time in seconds (default is 7200 seconds = 2 hours)
# defrag_only_sleep=1 : Defrag only when parity is not active (set to 0 to disable)
# notification=1 : Notify yourself (set to 0 to disable)

defrag_seconds=7200
defrag_only_sleep=1
notification=1
# #####################################
# 
# ######### Script ####################
# make script race condition safe
if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT;

# defrag only if parity is not spinning
if [[ $defrag_only_sleep == "1" ]]; then
    # obtain parity disk's device id
    parity_id=$(mdcmd status | grep rdevName.0 | cut -d = -f 2)
    echo "Parity has device id $parity_id"
    # we defrag only if parity is not spinning
    parity_state=$(smartctl -n standby "/dev/$parity_id")
    if [[ $parity_state != *"STANDBY"* ]]; then
        echo "Defragmentation has been skipped because of active parity disk"
        exit
    fi
fi

echo "Search for HDDs..."

# parse /etc/mtab and check rotational status
declare -a mtab
while IFS= read -r -d '' line; do
    disk_id_regex="^/dev/mapper/md([0-9]+)p([0-9]+)"
    if [[ "$line" =~ $disk_id_regex ]]; then
        disk_id=${BASH_REMATCH[1]}
        part_id=${BASH_REMATCH[2]}
        rotational=$(cat /sys/block/md${disk_id}p${part_id}/queue/rotational)
        if [[ "$rotational" == "1" ]]; then
            mtab+=("$line")
            echo "Found HDD with id md${disk_id}p${part_id} (added)"
            continue
        fi
        echo "Found SSD with id md${disk_id}p${part_id} (skipped)"
    fi
done < <(cat /etc/mtab | grep -E '^/dev/mapper/md' | tr '\n' '\0')

if [ ${#mtab[@]} -eq 0 ]; then
    /usr/local/emhttp/webGui/scripts/notify -i alert -s "XFS Defragmentation failed!" -d "No HDD found!"
    exit
fi

printf "%s\n" "${mtab[@]}" > /tmp/.xfs_mtab
echo "Content of /tmp/.xfs_mtab:"
cat /tmp/.xfs_mtab

# defrag
xfs_fsr -v -m /tmp/.xfs_mtab -t "$defrag_seconds" -f /tmp/.xfs_fsrlast

 

 

Edited March 28 by DoINeedATag
Edit cleaned up and clarified settings.

[anthony0030]

I tried running the script in the background and it looks like nothing happens

 

 

Script Starting Mar 28, 2024  19:09.13

Full logs for this script are available at /tmp/user.scripts/tmpScripts/XFS Extended Defragmentation v0.3/log.txt

Parity has device id sdf
Search for HDDs...
Script Finished Mar 28, 2024  19:09.13

I tried also disabeling defrag_only_sleep

I am running unraid 6.12.8

Edited March 28 by anthony0030