[6.11.5] Cannot write to deep directories with btrfs compression when supported by cache - only 1 directory made each write request

Works for me.

 

I created a share called 'test' with all defaults except set Use cache to Yes.  Creating the share resulted in creating the 'test' directory on /mnt/disk1

 

root@Tower:~# cd /mnt/disk1
root@Tower:/mnt/disk1# v
total 0
drwxrwxrwx 2 nobody users 6 Jan 27 11:50 test/
root@Tower:/mnt/disk1# cd test/
root@Tower:/mnt/disk1/test# mkdir 1
root@Tower:/mnt/disk1/test# cd 1/
root@Tower:/mnt/disk1/test/1# mkdir 2
root@Tower:/mnt/disk1/test/1# cd 2/
root@Tower:/mnt/disk1/test/1/2# mkdir 3
root@Tower:/mnt/disk1/test/1/2# cd /mnt/user/test/1/2/3/
root@Tower:/mnt/user/test/1/2/3# touch a
root@Tower:/mnt/user/test/1/2/3# v
total 0
-rw-rw-rw- 1 root root 0 Jan 27 11:51 a
root@Tower:/mnt/user/test/1/2/3# v /mnt/cache/test/1/2/3/a 
-rw-rw-rw- 1 root root 0 Jan 27 11:51 /mnt/cache/test/1/2/3/a
root@Tower:/mnt/user/test/1/2/3# v /mnt/disk1/test/1/2/3/
total 0
root@Tower:/mnt/user/test/1/2/3# 

 

With Btrfs-encrypted array disks and xfs-encrypted cache disk?

 

Any diagnostic recommendations on my end? 

1 hour ago, robobub said:

With Btrfs-encrypted array disks and xfs-encrypted cache disk?

 

Any diagnostic recommendations on my end? 

Encryption doesn't matter.  You can try to reproduce my test on your server.

Ah, with a new share the issue doesn't exist.  The issue is caused by using any form of btrfs compression on the disk.  Please see if you can reproduce:

 

## no compression, no issue
root@Tower:/mnt/user/test# rm -rf /mnt/user/test/1; 
root@Tower:/mnt/user/test# rm -rf /mnt/cache/test/1 # should be no-op, but just in case
root@Tower:/mnt/user/test# cd /mnt/disk1/test
root@Tower:/mnt/disk1/test# v
total 0
root@Tower:/mnt/disk1/test# btrfs property get .
root@Tower:/mnt/disk1/test# ## no compression, no issue
root@Tower:/mnt/disk1/test# mkdir -p 1/2/3
root@Tower:/mnt/disk1/test# cd /mnt/user/test/1/2/3/
root@Tower:/mnt/user/test/1/2/3# touch a
root@Tower:/mnt/user/test/1/2/3# v /mnt/cache/test/1/2/3/a
-rw-rw-rw- 1 root root 0 Jan 31 18:13 /mnt/cache/test/1/2/3/a
root@Tower:/mnt/user/test/1/2/3# cd /mnt/user/test


## re-do test with compression
root@Tower:/mnt/user/test# rm -rf /mnt/user/test/1; 
root@Tower:/mnt/user/test# rm -rf /mnt/cache/test/1 # should be no-op, but just in case
root@Tower:/mnt/user/test# cd /mnt/disk1/test
root@Tower:/mnt/disk1/test# btrfs property get .
root@Tower:/mnt/disk1/test# btrfs property set . compression zlib
root@t1000:/mnt/disk1/test# btrfs property get .
compression=zlib
root@Tower:/mnt/disk1/test# mkdir -p 1/2/3
root@Tower:/mnt/disk1/test# cd /mnt/user/test/1/2/3/
root@Tower:/mnt/user/test/1/2/3# touch a
touch: cannot touch 'a': Operation not supported
root@Tower:/mnt/user/test/1/2/3# tree /mnt/cache/test/
/mnt/cache/test/
└── 1

1 directory, 0 files
root@Tower:/mnt/user/test/1/2/3# touch a
touch: cannot touch 'a': Operation not supported
root@Tower:/mnt/user/test/1/2/3# tree /mnt/cache/test/
/mnt/cache/test/
└── 1
    └── 2

2 directories, 0 files

Occurs with zlib, std, lzo, or chattr +C

Edited January 31, 2020 by robobub

We've never enabled btrfs compression and there's no standard way to do so, so we've never tested this - apparently there's an issue 🤨  doesn't make sense tho

Has this issue been reproduced and triaged?

Your exact sequence seems to work, using 6.9.0-beta

On 7/1/2020 at 8:23 PM, limetech said:

Your exact sequence seems to work, using 6.9.0-beta

 

I just tested this on 6.10.2 and the problem exists. I do feel at some point around 6.9 it was working though.

EDIT: I see 6.10.3 is out, will test on that.

EDIT2: Problem replicated on 6.10.3

Edited January 30, 2023 by robobub

On 7/1/2020 at 8:23 PM, limetech said:

Your exact sequence seems to work, using 6.9.0-beta

I never ended up testing 6.9.0-beta but did replicate this on 6.11.5. Can you or anyone else try this sequence again?

 

Side note: When I first upgraded to 6.11.5 it initially appeared to work, but I may not have had the conditions setup correctly and got distracted by other things. So there certainly could be some other configuration I forgot to add.

 

Motivation: After reading more about zstd's impressive capabilities and thus have every drive set to compress-force=zstd:5 and am now running into this all the time

Edited January 30, 2023 by robobub

I cannot reproduce this:

 

root@Tower15:/mnt/user/test/1/2/3# rm -rf /mnt/user/test/1;
root@Tower15:/mnt/user/test/1/2/3# rm -rf /mnt/user/test/1
root@Tower15:/mnt/user/test/1/2/3# cd /mnt/disk1/test
root@Tower15:/mnt/disk1/test# btrfs property set . compression zstd
root@Tower15:/mnt/disk1/test# btrfs property get .
compression=zstd
root@Tower15:/mnt/disk1/test# mkdir -p 1/2/3
root@Tower15:/mnt/disk1/test# cd /mnt/user/test/1/2/3/
root@Tower15:/mnt/user/test/1/2/3# touch a
root@Tower15:/mnt/user/test/1/2/3# tree /mnt/cache/test/
/mnt/cache/test/
└── 1
    └── 2
        └── 3
            └── a

3 directories, 1 file
root@Tower15:/mnt/user/test/1/2/3#

 

On 2/2/2023 at 7:17 AM, JorgeB said:

I cannot reproduce this:

 

root@Tower15:/mnt/user/test/1/2/3# rm -rf /mnt/user/test/1;
root@Tower15:/mnt/user/test/1/2/3# rm -rf /mnt/user/test/1
root@Tower15:/mnt/user/test/1/2/3# cd /mnt/disk1/test
root@Tower15:/mnt/disk1/test# btrfs property set . compression zstd
root@Tower15:/mnt/disk1/test# btrfs property get .
compression=zstd
root@Tower15:/mnt/disk1/test# mkdir -p 1/2/3
root@Tower15:/mnt/disk1/test# cd /mnt/user/test/1/2/3/
root@Tower15:/mnt/user/test/1/2/3# touch a
root@Tower15:/mnt/user/test/1/2/3# tree /mnt/cache/test/
/mnt/cache/test/
└── 1
    └── 2
        └── 3
            └── a

3 directories, 1 file
root@Tower15:/mnt/user/test/1/2/3#

 

 

 

Interesting, thanks for trying it out. Just now I reproduced the operation not supported again on my system after recreating my array (though not reformatting all of my disks). Any thoughts as to what it could?

 

You are on 6.11.5 as well? What filesystem/encryption type do you have on cache and disk1 (I am using xfs-encrypted and btrfs-encrypted for reference)? Is disk1 the next drive to be written to based on your allocation method? Do you have space_cache=v2 on your btrfs mount options `mount | grep btrfs`?

 

EDIT: Or a diagnostics would be sufficient instead of these individual questions . My diagnostics for the original 6.8.2 and with 6.11.5 are in the OP

 

If you are up for trying more commands, can you do a `ls -l /mnt/cache/test; ls -l /mnt/disk*/test` ? 

 

Also perhaps a `tree /mnt/cache/test` and `tree /mnt/disk*/test` before and after the mkdir and the touch steps? And maybe with `mount -o remount,compress-force=zstd /mnt/disk1`? You can undo it with `mount -o remount,compress-force=none /mnt/disk1`

 

EDIT: Some further testing:

I tried making /mnt/disk1/test/1/2/3 with compress=no then changing compress=zstd on just /mnt/disk1/test and it succeeds. Then I did the inverse, and (folders made with compress=zstd, then setting the root to compress=no) and it fails.

 

I then made folder test/1/2 both with with compress=no, then went into that and made 3/4/5 after setting test/1/2 to compress=zstd, and then verified that when touching a file and having operation not supported, it made /mnt/cache/1/2/ (e.g. the number of folders deep with compress=no) then stopped

 

Any thoughts as to what it could be are welcome.

Edited February 13, 2023 by robobub

So it actually does not occur with mount -o remount,compress-force=zstd by itself. My disks still had btrfs set compress on specific folders, which at least for my setup, causes this issue. When removing per-folder settings, I do not get this error.

 

It still would be nice to set higher compression levels and more expensive algorithms on certain folders (at least for whatever is different in my setup than JorgeB's), but this is good enough for now, so I'll downgrade this to "Annoyance"