My cache is an M.2 disk, working fine for a long time.
It is ZFS formatted (which I slowly but surely regret).

I don't think it is a hardware issue, and UNRAID doesn't report one.

 

I am doing some other (huge) file transfers between other disks in the array (not cache). Think of some UNBALANCE work, but manually using DoubleCommander.

I did some moves out of cache yesterday (to within the array), because I didn't want to wait for mover and it was some tiny files.

 

I also tried to delete some empty folders from cache (remaining from moving the contents). System (tried both DoubleCommander and UNRAID file manager) didn't seem to delete them (!), but without an error!

Today, the folders where missing though. It is like ZFS had that in some queue and finally did it.

 

Now I am trying to copy a couple of tiny files to my UNRAID from my PC. Files that belong to a share that is configured cache->disk whatever.

It fails!

 

In UNRAID log it writes:

Oct  9 23:56:48 quasar-ultima shfs: /usr/sbin/zfs create 'cache/emulation' 2>&1

(a few times)

And it did NOT create that folder!

 

I tried moving a folder (with a single file) from array to cache from DoubleCommander. It worked fine. Then I deleted the file from cache, again just fine.

But I stopped being able to write to array and looks like the intermediate step of going first to cache is the issue.

(but I cannot be sure as I don't have a share that doesn't use the cache OR direct access to array disks from my PC)

 

ZFS bar in UNRAID dashboard, reports "82% 4.80GB".

I've seen that bar full in the past, yet no issue to write on my cache.

 

Cache also hosts a working VM and many containers. Everything works.

 

Any ideas?

 

EDIT: I did find a share that is not using the cache at all and indeed I could write there. So it is indeed some fresh issue possibly with ZFS or cache access over LAN? (I didn't make any changes)

 

EDIT #2: OK a breakthrough!

I created the share folder manually in cache (the folder that also exists in array and has TB of content).

After I did that, I COULD write in the share (and I mean 2-3 folders deep in the share)!

In other words it COULD create subfolders within the cache share folder to put the file in the proper path.

 

I never had this issue before in the years I use UNRAID (including with ZFS cache, maybe more than a year). If a share folder was not also present in cache, UNRAID just created it. How did this stop?

 

Also (it can be related), the issue of moving folders OUT of cache. When I moved a complete share folder from cache to array, the folder (as expected) was in the end removed from cache. Now it just stays there (but empty). I ignored this as a trivial issue. But now this is weird. Now I need to make ALL my shares also in my cache disk, to be able to write to UNRAID from LAN.

 

Any ideas???

 

Edited October 9, 20241 yr by NLS

Check the floor setting for the share you cannot write to.

1 hour ago, JorgeB said:

Check the floor setting for the share you cannot write to.

 

Thanks. I did. The files were mere kilobytes after all.
It wasn't that.

 

Today something else weird happened, which seems to be related.

I just wrote a file to another share (as small mp3) from Windows. It didn't error out.

I see the file in windows explorer in the share and I can even play it.

This share is configured to be primary array, secondary cache, move cache to array.

Well... I CANNOT PHYSICALLY FIND IT!

I see it in windows explorer in the proper path (Q:\music\_CLEAN_UP).

In Double Commander in cache the copy (from Windows) DID create a "music" folder, but it is empty (!?). I refreshed many times.

In Double Commander in ANY of the disks it could possibly move the file, it is not there! (folder is not empty though, it has an older file, which resides in array)
In UNRAID file manager, I checked both "SHARES" (folder music\_CLEANUP) and "MAIN" tab. The file IS physically in cache (in music\_CLEAN_UP) as expected!

I can see that the folder (and subfolder) belong to "nobody", which I believe is expected behavior and full rw permissions.

Why double commander doesn't show the file or the subfolder that is in it? It uses normal Linux calls to retrieve the directory (being a container within UNRAID).

Very weird behavior and I cannot reboot the server for at least 2-3 days more, in case this corrects itself.

 

EDIT: After I successfully edited the file in music, I tried to move it to other share (also cache to array type). It didn't allow me, with the same error as in OP. It only worked after I manually created the other share's folder in cache. (then it had no problem creating the subfolder structure in it)

Edited October 10, 20241 yr by NLS

Post the diagnostics and the share name you are trying to write to.

2 hours ago, NLS said:

This share is configured to be primary array, secondary cache, move cache to array.

I did not think this was a possible configuration - are you sure it is not primary cache and secondary array?

 

2 hours ago, NLS said:

The file IS physically in cache (in music\_CLEAN_UP) as expected!

This is what I would expect.

 

2 hours ago, NLS said:

Why double commander doesn't show the file or the subfolder that is in it?

What access mode did you set in the volume mappings?

2 hours ago, JorgeB said:

Post the diagnostics and the share name you are trying to write to.

WIll do and revert later.

 

8 minutes ago, itimpi said:

I did not think this was a possible configuration - are you sure it is not primary cache and secondary array?

 

This is what I would expect.

 

What access mode did you set in the volume mappings?

Erm you are right, I meant primary cache, secondary array.

...Double Commander was always like that (for years) and I never had an issue.

For the diagnostics, I tried to copy a tiny text file from my desktop to a share.

chuck-to-check.txt to Q:\photos

IT WORKED! I know for a fact that in cache, there was no folder "photos" before and now there is. So this worked as expected.

I opened the file fine and then deleted it.

Then I tried to do the same to share emulation. This folder also doesn't exist in cache (I deleted it before).
chuck-to-check.txt to Q:\emulation failed.

Of course I am still doing huge moves in DoubleCommander  (TBytes, that take several hours, between disks for folders and files that belong to share "emulation" - but not in the folders that are actually in the queue to move). Again: I tried to write to "emulation" but NOT in the folders that are moving.

Also before, another folder ("generic") that was not part of the huge move, also I failed to write to it. Until I manually created the folder in cache.

 

Note that all shares (I will recheck) have between 1GB and 30GB "floor" and all involved disks have more than enough (and remember cache itself has 700GB free).

I somehow feel that after all the moves I need to do finish and I do an extra reboot on the server, things will be fixed - still I think we need to know what is the issue right now.

 

@JorgeB I attach diags.

quasar-ultima-diagnostics-20241010-1520.zip

56 minutes ago, NLS said:

Double Commander was always like that (for years) and I never had an issue.

I think you need Read/Write (slave) for it to see the changes.

39 minutes ago, itimpi said:

I think you need Read/Write (slave) for it to see the changes.

I will change it if after reboot I still doesn't work as expected, but as I said, it was like that for years and didn't notice an issue before.

 

According to this:

https://docs.docker.com/engine/storage/bind-mounts/#configure-bind-propagation

I am correct now.

Edited October 10, 20241 yr by NLS

Post the output from

zfs list -t all

 

1 hour ago, JorgeB said:

Post the output from

zfs list -t all

 

 

 

output.txt

Emulation dataset exists, but this is strange:

 

Oct 10 15:10:44 quasar-ultima shfs: /usr/sbin/zfs create 'cache/emulation' 2>&1
### [PREVIOUS LINE REPEATED 2 TIMES] ###

 

It tried to create it multiple times, or were you manually destroying the dataset in between?

 

Also make sure there's not a regular folder below the dataset, i.e., destroy the dataset, and see if you can still browse /mnt/cache/emulation locally.

29 minutes ago, JorgeB said:

Emulation dataset exists, but this is strange:

 

Oct 10 15:10:44 quasar-ultima shfs: /usr/sbin/zfs create 'cache/emulation' 2>&1
### [PREVIOUS LINE REPEATED 2 TIMES] ###

 

 

It tried to create it multiple times, or were you manually destroying the dataset in between?

 

Also make sure there's not a regular folder below the dataset, i.e., destroy the dataset, and see if you can still browse /mnt/cache/emulation locally.

I have no knowledge of zfs (I need to do some reading some time), so I don't even know what you mean.

No I didn't do anything in between. Probably just the retries of the OS to write to share.

Meanwhile:

...this is how my cache disk looks right now.
"generic" I created manually because else I couldn't write to the share

"music" got created automatically when I wrote to the share (but showed empty in Double Commander, at least it showed ok in UNRAID file manager - could be a different issue)

"photos" got created automatically and worked ok

"emulation" as you can see is not there - to be honest I don't remember if I deleted it with all the experiments, but I did create it manually to write to it

Let me point out once again, that my cache is zfs for a few months already and UNRAID is many years that I have it, yet this happened now. I didn't have the issue before. Folders where created automatically on cache when needed and mover used to remove them after they were empty (although I think mover MIGHT not delete them even if empty for quite some time - not sure if there is a change in mover or an issue, but wasn't a real issue anyway).

If we can find the issue before I reboot the server in 2-3 days (after my manual "intra-array" shifts are done) it would be great as it may help identify a real problem in UNRAID.
This definitely doesn't look normal.

Now that UNRAID moves to full zfs support with 7.0, I might just revert my cache to another filesystem and not follow the trend. Unnecessary complexity.
(and I know how strong many people seem to love zfs - but I guess, to each their own)

Edited October 10, 20241 yr by NLS

16 minutes ago, NLS said:

...this is how my cache disk looks right now.

The zfs output show that the emulation dataset exists:

 

cache/emulation                                                                          96K   673G       96K  /mnt/cache/emulation

 

So it should show up on the cache folder list, type:

 

zfs destroy cache/emulation

 

then 

 

cd /mnt/cache/emulation

 

Does it still exist?

3 hours ago, JorgeB said:

The zfs output show that the emulation dataset exists:

 

cache/emulation                                                                          96K   673G       96K  /mnt/cache/emulation

 

 

So it should show up on the cache folder list, type:

 

zfs destroy cache/emulation

 

 

then 

 

cd /mnt/cache/emulation

 

 

Does it still exist?

 

Will this affect my moves happening between other disks in the same share?
(not cache)

If not, I will try this tomorrow, thanks.

 

10 hours ago, NLS said:

Will this affect my moves happening between other disks in the same share?

Nope, and it should automatically be created again if you try to write something to that share.

41 minutes ago, JorgeB said:

Nope, and it should automatically be created again if you try to write something to that share.

 

OK I did this then.

I tried to cd to /mnt/cache/emulation even BEFORE doing the "destroy" - it said no such file or directory.

I then did the destroy command, it didn't output any error (or anything) - took a few seconds.

Then I did again cd to /mnt/cache/emulation which again of course as expected, said no such file or directory.

 

After I did these, I tried to write to Q:\emulation (which should create the folder in cache)...

It wrote it!

 

I then did the same to another share that failed earlier today, "generic".
I tried to see the directory /cache/generic ...it wasn't there.

I did zfs destroy cache/generic.
I then tried to write to Q:\generic and it properly created the folder in cache and write the file!

 

So, the question is why those orphan zfs datasets?

I should do that for all my shares?

 

This is definitely weird and MAYBE there should be a check mechanism within UNRAID for that.

 

...and a more interesting experiment I did just now.

 

I wanted to write a small file to another share I haven't touched for a long time, named "programs".

I created a subfolder in it, using Windows Explorer. It created the subfolder fine.

Checking in UNRAID file explorer, I see a folder "programs" in cache disk and the subdirectory inside.

BUT WHEN I TRY to write a file in that folder, no go!!!

 

It created the folder, but couldn't write the file.

 

I then did the same experiment (without deleting the folders):

zfs destroy cache/programs

...this as expected destroyed also the folder I created inside, so I don't see it in Windows Explorer any more.

Now I create that folder again...

It creates it.

I try to write file in the folder...

NO GO AGAIN!

 

Damn that is weird.

 

1 hour ago, NLS said:

So, the question is why those orphan zfs datasets?

Not really sure what happened there, don't remember seeing this issue before.

 

1 hour ago, NLS said:

Damn that is weird.

It is, not sure what it could be, I assume no issues if the share is not set to use the zfs cache and goes directly to the array?

41 minutes ago, JorgeB said:

Not really sure what happened there, don't remember seeing this issue before.

 

It is, not sure what it could be, I assume no issues if the share is not set to use the zfs cache and goes directly to the array?

Correct. It was one of my first experiments if you see above.

 

Also I want to stress for the nth time, this is the first time it occurs to me also, after several months that I switched my cache to zfs (I did it very soon after UNRAID allowed for that - because I thought it would be cool to have at least one zfs back then).

What got me more scared was this latest incident where I could create the folder but I couldn't write to it even after I destroyed the dataset and redid it and it still failed. That's a first.
 

I somehow believe it will correct itself after reboot (which I won't be able to do for at least 30 more hours), but if it doesn't, I might need to somehow "cure" this zfs device or in the end, turn it back to another file-system (which would be painful).

 

Edited October 11, 20241 yr by NLS

Worth trying the reboot, if that doesn't help I would suggest recreating the pool, with zfs or a different fs.

4 minutes ago, JorgeB said:

Worth trying the reboot, if that doesn't help I would suggest recreating the pool, with zfs or a different fs.

...this doesn't say much about zfs and its supposed robustness and self healing abilities eh?

 

Don't think that's a zfs issue, something weird is going on there.

16 minutes ago, NLS said:

...this doesn't say much about zfs and its supposed robustness and self healing abilities eh?

 

ZFS can only self-heal if you have a redundant pool, not with a single drive...

5 hours ago, Kilrah said:

ZFS can only self-heal if you have a redundant pool, not with a single drive...

Yeah, I don't think there is a corruption issue, this is a rather fresh and quality M.2 that works fine otherwise. This is a matter of FS robustness.

 

It kind of scares me that we didn't find a solution, now that UNRAID embraces ZFS more and more.

 

This is the second time I have a zfs issue, last time was with a proxmox server.

I understand how "more modern" this fs is, but...