mover - create_parent: No space left on device

[martinjuhasz]

I'm running into an issue with mover that i cannot resolve.
It's refusing to move any files and errors with:

 

move: file: /mnt/disk2/martin/folder/to/img/IMG_0194_square_Fotor_edit.png
move: create_parent: /mnt/disk2/martin error: No space left on device

 

so everything is in its place, except for the share `martin`. I changed that from `cache: yes` to `cache: prefer` which should move it from my array to the cache. But every file fails. And it's not related to that share as it happened to other shares also and its not to a specific array disk , as both disk1 and two fail.

I still have plenty of space left on each drive and the `martin` folder only contains documents and is under 10gb, prob even under 3gb.

 

What i did before this happened is to move from a single cache drive with xfs to a cache pool with btrfs. for this i moved every share to the array with "cache: yes", verified everything is fine, rebuilt the array with a btrfs cache pool and here i am with this problem.

Thanks for any help with this, its hard without my dockers up

[Squid]

Can you post your diagnostics

[martinjuhasz]

sure thing winston-diagnostics-20220129-1645.zip

[Squid]

Unfortunately, that "create_parent" doesn't exist in the diagnostics.  Can you disable mover logging, reboot and then run mover again.  We won't see the errors, but we should be able to see what happens (appdata is being successfully moved to the cache drive in those diagnostics)

[martinjuhasz]

hm weird, the syslog in the diagnostics is not up to date?!

 

When i tail my syslog and run mover this will get written to log

 

Jan 29 17:37:23 Winston root: mover: started
Jan 29 17:37:23 Winston move: file: /mnt/disk1/martin/Rechnungen/Sonstiges/account_111198950_20110614.pdf
Jan 29 17:37:23 Winston move: create_parent: /mnt/disk1/martin error: No space left on device
Jan 29 17:37:23 Winston move: file: /mnt/disk1/martin/Rechnungen/Sonstiges/regfish_10835224.pdf
Jan 29 17:37:23 Winston move: create_parent: /mnt/disk1/martin error: No space left on device
Jan 29 17:37:23 Winston move: file: /mnt/disk1/martin/Rechnungen/Sonstiges/Druckbares Auftragsformular.pdf
Jan 29 17:37:23 Winston move: create_parent: /mnt/disk1/martin error: No space left on device
Jan 29 17:37:24 Winston move: file: /mnt/disk1/martin/Rechnungen/Sonstiges/regfish_10832244.pdf
Jan 29 17:37:24 Winston move: create_parent: /mnt/disk1/martin error: No space left on device
Jan 29 17:37:24 Winston move: create_parent: /mnt/disk1/martin error: No space left on device

 

Disabling the movers log does not work as this "create_parent" is created by the movers log. When disabling it nothing will show.
 

Edited January 29, 2022 by martinjuhasz

[martinjuhasz]

so the mover logs as shown above will not be displayed in the diagnostics syslog somehow. i tried also restarting, moving (i see it in tail -f /var/log/syslog), then dl diagnostics without it being reflected there

[Squid]

What I'm trying to catch is the I think there's an issue on the cache pool, but for some reason the logs were truncated...

[martinjuhasz]

yeah, since the cache pool is the main thing that changed, this might be true.

 

i mainly moved from xfs to btrfs because i wanted to have redundancy so that my files are save.

i think i had issues with btrfs when i set this thing up, which is why i chose xfs.

Any idea how i could verify its the cache pool?

[Squid]

Can you write a new file to a share that uses the cache drive and confirm it appears (Main, click on cache and then the folder icon way off to the right)

 

Alternatively install Fix Common Problems and hit rescan.  It'll do the same rough test as the above.

[martinjuhasz]

ok so with a new share the same happened. but somehow after i stopped the array and restarted it (without reboot) mover was working fine once. now after adding another array it failed again. stopped -> started, everything seems ok for the moment.

[martinjuhasz]

well, its weird. created another share where i can't get it working currently no matter if i start/stop the array.
the error message tho has changed a bit this time

 

Jan 29 19:50:56 Winston move: create_parent: /mnt/disk2 error: Software caused connection abort
Jan 29 19:50:56 Winston move: move_object: /mnt/disk2/test2: Transport endpoint is not connected
Jan 29 19:50:56 Winston root: mover: finished

 

if i cant get this to work i might have to switch back to xfs. that would be a bummer not to be able to have a cache drive thats guaranteed to be backed up.

Edited January 29, 2022 by martinjuhasz

[Squid]

16 minutes ago, martinjuhasz said:

Transport endpoint is not connected

That definitely implies a problem with the cache drive.

 

Do this

cp /var/log/syslog /boot

and post the syslog file now stored on the flash drive.  If it still doesn't show anything (for whatever reason), you're going to have to reboot and we'll go from there.

[martinjuhasz]

not sure if this syslog is any different?

syslog

[Squid]

Post the output of 

ls -ail /mnt/

But I don't think the current situation your in is recoverable until you reboot.

[martinjuhasz]

i don't mind rebooting. but i think this problem persists through reboots.

 

/bin/ls: cannot access '/mnt/user': Transport endpoint is not connected
total 16
                1930 drwxr-xr-x  9 root   root  180 Jan 29 19:52 ./
                   1 drwxr-xr-x 20 root   root  420 Jan 29 17:52 ../
                 256 drwxrwxrwx  1 nobody users  38 Jan 29 20:02 cache/
                 128 drwxrwxrwx 11 nobody users 187 Jan 29 20:02 disk1/
                 128 drwxrwxrwx 16 nobody users 282 Jan 29 20:02 disk2/
                   1 drwxrwxrwt  2 nobody users  40 Jan 29 17:46 disks/
                   1 drwxrwxrwt  2 nobody users  40 Jan 29 17:46 remotes/
                   ? d?????????  ? ?      ?       ?            ? user/
18302628885633695872 drwxrwxrwx  1 nobody users 187 Jan 29 20:02 user0/

 

 

Edited January 29, 2022 by martinjuhasz
carful

[martinjuhasz]

whoops, i missed the error, edited output

[Squid]

Yeah, shfs definitely crashed.  Can't see why exactly but the resolution is the same.  Reboot and go from there.

[martinjuhasz]

uhrg ok. any idea what i can try next? i rebooted and mover seems to move the file of the newly created test2-share now.

[Squid]

You can post a new set of diagnostics, but I suspect that they won't show anything.  This might boil down to a poltergeist deciding to have fun with you.  We'll just have to see if the problem repeats itself.

[martinjuhasz]

i created a new share, same problem

 

Jan 29 21:01:07 Winston emhttpd: shcmd (197): /usr/local/sbin/mover |& logger &
Jan 29 21:01:07 Winston root: mover: started
Jan 29 21:01:07 Winston move: file: /mnt/disk2/t3/Monsieur Claude 2.mkv
Jan 29 21:01:07 Winston kernel: shfs[17432]: segfault at 0 ip 00000000004043cc sp 0000151c89700780 error 4 in shfs[402000+c000]
Jan 29 21:01:07 Winston kernel: Code: 48 8b 45 f0 c9 c3 55 48 89 e5 48 83 ec 20 48 89 7d e8 48 89 75 e0 c7 45 fc 00 00 00 00 8b 45 fc 48 63 d0 48 8b 45 e0 48 01 d0 <0f> b6 00 3c 2f 74 43 8b 05 67 df 00 00 85 c0 78 2f e8 fe df ff ff
Jan 29 21:01:07 Winston move: create_parent: /mnt/disk2 error: Software caused connection abort
Jan 29 21:01:07 Winston move: move_object: /mnt/disk2/t3: Transport endpoint is not connected
Jan 29 21:01:07 Winston root: mover: finished
Jan 29 21:01:07 Winston root: Fix Common Problems: Other Warning: Mover logging is enabled

 

winston-diagnostics-20220129-2102.zip