Mover not moving files because "File exists"

[Cressio]

To try and keep it succinct, I'm trying to using Lidarr combined with a script that integrates Deemix (https://github.com/RandomNinjaAtk/arr-scripts), and the default configuration of the script makes it to where it downloads in the appdata folder (I tried to change this, but the behavior got weird so I was planning to just keep it this way for now). Lidarr then atomic moves the music files into my designated music directory (which isn't a child folder of appdata).

Unraid then seems to get confused and won't move these files because they "exist" already. Well... they don't exist where I want them. My appdata is set to use my NVMe cache, while my music directory is set to be on my array, but Unraid won't move them. I've checked the disks and files individually and they are indeed not on the array but instead on the cache disk and permanently filling up my cache. Any idea how to fix this?

 

This is the Github issue I posted for the script that has more details and screenshots attached. It's probably not a script issue but I just went there first to see what he had to say.

 

As far as I can tell, all my Unraid settings are correct so I'm pretty confused. Maybe I have something configured wrong though. Container PUID/PGID is 99/100. Maybe it needs to be root 0/0? Doesn't seem it like though since it theoretically is the literal root operating system that's struggling to move these files regardless who owns what. But idk considering the non root-owned files like the extra .lrc and .jpg are moving successfully maybe it has something to do with it. I just don't see why it would.

SOLUTION: files that are created on a share, and then moved to a share with different disk/mover settings, will break mover. Unraid/Linux doesn't support this, you need to handle them on the same share. Unraid's error message is incorrect and misleading. 

 

Edited April 26 by Cressio
Added solution

[itimpi]

You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread. It is always a good idea when asking questions to supply your diagnostics so we can see details of your system, how you have things configured, and the current syslog.

I strong;y suspect that you have something incorrectly configured.    It seems unlikely that mover will report files ‘exists’  when it does not so I suspect that you do not have things set up correctly to get the behaviour you want.

 

 

[Cressio]

10 minutes ago, itimpi said:

You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread. It is always a good idea when asking questions to supply your diagnostics so we can see details of your system, how you have things configured, and the current syslog.

alright I'll try to get that done. I think I just crashed the web page trying to generate the log because it's so large... as far as I can tell the only relevant mover information is the "file exists" part so maybe I'll regenerate a log without mover logging and post that for general system information. But otherwise there's gonna be roughly 10 thousand lines (number of tracks I have downloaded right now) of the same thing

 

edit: yeah the page ran out of memory trying to download the existing log lol

Edited April 22 by Cressio

[Cressio]

Little bump on this after I've added logs. I hope I have something misconfigured, that'd be an easy fix, but I can't really see where I do.

The Mover logic flow should basically be: Share files are supposed to be stored on array -> files exists on share that isn't on array -> files gets moved to array, right? The file(s) definitely don't exist on array, I checked each disk 1 by 1 and the Unraid GUI also reflects their actual location (not on array, and definitely not "exist"-ing on array). The share they're on (media) moves to the array properly in every other scenario. This is the only time I've ever had a problem with Mover and it's also the only time I've had files get downloaded and moved from appdata -> my main media share, so it makes me suspect Unraid has become confused in some way. 

[JorgeB]

Reboot to clear the logs, run the mover again and post new diags but without being anonymized, so we can see the file names.

[Cressio]

On 4/23/2024 at 12:35 AM, JorgeB said:

Reboot to clear the logs, run the mover again and post new diags but without being anonymized, so we can see the file names.

Done

 

Edited April 26 by Cressio
Removed logs for privacy

[JorgeB]

Post the output of

 

find /mnt -name 01\ -\ Cannonball.mp3

 

Search can take some time, wait until you get the cursor back before posting the output

[Cressio]

13 minutes ago, JorgeB said:

Post the output of

 

find /mnt -name 01\ -\ Cannonball.mp3

 

Search can take some time, wait until you get the cursor back before posting the output

/mnt/user/media/processed/Music/Flux Pavilion/Cannonball/01 - Cannonball.mp3
/mnt/server/media/processed/Music/Flux Pavilion/Cannonball/01 - Cannonball.mp3

[JorgeB]

Hmm, error may be caused by the file existing in a different pool, media share is set to use the pool "downloads", but those files are in the pool "server", change the share to use the pool server, other settings should remain the same, and run the mover again.

[Cressio]

24 minutes ago, JorgeB said:

Hmm, error may be caused by the file existing in a different pool, media share is set to use the pool "downloads", but those files are in the pool "server", change the share to use the pool server, other settings should remain the same, and run the mover again.

Yeah that was my thesis as well.

 

Looks like they’re actually moving now. I imagine/hope this is worthy of a bug report and not intended behavior?

[JorgeB]

IMHO it can be considered a bug regarding the errors logged, which are misleading, the behavior itself would be expected, i.e., if the share is set to use one pool, the mover won't move any data from the same share in a different pool.

[itimpi]

19 minutes ago, Cressio said:

Looks like they’re actually moving now. I imagine/hope this is worthy of a bug report and not intended behavior?

Looks like you might be falling foul of the behaviour described in the Caution in this section of the online documentation accessible via the Manual link at the bottom of the Unraid GUI.   Not sure there is much that can be done about it as it is a quirk of the way Linux handles move operations.

 

If you can configure the docker container to use Copy/Delete instead of 'move' then you would get the result you expected.  The alternative is to set that share to have the 'downloads' pool as its primary storage.

[Cressio]

On 4/24/2024 at 3:03 AM, JorgeB said:

IMHO it can be considered a bug regarding the errors logged, which are misleading, the behavior itself would be expected, i.e., if the share is set to use one pool, the mover won't move any data from the same share in a different pool.

 

On 4/24/2024 at 3:14 AM, itimpi said:

Looks like you might be falling foul of the behaviour described in the Caution in this section of the online documentation accessible via the Manual link at the bottom of the Unraid GUI.   Not sure there is much that can be done about it as it is a quirk of the way Linux handles move operations.

 

If you can configure the docker container to use Copy/Delete instead of 'move' then you would get the result you expected.  The alternative is to set that share to have the 'downloads' pool as its primary storage.

I see. I was hoping the mover would always just move to the correct disk based on its current share regardless of its originating share. Sounds like that Caution section explains why that isn't possible (although I don't fully understand it... but that's neither here nor there)

Thanks for the help. I managed to update and adjust the script to get it working all within the same share. That error *should* probably be updated and clarified if possible though so maybe I'll try to report that suggestion.