June 17, 200818 yr How does unraid handle files that are in use when the mover script kicks in? I had an issue where the mover script did nothing at all. I ran it myself with no options and it would give no output. Checking the syslog gives no lines of interest and a 'ps -ef' shows no lines with "mover" or "mv". Counters on the web page do not change. After I rebooted it worked and moved the files it could not see or move before. Later, I realized I had a DVD image file mounted in Daemon Tools over the network and I am assuming that caused it. If it did is there way to skip that file and move on?
June 18, 200818 yr If a file is busy, the mover script would probably just move the file the next time it ran (when the file is no longer busy)
June 18, 200818 yr From what I saw of the script, it's not smart enough to tell if a file is busy and do it later. (unless it has been enhanced since the last time I saw it). It's a find exec'ing a mv command. (or was at least). If anything the file might be copied, but the unlink(rm) fails because the file is busy. What should probably happen is the file's link in the original directory disappears, but the space does not get unallocated until the file is closed. If the find has been updated to not process files with a recent access time then this could have something to do with it. Can someone post the most recent mover script? I still have not upgraded to 4.3.1 as I know I'll have to recompile everything and I have a bunch of large torrents running. Sheesh one is 45G and has been running for a few weeks.
June 18, 200818 yr What should probably happen is the file's link in the original directory disappears, but the space does not get unallocated until the file is closed. That's probably exactly what happens. It would be the normal unix behavior. Any reference to the file prevents the blocks to be freed (references would include any open file descriptors) I just looked at the "mover" script in unRAID 4.3.1. As you can see below, it uses "mv" and does not do anything special to determine if the file is busy. Since the file is being moved between file systems, it would actually be a copy to the new file system followed by a "unlink" (delete) from the old file system on the cache drive. (cd /mnt/cache ; \ find . \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/.*/.*' -exec mv -v {} /mnt/user0/{} \; \) \ ) I just did an experiment, I first used daemon tools to mount an ISO image on one of my unRAID disks. Then, logging in via telnet, using "mv" I moved the ISO image file from one physical disk to a different one, I even moved it to a different folder. Looking at the output of "smbstatus" the file is locked... [pre] root@Tower:/mnt/disk3# smbstatus Samba version 3.0.28a PID Username Group Machine ------------------------------------------------------------------- 18292 guest guest _none_ (192.168.2.21) 11765 guest guest _none_ (192.168.2.21) 23440 guest guest _none_ (192.168.2.21) 2907 guest guest _none_ (192.168.2.251) 13775 joe joe dell630 (192.168.2.10) 11847 guest guest _none_ (192.168.2.21) Service pid machine Connected at ------------------------------------------------------- Movies 13775 dell630 Mon Jun 16 12:24:57 2008 Movies - All 11847 _none_ Mon Jun 16 09:39:53 2008 Movies - All 11765 _none_ Mon Jun 16 09:32:46 2008 Mp3 13775 dell630 Mon Jun 16 16:49:16 2008 Movies 18292 _none_ Mon Jun 16 19:03:29 2008 disk3 13775 dell630 Tue Jun 17 14:25:47 2008 Movies 2907 _none_ Tue Jun 17 23:05:01 2008 Movies 23440 _none_ Tue Jun 17 03:30:51 2008 Locked files: Pid Uid DenyMode Access R/W Oplock SharePath Name Time -------------------------------------------------------------------------------------------------- 13775 0 DENY_WRITE 0x81 RDONLY NONE /mnt/user/Movies Demo-Disks/sabrina_30_sec_clip.iso Wed Jun 18 00:39:55 2008 [/pre] I then tried to play the ISO image on the PC where I had it mounted via Daemon Tools. As expected, it played just fine. I then did two "df" commands, one before I unmounted the ISO image from within Daemon Tools, the other after I unmounted it. before unmount of ISO image. root@Tower:/mnt/disk3# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1000640 411824 588816 42% /boot /dev/md3 488371640 469154592 19217048 97% /mnt/disk3 /dev/md4 390699424 380706216 9993208 98% /mnt/disk4 /dev/md5 390699424 335714728 54984696 86% /mnt/disk5 /dev/md2 390699424 377142036 13557388 97% /mnt/disk2 /dev/md6 390699424 380313528 10385896 98% /mnt/disk6 /dev/md7 488371640 479828380 8543260 99% /mnt/disk7 /dev/md1 488371640 481889196 6482444 99% /mnt/disk1 /dev/md8 488371640 224050808 264320832 46% /mnt/disk8 /dev/md10 732552188 711678572 20873616 98% /mnt/disk10 /dev/md9 245109856 234457296 10652560 96% /mnt/disk9 shfs 4493946300 4074935352 419010948 91% /mnt/user After unmount of ISO image in Daemon Tools, disk blocks held by open file reference from Daemon Tools were freed. root@Tower:/mnt/disk3# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 1000640 411824 588816 42% /boot /dev/md3 488371640 469154592 19217048 97% /mnt/disk3 /dev/md4 390699424 380706216 9993208 98% /mnt/disk4 /dev/md5 390699424 335714728 54984696 86% /mnt/disk5 /dev/md2 390699424 377107752 13591672 97% /mnt/disk2 /dev/md6 390699424 380313528 10385896 98% /mnt/disk6 /dev/md7 488371640 479828380 8543260 99% /mnt/disk7 /dev/md1 488371640 481889196 6482444 99% /mnt/disk1 /dev/md8 488371640 224050808 264320832 46% /mnt/disk8 /dev/md10 732552188 711678572 20873616 98% /mnt/disk10 /dev/md9 245109856 234457296 10652560 96% /mnt/disk9 shfs 4493946300 4074901068 419045232 91% /mnt/user Note the lines highlighted in red, Unless something really odd is done with the cache drive working through the user-file-system, it should work the same way. Daemon tools would not prevent a move from occurring from the cache drive to the data drive. You could still play the movie (or continue to play it) from daemon tools. Once the file was un-mounted from within Daemon Tools, the blocks on the cache drive would be freed. This brings up an interesting scenario. What would happen if you were in the process of writing a large file on the cache drive when the mover script moved it. From what I can see, the mover script would copy the partially written file (as much as was written so far) and then unlink it from the cache drive. The process writing it would still continue to write to it, using blocks on the cache drive. When the writing was complete, the file descriptor would close and the completely written file would probably free all the blocks it used. The partially written file would be on the data disk. Ouch... I don't have a cache drive installed. Looks like somebody with one needs to do an experiment. Joe L.
June 18, 200818 yr Something has to change with the mover script. I've always known it, just have not gotten around to find the right answer yet. The -mtime parameter is not granular enough. Besides, I think it needs to use -atime. Or at least check files with the output of fuser so open files are skipped. It's a problem waiting to happen.
June 18, 200818 yr (cd /mnt/cache ; \ find . \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/.*/.*' -exec mv -v {} /mnt/user0/{} \; \) \ ) A better command sequence for the "find" command might be: (cd /mnt/cache ; \ find . \( -type d -regex '[.]/[^.].*' -exec mkdir -p /mnt/user0/{} \; \) , \ \( -type f -regex '[.]/.*/.*' ! -exec fuser -s {} \; -exec mv -v {} /mnt/user0/{} \; \) \ ) If I got my syntax right, the added call to fuser will not exec the mv if a file is busy. This should prevent the error where a partially written file is copied. Joe L.
June 23, 200818 yr WeeboTech, Joe L. - what do you think? Should we add the '! -exec fuser -s {}' clause??
June 23, 200818 yr WeeboTech, Joe L. - what do you think? Should we add the '! -exec fuser -s {}' clause?? I think that if you do not do something, it is likely that a partially written file will eventually be copied from the cache when the mover script kicks in as it is in the middle of being created, and the completely written file deleted from the cache when the last reference to the file closes. Now I did not confirm this, but I can see no reason why it would not occur. I can tell you that using a similar "find" command (with the additional ! -exec fuser call) on my Movies share would NOT list a file being played by my media player, since it had an open file descriptor reading it. Joe L.
Archived
This topic is now archived and is closed to further replies.