November 2, 201015 yr Greetings! I have been using unRAID for nearly a year now and all has been great, loving everything about it I recently got myself an IP cam (Y-CAM Bullet). The device writes the captured footage to a 16GB SD card which is built into the camera. It can copy the files from the SD card (via FTP) to a ftp server on a customisable schedule. I have set this up and successfully managed to transfer files to my unraid server - part 1 of the task complete! So here comes the request for assistance (as i am from a DOS world and my linux knowledge is really bad at the moment). I want to try and learn more but i am hoping advice from the linux gurus on this board will assist my knowledge. What i want to essentially do is manage all the video files and have them archived on my unraid for a period of time (say 1 month) and then clear all the old footage. I want to try and limit the volume of footage i keep to approximate 400GB, which is roughly 1 month worth of footage. So here is what i have... Unraid Server: File & folder structure: /mnt/disk4/ycam/temptransfer (Folder containing the video files which are copied from the CCTV camera to the unraid server automatically between 2AM and 4AM every day) /mnt/disk4/ycam/YYYY-MM-DD (Folders containing the CCTV camera files, organised by date) CCTV Camera video file naming: S_2010-11-01_16-22-42_180.mov <prefix>_YYYY-MM-DD_HH-MM-SS_<file length in seconds>.mov Pseudo code: Sorting Script 1. Everyday at 04:00 (4am) run this script on the unraid server 2. Script will scan source files folder (/mnt/disk4/ycam/temptransfer) and arrange the files as follows: 2.1. Determine the date of the file (by parsing the filename) and check to see if a folder exists in /mnt/disk/ycam folder which reflects the filename YYYY-MM-DD (the date component) 2.2. If not then create folder YYYY-MM-DD based on the date component of the filename 2.3. Move all files for corresponding day into appropriate folder as long as file doesn’t already exist in the folder (ignore duplicate files, just leave them in the temptransfer folder) 3. Delete all files in temptransfer folder (essentially cleans out duplicate files) Pseudo code: Cleaning Script 1. Every Sunday at 04:00 (4am) run this script on the unraid server 2. Script will scan folder /mnt/disk4/ycam and delete all folders (and their containing files) over 1 month old (not sure if this is best accomplished using the folder atributes or by parsing, the folder name in this instance) I was thinking if maybe the script can have 'variables' used which a n00b like me can edit to change the way things work within the scripts such as source file folders etc.. It may also mean that the scripts can be used by other members in their own particular setup. Appreciate any assistance and advice you may have, thanks for taking the time to read this!
November 4, 201015 yr Author Anyone want to take a stab...even some general pointers would be greatly appreciated. Cheers!
November 5, 201015 yr If it were me I would write a small Perl program for this. Perl is excellent for handling files, directories, etc. It's available as a software package that can be installed from unmenu (if you have unmenu installed then click the pkg_manager link at the top of the unmenu page then scroll down until you find perl.) I'm sure others may have other suggestions but Perl is my system-admin programming tool of choice. Note: I haven't installed Perl on my unraid server because I don't (yet) have a need for it there. I do use Perl on many servers at work though and it works very well. You can Google for information on how to accomplish specific tasks using perl. There are also many good books available on learning perl. If you decide to go this route and you get stuck then I may be able to help you with your program whenever I have some free time. Good luck!
November 5, 201015 yr Attached is a starting point for you. You'll need to keep it on your flash drive and add two lines to your "config/go" script to copy it to the /etc/cron.daily folder every time you reboot. If you invoke this line at the command prompt you can see how the date is extracted from the file name echo S_2010-11-01_16-22-42_180.mov | sed "s/\(.*\)_\(20[0-9][0-9]\)-\([0-9]*\)-\([0-9]*\)_\(.*\).mov/\2-\3-\4/" It works by using the stream editor to match and print the parts of the original file name string. There are three variables in the program near the top. They are currently: TRANSFER_DIR=/mnt/disk4/ycam/temptransfer STORAGE_DIR=/mnt/disk/ycam DAYS_HISTORY=30 If you edit the file, use an editor that does not add carriage returns to the ends of the file. (do not use notepad, or word, or any windows editor) Easiest is to use the editor built into mc If you put a copy of this on your flash drive you need to add two lines like this to the end of your config/go script cp /boot/move_security.sh /etc/cron.daily/move_security.sh chmod +x /etc/cron.daily/move_security.sh Scripts in the cron.daily directory on the unRAID server are executed once each day at 4:40 AM. I figure this is close enough for your usage. You can do the research using "google" for details on any of the commands in the script. To locate the manual page for any given command search for man find or man sed or man mkdir etc... Note that the storage disk is as you requested /mnt/disk/ycam, but that is not one of the disks on unRAID. If you leave the script as it is the files will be copied to memory and not to a physical disk. I suspect you want the /mnt/disk4/ycam directory instead. It will be your first task... To correctly set the storage directory in the script. Joe L. move_security.zip
November 5, 201015 yr Author thank you so much for taking the time to write that script! It is exactly the starting point i needed (and much more!!!) OK, so I have been playing around with the script for the last hour (okay, okay the last two or three hours) and i am stuck on an error. As you correctly pointed out, the directory i mentioned in my original post was innacurate, this has been changed to /mnt/disk4/ycam in the script. So after making that one change, i ran the script. The error i am getting is: cp: cannot stat `/mnt/disk4/ycam/temptransfer/2010-11-04//mnt/disk4/ycam/temptransfer/S_2010-11-04_14-58-33_180.mov': No such file or directory the error repeats 13 times for each of the 13 .mov files in the temptransfer directory. The confusing bit is the copy command occurs after the mkdir command, however no directories are created when the script is run? The error above has some problem with paths maybe (as i can see there are two // in the path which is not right) and also i think the source and destination are in the wrong order? I did try to make changes, but didnt really get anywhere (apart from being able to remove the extra forward slash).. i made some additions to the script to try and assist with debugging, which now looks like this: #$CCTV Camera video file naming: #S_2010-11-01_16-22-42_180.mov #<prefix>_YYYY-MM-DD_HH-MM-SS_<file length in seconds>.mov # # #Pseudo code: Sorting Script #1. Everyday at 04:00 (4am) run this script on the unraid server #2. Script will scan source files folder (/mnt/disk4/ycam/temptransfer) and arrange the files as follows: #2.1. Determine the date of the file (by parsing the filename) and check to see if a folder exists in /mnt/disk/ycam folder which reflects the filename YYYY-MM-DD (the date component) #2.2. If not then create folder YYYY-MM-DD based on the date component of the filename #2.3. Move all files for corresponding day into appropriate folder as long as file doesn.t already exist in the folder (ignore duplicate files, just leave them in the temptransfer folder) #3. Delete all files in temptransfer folder (essentially cleans out duplicate files) #2. Script will scan source files folder (/mnt/disk4/ycam/temptransfer) and arrange the files as follows: TRANSFER_DIR=/mnt/disk4/ycam/temptransfer STORAGE_DIR=/mnt/disk4/ycam DAYS_HISTORY=30 find "${TRANSFER_DIR}" -type f -name "*.mov" -print | while read image_file do #2.1. Determine the date of the file (by parsing the filename) file_date=`echo $image_file | sed "s/\(.*\)_\(20[0-9][0-9]\)-\([0-9]*\)-\([0-9]*\)_\(.*\).mov/\2-\3-\4/"` #check to see if a folder exists in /mnt/disk/ycam folder which reflects the filename YYYY-MM-DD (the date component) echo $file_date echo $STORAGE_DIR echo $image_file if [ ! -d "${STORAGE_DIR}/${file_date}" ] then #2.2. If not then create folder YYYY-MM-DD based on the date component of the filename mkdir -p "${STORAGE_DIR}/${file_date}" fi #2.3. copy all files for corresponding day into appropriate folder as long as file doesn.t already exist in the folder (ignore duplicate files, just leave them in the temptransfer folder) if [ ! -f "${STORAGE_DIR}/${file_date}"/${image_file} ] then cp "${TRANSFER_DIR}/${file_date}"/${image_file} "${STORAGE_DIR}/${file_date}/" else # file already exists in STORAGE_DIR, just delete it. rm "${TRANSFER_DIR}/${file_date}"/${image_file} fi done # delete old files and empty directories find "${STORAGE_DIR}" -type f -mtime +$DAYS_HISTORY -exec /bin/rm -f '{}' \; find "${STORAGE_DIR}"/[0-9][0-9][0-9][0-9]-* -depth -type d -empty -exec /bin/rmdir '{}' \; the output is: 2010-11-04 /mnt/disk4/ycam /mnt/disk4/ycam/temptransfer/S_2010-11-04_14-58-33_180.mov cp: cannot stat `/mnt/disk4/ycam/temptransfer/2010-11-04//mnt/disk4/ycam/temptransfer/S_2010-11-04_14-58-33_180.mov': No such file or directory so it looks like it is getting the right data into the first two variables, but again no folder is being created. Not sure, but i think the $image_file variable is also using the entire path, which could be causing the problem as well? Any ideas? Sorry if i have missed something really obvious....my programming skills leave a lot to be desired! thanks again Joe and thanks Seven for your advice... ill look into Pearl anyhow just to see what more i can learn!
November 5, 201015 yr thank you so much for taking the time to write that script! It is exactly the starting point i needed (and much more!!!) You are welcome... OK, so I have been playing around with the script for the last hour (okay, okay the last two or three hours) and i am stuck on an error. You did exactly what you needed to do, add a few echo statements to see what was going on. To actually "watch" what is happening you can invoke the script with sh -xv move_security.sh Invoking it in that way prints the lines before they are evaluated and also shows you the expanded results of all the shell variables. The line that is wrong is this one cp "${TRANSFER_DIR}/${file_date}"/${image_file} "${STORAGE_DIR}/${file_date}/" It should be: cp "${image_file}" "${STORAGE_DIR}/${file_date}/" In the same way, two lines below, the remove of dupes should be: rm "${image_file}" image_file already is the full path to the file. No need to concatenate to it to form the source file name. (I was mistakenly concatenating the TRANSFER_DIR and file_date, and together they all made up the long file name you saw in the error output of cannot "stat") As far as it not making the directory... I see nothing obvious, but if you remove the "-p" option from the mkdir command it will print more error output if it cannot succeed. It is not really needed as long as the "ycam" directory already exists on disk4.
November 5, 201015 yr Author wow - now that is what i call a speedy reply!!! ok, things are getting interesting now So changing the copy command worked! (i actually figured it out myself just after i posted - but im glad you responded with the same thing i got!). The tip on stepping through the script is also very useful...helps a lot! Also, directories are now being created as expected - yaay...not sure why it was not working before, but it is now so great! Only thing which is left to figure out is after the script has run, all the "source" .mov files remain in the temptransfer directory. So it seems like the last part of the script (# delete old files and empty directories) is not functioning, or a more likely explanation is i dont understand it fully! Anyway, I was thinking of changing the script to move the files from the temptransfer folder, instead of copying them (i guess this will save having to clean the temptransfer folder out). I guess this would just mean using mv instead of cp Also, as a test, i re-ran the script file twice (one after the other), and the second pass simply tries to copy the files from the temptransfer folder again - so i think there is something in the last programming loop which isn't quite picking up that the files already exist in the folder which were created in the first exectution of the script. Getting there ... programming/debugging is fun hey thanks
November 5, 201015 yr wow - now that is what i call a speedy reply!!! ok, things are getting interesting now So changing the copy command worked! (i actually figured it out myself just after i posted - but im glad you responded with the same thing i got!). The tip on stepping through the script is also very useful...helps a lot! Also, directories are now being created as expected - yaay...not sure why it was not working before, but it is now so great! Only thing which is left to figure out is after the script has run, all the "source" .mov files remain in the temptransfer directory. So it seems like the last part of the script (# delete old files and empty directories) is not functioning, or a more likely explanation is i dont understand it fully! Anyway, I was thinking of changing the script to move the files from the temptransfer folder, instead of copying them (i guess this will save having to clean the temptransfer folder out). I guess this would just mean using mv instead of cp Also, as a test, i re-ran the script file twice (one after the other), and the second pass simply tries to copy the files from the temptransfer folder again - so i think there is something in the last programming loop which isn't quite picking up that the files already exist in the folder which were created in the first exectution of the script. Getting there ... programming/debugging is fun hey thanks yes, changing the command from "cp" to "mv" would do it. You are right, I did not delete the files from the temptransfer directory. You are well on your way. Good luck. The tricky part is the "sed" script to extract the date from the file name. Google "regex" (regular expressions) for how to figure those out. Basically, it is just a pattern matching language.
November 5, 201015 yr So it seems like the last part of the script (# delete old files and empty directories) is not functioning, or a more likely explanation is i dont understand it fully! The last two lines deal with deleting files in the history directory that are more than 30 days old and then, if the result leaves a directory empty, deleting the empty directory. You were creating an empty directory because of the logic error, not copying anything to it. (leaving it EMPTY) and then, you guessed it, Deleting any files older than 30 days from in it (there were none. but it is still EMPTY) Then... deleting any empty directories... (Yes... it deleted the empty directory, exactly as coded) So, when you went to look afterwords, there was no directory... but it had created it, and then deleted it when it found it was empty. You just did not look while it was there (for that fraction of a second). Joe L.
Archived
This topic is now archived and is closed to further replies.