October 20, 200916 yr i've generally gone with the process of just having data on the array while i built my new server and got everything running. as we're now coming into summer here, i was thinking about arranging my data properly on the drives. all seasons from a tv series on one drive, all HD movies on 1 or 2 drives, and so forth what would be the best way to go about moving the data in this way? not specific instructions, just generally. what i was thinking was - stopping the array - moving data - starting array is there anything i need to look out for in the process of doing this?
October 20, 200916 yr - stopping the array - moving data - starting array Why would you want to do this?? That would defeat the whole idea of having unRaid, and your data protected. If you do it, and if some of your disks craps out while you're doing it, then you are screwed. Purko
October 20, 200916 yr Indeed I don't get why would you stop the array... Are you sure you understand the concept of unRAID or a NAS?
October 20, 200916 yr Indeed I don't get why would you stop the array.. He wants to get around the slow write speeds with parity protection, and then just rebuild the parity later. Bad idea all around.
October 20, 200916 yr Author what im trying to achieve.. let's say i have a tv show. the structure looks like this tvseries - show -- season 01 --- show.s01e01.avi --- show.s01e02.avi .. --- show.s01e24.avi all of those episodes right now are spread out on multiple disks. the problem with this is that if i enable the disk spin down option, all of the drives these files reside on need to be spun up again. what i'm trying to achieve is spinning up the smallest amount of disks as possible. if i can just ssh to the array and move the data on the command line then i'll do that.
October 20, 200916 yr what im trying to achieve.. let's say i have a tv show. the structure looks like this tvseries - show -- season 01 --- show.s01e01.avi --- show.s01e02.avi .. --- show.s01e24.avi all of those episodes right now are spread out on multiple disks. the problem with this is that if i enable the disk spin down option, all of the drives these files reside on need to be spun up again. what i'm trying to achieve is spinning up the smallest amount of disks as possible. if i can just ssh to the array and move the data on the command line then i'll do that. Yes you can just ssh in and mv the files around (some people prefer to use mc for a graphical interface). As above this will be slow due to the parity writes at the same time. You could disable parity, move the files and then rebuild parity but you will have to be comfortable that you have a window where your data is not protected. Are you running the cache dir script? I have a similar setup to you but don't have too much of a disk spin up issue. You could also set your split level for your TV share such that the contents of each individual 'season' folder are kept on the same drive to minimise potential spin up. The latter won't help you immediately but for future additions...
October 20, 200916 yr all of those episodes right now are spread out on multiple disks. the problem with this is that if i enable the disk spin down option, all of the drives these files reside on need to be spun up again. what i'm trying to achieve is spinning up the smallest amount of disks as possible. Now this makes sense. if i can just ssh to the array and move the data on the command line then i'll do that. Yes, just telnet to the unRaid and do that. You may want to use the "mc" tool that's in unRaid for moving lots of stuff around. Better yet, you can use mc on top of screen. This way you can stat a humungus copy process, disconnect from screen, terminate your telnet session and go to bed. The next day you log back in, reattach to screen and see how far the work is done. No need to stop your array for any of that. Good luck! Purko
October 20, 200916 yr This will help. It i a hack and I am no coder but it works (I call it undist.sh): #!/usr/bin/bash #Get a list of drives in use VALUES=($(mount | awk '/disk[0-9]/ {print $3}')) #Get the intersting bit of the foldeer this tool was called from #either from command line in form "TV/A" or if no switch PWD if [ "$#" = "0" ] then FOLDER=`pwd | sed 's/\/mnt\/user\(\/.*\)/\1/'` else FOLDER=$1 fi echo "Folder: ${FOLDER}"; #Loop through the values of the drive that are installed for ((i=0; i<${#VALUES[@]}; i++)) do if [ -d "${VALUES[$i]}$FOLDER" ] then #Get the size on disk of the folder but on one of the disks fld=($(du -sh "${VALUES[$i]}$FOLDER")); #Get the space free on this same disk dsk=($(df -h "${VALUES[$i]}" | sed -n '2p'|awk '{print $4}')); echo "Disk:${VALUES[$i]} Using:${fld} Free:${dsk}"; fi done Call the script from the shell in the usersahre you want to find out where all the data is like: root@TOWER:/mnt/user/TV - HD/A# /boot/scripts/undist.sh Folder: /TV - HD/A Disk:/mnt/disk6 Using:0 Free:11G Disk:/mnt/disk7 Using:4.0K Free:11G Disk:/mnt/disk11 Using:1.5G Free:11G Disk:/mnt/disk4 Using:8.9G Free:5.0G Disk:/mnt/disk14 Using:4.0K Free:11G So you can see unRAID has spread around my data when it doesnt need to be. So i move some data around and run it again: root@JABBA:/mnt/user/TV - HD/A# /boot/scripts/undist.sh Folder: /TV - HD/A Disk:/mnt/disk11 Using:1.5G Free:11G Disk:/mnt/disk4 Using:8.9G Free:5.0G ta da
October 20, 200916 yr This will help. It i a hack and I am no coder but it works (I call it undist.sh): #!/usr/bin/bash #Get a list of drives in use VALUES=($(mount | awk '/disk[0-9]/ {print $3}')) #Get the intersting bit of the foldeer this tool was called from #either from command line in form "TV/A" or if no switch PWD if [ "$#" = "0" ] then FOLDER=`pwd | sed 's/\/mnt\/user\(\/.*\)/\1/'` else FOLDER=$1 fi echo "Folder: ${FOLDER}"; #Loop through the values of the drive that are installed for ((i=0; i<${#VALUES[@]}; i++)) do if [ -d "${VALUES[$i]}$FOLDER" ] then #Get the size on disk of the folder but on one of the disks fld=($(du -sh "${VALUES[$i]}$FOLDER")); #Get the space free on this same disk dsk=($(df -h "${VALUES[$i]}" | sed -n '2p'|awk '{print $4}')); echo "Disk:${VALUES[$i]} Using:${fld} Free:${dsk}"; fi done Call the script from the shell in the usersahre you want to find out where all the data is like: root@TOWER:/mnt/user/TV - HD/A# /boot/scripts/undist.sh Folder: /TV - HD/A Disk:/mnt/disk6 Using:0 Free:11G Disk:/mnt/disk7 Using:4.0K Free:11G Disk:/mnt/disk11 Using:1.5G Free:11G Disk:/mnt/disk4 Using:8.9G Free:5.0G Disk:/mnt/disk14 Using:4.0K Free:11G So you can see unRAID has spread around my data when it doesnt need to be. So i move some data around and run it again: root@JABBA:/mnt/user/TV - HD/A# /boot/scripts/undist.sh Folder: /TV - HD/A Disk:/mnt/disk11 Using:1.5G Free:11G Disk:/mnt/disk4 Using:8.9G Free:5.0G ta da Very very useful, thanks.
October 20, 200916 yr Why would you want to do this?? That would defeat the whole idea of having unRaid, and your data protected. If you do it, and if some of your disks craps out while you're doing it, then you are screwed. Purko The main thinking behind this operation is "speed". If user has only couple hundreds MB or few GB of data to move then it is fine to keep RAID intact while moving, otherwise when moving hundreds of GB or even TB of data while keeping RAID intact, this will take long time to finish because of those parity read/write. Last time when i did it, i took the parity disk offline then rearrange data among data disks then put parity disk back and kick off parity re-generation. It is true while taking parity offline user will lose data protection while moving data and if there are other disk failures at same time but if this is the risk s/he willing to take then it has no harm to move data in this fashion.
October 22, 200916 yr Could you move the data to the cache drive and then set the shares (drives, split level, fill level etc) correctly and run the mover? If you have a lot of small files spread out all over the place I could see the moves to the cache to be much faster when sorting the data. Run the mover once you are done sorting for the evening. You could also copy and allow the mover to put it in it's new spot and then go back and delete the old files. Peter
October 22, 200916 yr Could you move the data to the cache drive and then set the shares (drives, split level, fill level etc) correctly and run the mover? If you have a lot of small files spread out all over the place I could see the moves to the cache to be much faster when sorting the data. Run the mover once you are done sorting for the evening. You could also copy and allow the mover to put it in it's new spot and then go back and delete the old files. Peter That's a pretty good idea!
October 23, 200916 yr Could you move the data to the cache drive and then set the shares (drives, split level, fill level etc) correctly and run the mover? If you have a lot of small files spread out all over the place I could see the moves to the cache to be much faster when sorting the data. Run the mover once you are done sorting for the evening. You could also copy and allow the mover to put it in it's new spot and then go back and delete the old files. Peter That's a pretty good idea! It is, but it does still put your data at temporary risk, since the cache drive is not part of the protected array. If the cache drive were to fail during this process, you would lose data.
Archived
This topic is now archived and is closed to further replies.