February 21, 201313 yr So I'm copying data for my 2TB disk share on 4.7 to my test 5.0 array's 2TB disk share. Weird thing is it's saying that there isn't enough space on the 5.0 array's disk. and that I need an additional 945MB of disk space. How can that be? does the file system of 5.0 use more space to store the same files? Both are 4K aligned
February 21, 201313 yr You're not, by any chance, running a Gigabyte motherboard, or using drives that have, at some time, been connected to a Gigabyte motherboard?
February 21, 201313 yr Author You're not, by any chance, running a Gigabyte motherboard, or using drives that have, at some time, been connected to a Gigabyte motherboard? hmm nope. ASUS. connected via the BR10i
February 21, 201313 yr Are you using "windows" to make the copy? If so, I think one of the only reasons it ever gives is out-of-space (other than no permission) In any case, it can easily happen in one other way... sparse files. These are files that contain mostly empty space (all zeros) and in UNIX/Linux it is possible to create files like this that appear to be huge, but in reality are not. Since the space is all zeros, Linux does not allocate actual data blocks to store the file. It only allocates the spaces with non-zero data. Not every command will take advantage of this, but some do... (the "dd" command can create sparse files) As an example you can type: echo "hello" | dd of=/tmp/joe seek=2G It will create a file that actually only uses 1 block of space. However, if you try to list it, it will look like this root@Tower3:/boot# ls -lh /tmp/joe -rw-r--r-- 1 root root 1.1T Feb 21 12:05 /tmp/joe The "seek" is in 512 byte blocks, so the size ends up around 1.1TB. Trust me, I do NOT have 1TB of ram on my server. If I were attempt to copy that file with a utility that did not recognize sparse files, it would fail if the target drive had less than 1.1TB of free space. These types of files have been in Linux/Unix forever, and have caused grief for system administrators attempting to create backups to tape. (The tape drives 25 years ago had nowhere near terabytes of space) Who knows, you might have run into one of these files. To remove the sparse file you created in the above experiment, just type rm /tmp/joe In any case, you can tell the actual free space on a disk by typing df Many copy programs copy to a temporary name and then rename the file once the copy is completed to its final name. If you are overwriting an existing file you can run out of room that way too (not enough room for both the temp file and the original file at the same time) Joe L.
February 21, 201313 yr Author Are you using "windows" to make the copy? If so, I think one of the only reasons it ever gives is out-of-space (other than no permission) In any case, it can easily happen in one other way... sparse files. These are files that contain mostly empty space (all zeros) and in UNIX/Linux it is possible to create files like this that appear to be huge, but in reality are not. Since the space is all zeros, Linux does not allocate actual data blocks to store the file. It only allocates the spaces with non-zero data. Not every command will take advantage of this, but some do... (the "dd" command can create sparse files) As an example you can type: echo "hello" | dd of=/tmp/joe seek=2G It will create a file that actually only uses 1 block of space. However, if you try to list it, it will look like this root@Tower3:/boot# ls -lh /tmp/joe -rw-r--r-- 1 root root 1.1T Feb 21 12:05 /tmp/joe The "seek" is in 512 byte blocks, so the size ends up around 1.1TB. Trust me, I do NOT have 1TB of ram on my server. If I were attempt to copy that file with a utility that did not recognize sparse files, it would fail if the target drive had less than 1.1TB of free space. These types of files have been in Linux/Unix forever, and have caused grief for system administrators attempting to create backups to tape. (The tape drives 25 years ago had nowhere near terabytes of space) Who knows, you might have run into one of these files. To remove the sparse file you created in the above experiment, just type rm /tmp/joe In any case, you can tell the actual free space on a disk by typing df Many copy programs copy to a temporary name and then rename the file once the copy is completed to its final name. If you are overwriting an existing file you can run out of room that way too (not enough room for both the temp file and the original file at the same time) Joe L. yup, using windows to copy .. odd when i run the df, it shows 1% of disk2 used, but to my knowledge, there's nothing there.. Linux 3.4.26-unRAID. root@unraid5:~# echo "hello" |dd of=/tmp/joe seek=2G 0+1 records in 0+1 records out 6 bytes (6 B) copied, 0.000349796 s, 17.2 kB/s root@unraid5:~# ls -lh /tmp/joe -rw-rw-rw- 1 root root 1.1T 2013-02-21 13:03 /tmp/joe root@unraid5:~# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 31243264 41136 31202128 1% /boot /dev/md1 1953454928 1746747984 206706944 90% /mnt/disk1 /dev/md2 1953454928 32840 1953422088 1% /mnt/disk2 root@unraid5:~# rm /tmp/joe
February 21, 201313 yr Author can someone give me a linux command line to copy everything from disk2 on tower to disk2 on unraid5 ? something like this: http://www.crucialp.com/resources/tutorials/server-administration/how-to-copy-files-across-a-network-internet-in-unix-linux-redhat-debian-freebsd-scp-tar-rsync-secure-network-copy.php
February 22, 201313 yr Author i tried this, and it didn't work: rsync -ve ssh -r -S -W -h //tower/disk2/ /mnt/disk2
February 24, 201313 yr Author See here: http://lime-technology.com/forum/index.php?topic=13432.0 thanks, i cheated and used terracopy
February 24, 201313 yr can someone give me a linux command line to copy everything from disk2 on tower to disk2 on unraid5 ? something like this: http://www.crucialp.com/resources/tutorials/server-administration/how-to-copy-files-across-a-network-internet-in-unix-linux-redhat-debian-freebsd-scp-tar-rsync-secure-network-copy.php This is based on what I use to create a copy of my main machine (Tower-V5) on my backup machine (BackupServer)... rsync -rltDvH --delete --timeout=600 --progress /mnt/disk2/* BackupServer::mnt/disk2 Note that with the --delete option, deletions are mirrored as well. That is not what everybody would want, so check the rsync command line options that suit your needs best. I also make sure that the rsync daemon is running on the destination server with this line in the /boot/config/go file on that machine... #rsync daemon rsync --daemon --config=/boot/config/rsyncd.conf And I created the /boot/config/rsyncd.conf file on the destination server containing this... uid = nobody gid = users use chroot = no max connections = 4 pid file = /var/run/rsyncd.pid timeout = 600 log file = /var/log/rsyncd.log incoming chmod = Dug=rwx,Do=rx,Fug=rw,Fo=r [mnt] path = /mnt comment = /mnt files read only = FALSE It's probably not perfect (I am certainly no rsync expert) but it works for me.
February 24, 201313 yr Author can someone give me a linux command line to copy everything from disk2 on tower to disk2 on unraid5 ? something like this: http://www.crucialp.com/resources/tutorials/server-administration/how-to-copy-files-across-a-network-internet-in-unix-linux-redhat-debian-freebsd-scp-tar-rsync-secure-network-copy.php This is based on what I use to create a copy of my main machine (Tower-V5) on my backup machine (BackupServer)... rsync -rltDvH --delete --timeout=600 --progress /mnt/disk2/* BackupServer::mnt/disk2 Note that with the --delete option, deletions are mirrored as well. That is not what everybody would want, so check the rsync command line options that suit your needs best. I also make sure that the rsync daemon is running on the destination server with this line in the /boot/config/go file on that machine... #rsync daemon rsync --daemon --config=/boot/config/rsyncd.conf And I created the /boot/config/rsyncd.conf file on the destination server containing this... uid = nobody gid = users use chroot = no max connections = 4 pid file = /var/run/rsyncd.pid timeout = 600 log file = /var/log/rsyncd.log incoming chmod = Dug=rwx,Do=rx,Fug=rw,Fo=r [mnt] path = /mnt comment = /mnt files read only = FALSE It's probably not perfect (I am certainly no rsync expert) but it works for me. awesome thanks - helps me keep my test machine fresh, and relevant throughout the test process.
Archived
This topic is now archived and is closed to further replies.