So I wanted to replace some of my old small drives with a bigger one. I found this guide from unraid's wiki:

https://wiki.unraid.net/Replacing_Multiple_Data_Drives_with_a_Single_Larger_Drive

I followed the "Safer method" and at the second part where I copy all the data from the small drive to the bigger one using the "rsync -avX ..." command even though I have doublechecked from the "main" tab that the destination (big new) drive had more than enough space for the data of the smaller drive the proccess has stopped with the following message:

 

rsync: [receiver] write failed on "/mnt/disk1/*******.mp4": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)

 

What am I suppose to do now?

 

I searched for a duplicate file finder plugin for unraid so I can find all the files that have been copied to the big drive from the small one and delete them so I can redo the process with another way maybe.

 

Please help if anyone has any clue or idea

 

Thank u

10 hours ago, Vr2Io said:

A simple question, how much source files size, destination used and free space.

 

Obviously I don't have the destination used and free space anymore since the destination drive is full now but the source is 714GB and the 2TB destination disk had 1.4+ TB free.

I am thinking that this rsync does something more than just copying files. I am not an expert but I thought that the official guide would be a foolproof process.

3 hours ago, tsakodim said:

I am thinking that this rsync does something more than just copying files.

It doesn't, but if there were any sparse files on source (like vdisks) they won't be sparse on dest unless you use the appropriate flag.

9 hours ago, JorgeB said:

It doesn't, but if there were any sparse files on source (like vdisks) they won't be sparse on dest unless you use the appropriate flag.

Sorry my english are not that good so I have to google what sparse file is. Do you know how can I check if there were any sparce files? And what is right flag and why it was not mentioned on the wiki guide 🤬

 

Thanks for your help really appreciate it

If you have any VM vdisks on that disk they will be sparse, sparse means that if you for example select 500G for the vdisk size but the OS only uses 50G, it will only occupy 50G on the disk, it can then grow as needed, if it gets copied/moved to another disk without using the flag it will occupy 500G on dest, instead of just 50G.

 

rsync flag for this is:

--sparse

 

Should the online manual page linked to in the 1st posting be updated to include this —sparse option to save future users from the same problem?

I searched the dest disk for vdisks it only had one 30GB. The source disk had none though. So there is no way I can understand how the 700GB of the source disk grew that much to occupy the 1.4TB of the dest disk and without any vdisk in source disk.

 

By the way if I repeat the rsync command with the --sparse flag will it replace and fix these elusive large files that occupy all this space?

 

And what about the --inplace flag could that help?

 

Thanks in advance

 

 

 

 

Maybe some information missing from this discussion.

 

Post your Diagnostics.

On 6/7/2021 at 9:43 PM, trurl said:

Post your Diagnostics

Do you mean the tools/diagnostics and then the download button and post the file?

45 minutes ago, tsakodim said:

Do you mean the tools/diagnostics and then the download button and post the file?

Yes.  

nas-diagnostics-20210608-0051.zip

Assuming you copied the data directly to the disk, i.e., without using a specif folder or share and it's now together with the data that already existed on that disk I would remove the disk from the array, then run rsync again but now from an unassign disk to the array, you'll need to do it multiple times if there are multiple shares on source, but it will start where it left of, i.e., just copy whatever is missing, and since the array is now the dest no issues with space.

21 hours ago, JorgeB said:

Assuming you copied the data directly to the disk, i.e., without using a specif folder or share and it's now together with the data that already existed on that disk I would remove the disk from the array, then run rsync again but now from an unassign disk to the array, you'll need to do it multiple times if there are multiple shares on source, but it will start where it left of, i.e., just copy whatever is missing, and since the array is now the dest no issues with space.

 

I think I have understood the process but as I mentioned the rsync was not completed because dest disk got full so now I don't know what files have not been transfered and are still only on the source disk. Unless I can do the process you said for both the source and the dest disk and the duplicate files will just be skiped or overwritten.

4 hours ago, tsakodim said:

rsync was not completed because dest disk got full so now I don't know what files have not been transfered and are still only on the source disk.

 

4 hours ago, tsakodim said:

Unless I can do the process you said for both the source and the dest disk and the duplicate files will just be skiped or overwritten.

That's not a problem, it will continue from where it left off, files already copied will be skipped.

 

 

After removing the disk from the array and mounting it with UD you can do this:

 

rsync -av /mnt/disks/UD_disk_name/share/ /mnt/user/share/

 

Change UD_disk_name and share to the actual names and if there are multiple shares on source repeat for each one, any files on the root of that disk won't be copied this way, but there shouldn't be any, and you can easily check.

One more thing make sure shares to be copied are set to "split any directory", since rsync creates all the folders before starting to copy, it will fail if it's not.

Ok I think I got it now thank u very much.

 

One more question - what about the dest disk that is now full. Do I just move files with mc for example to other disks?

 

And since its still a mystery why the disk got full from files half its free space do I have to worry about duplicate files and use a tool to fix it?

 

Thanks again

9 hours ago, tsakodim said:

Do I just move files with mc for example to other disks?

You can, just move form disk to disk, don't involve shares.

 

9 hours ago, tsakodim said:

And since its still a mystery why the disk got full from files half its free space do I have to worry about duplicate files and use a tool to fix it?

rsync can't create more files than existed on source, and there can't be any duplicates since it was writing to the same disk, unless you used multiple target folders.

I followed your advice everything ok, thanks again.

 

I did a stupid thing though. I changed the disks assignment and my dockers disappeared. Then I moved my docker.img from the old disk1 to the new disk1 and now i got the error -  Docker Service failed to start.

 

 

You can easily recreate the docker image:

https://forums.unraid.net/topic/57181-docker-faq/?do=findComment&comment=564309

 

On 6/11/2021 at 9:25 AM, JorgeB said:

You can, just move form disk to disk, don't involve shares.

Hi again, so I just moved quite a lot of data with mc from disk to disk to equalize the load on the disks but still at my main tab the source disk that was the almost full disk is still almost full even though the data on target disk that I moved seems to has been moved. Meaning that the capacity of the target disk has changed.

 

I did a parity check but still the same. Do I have to do the move with the array stopped?

 

What on earth am I doing wrong???

What on earth am I doing wrong???

I have no idea, if the files were really moved source files would be deleted, if it was a specific fodler/share you can run rsync with --remove-source-files so any existing files on dest will be skipped but source files that already exist on dest will still be deleted from source.

11 hours ago, JorgeB said:

you can run rsync with --remove-source-files

So u mean to run

rsync -avX --remove-source-files /mnt/"source disk"/"source folder" /mnt/"dest disk"/"dest folder"

You're missing the trailing slash on source, that will create a new folder on dest with the "dest folder" name, i.e., /mnt/dest_disk/folder/folder/, trailing slash on dest is optional but to avoid confusion I recommend always using on both, e.g.:

 

rsync -avX --remove-source-files /mnt/disk1/folder/ /mnt/disk2/folder/

 

 

Stop array and power down. Add new larger drive and power up. Stop array and use Tools - New Config. Assign new larger drive to new slot, unassign parity drive, and start array. Copy all files from the smaller drives to the larger drive. ... Stop array and power down. Remove smaller drives and power up.