Issues transferring larger files from Windows PC to UnRAID <SOLVED>

[Roscoe62]

Going to have to shelve this issue for a little while. Need to get my hands on another PC I can use to set up a mock unRAID trial, but haven't much spare cash right now. Will resurrect this thread when I'm able to.

[JorgeB]

One thing you can try in the meantime is to see if you have something slow on the server  causing the Windows to timeout, that would not leave anything logged server side, I've seen similar timeouts before when using SMR disks or most free allocation method, or worse using both together.

 

Your disks are not SMR, but there could be one one more not performing correctly, also are you using most free for any of the problem shares?

 

Could you add a cache device to the server, disk or SSD, and re-test?

[Roscoe62]

Hi JorgeB,

 

Yes - by far the biggest share on the server is the 'Media Share' share, which is currently set to use the MOST FREE allocation method. This IS the share I'm having the issue with.

 

Are you saying I should change the allocation method on this share to high-water?

 

Also - going forward - I'm pretty much sold on the idea of installing an SSD in the last drive space on the server to act as a cache drive. I really didn't like seeing 3 of my data drives have now gone past the 90% allocation threshold, so I'm replacing a number of my drives with newer higher capacity drives - starting with the parity drive. Once that drive, along with those over the 90% full have been replaced, I'm keen to get the cache drive up & running.

 

These steps are not a guaranteed solution to my file transfer issue, but it's certainly not going to hurt.

[JorgeB]

10 hours ago, Roscoe62 said:

Yes - by far the biggest share on the server is the 'Media Share' share, which is currently set to use the MOST FREE allocation method. This IS the share I'm having the issue with.

That can be very slow because parity writes will overlap, I also just noticed that you are still on reiserfs, reiser will become super slow when disks start to get close to full, I would bet that converting your disks to xfs and changing to highwater/fill up would solve your problem, probably xfs with most free would also be fine, but still slower, reiser with highwater/most free could still be a problem.

 

I forgot this before because these can be difficult to track, since there won't be any errors logged on Unraid, because the server is not experiencing any errors, just being extra slow, and that can cause Windows explorer to timeout, you could also increase the Windows timeout, but IMHO the other options are a much better solution.

[Roscoe62]

Thanks JorgeB,

 

Yes, I have been seeing the "ReiserFS is deprecated..." messages appearing and knew I'd have to do something about that fairly soon too. The list continues to grow!

 

I guess I can switch the 'media share' allocation method to highwater as a first option  - as that's very easy to do. However, for a next step, would I be better to replace the parity and data drives to newer, larger drives first, or look at changing over to xfs file system first? I've currently got 8 data drives & 1 parity drive - all currently 4TB in size, so I imagine changing over to xfs might take a while. I did a quick search on posts about moving from reiserFS to xfs. I also found the fs conversion process on the legacy unRAID wiki...

 

https://legacy.wiki.unraid.net/index.php/File_System_Conversion

 

...so I'm going to read through that. In my case all the disks are old enough to warrant replacement, so I won't be reusing any of them....and the first of my new larger disks just arrived at my door 10 minutes ago. I'll read through the above procedure and hopefully I'll find something to match what I'm trying to do.

[Roscoe62]

Some further things...

 

1. Is the file system used applicable to the parity drive at all? I see on the main page of the webGUI a file system is listed for each drive except the parity drive.

2. Is it possible/advisable to have 2 file systems on the server for any length of time? i.e. some disks on Reiser & some on xfs? My new larger drives are pricey, so it could take months to replace them all.

2. The conversion to xfs looks to be something requiring a lot of planning to avoid data loss. I'm a little wary of changing the file system at the same time as replacing with larger drives. On the other hand, moving to xfs could fix my file transfer issue. <sigh!>

I value your opinion of the best way forward.

[itimpi]

1.   The parity frive has no file system.

2.   Unraid is quite happy to have a mix of file system types in the main array.

3.   If you plan to move to larger drives it would be a good idea to combine it with the move away from reiserfs.    The move requires copying the data anyway so you might as well be copying it onto new larger drives in the desired target format.

[JorgeB]

11 hours ago, Roscoe62 said:

I guess I can switch the 'media share' allocation method to highwater as a first option  - as that's very easy to do.

Yes, and that may be enough, it's been many years since I had reiser in any of my servers, but IIRC, the main issue was that it could take several seconds for a transfer to begin when the disks were getting close to full, once the transfer started it would go more or less normally, now if you add most free to that, you start a transfer, it will eventually start writing to a disk, when that file is done is will look for the next disk, and again the reiser issue will manifest itself, and this can cause Windows to timeout.

[Roscoe62]

First off, thank you for your responses. This is a BIG help!

 

22 hours ago, itimpi said:

1.   The parity frive has no file system.

 

 

This is what I suspected. Thanks for confirming. One more question on this...Once I've changed a data drive over from reiserfs to xfs, does the parity for that drive need to be re-calculated? I guess I'm asking whether there's anything in changing the file system that would necessitate a change in the parity data held?

 

22 hours ago, itimpi said:

3.   If you plan to move to larger drives it would be a good idea to combine it with the move away from reiserfs.    The move requires copying the data anyway so you might as well be copying it onto new larger drives in the desired target format.

 

Okay - will definitely incorporate the file system change with the migration to larger drives.

 

12 hours ago, JorgeB said:

Yes, and that may be enough, it's been many years since I had reiser in any of my servers, but IIRC, the main issue was that it could take several seconds for a transfer to begin when the disks were getting close to full, once the transfer started it would go more or less normally, now if you add most free to that, you start a transfer, it will eventually start writing to a disk, when that file is done is will look for the next disk, and again the reiser issue will manifest itself, and this can cause Windows to timeout.

 

I knew unRAID was moving off reiserfs, but I didn't realize it was so long ago! Will definitely start the migration asap.

 

[itimpi]

8 minutes ago, Roscoe62 said:

This is what I suspected. Thanks for confirming. One more question on this...Once I've changed a data drive over from reiserfs to xfs, does the parity for that drive need to be re-calculated? I guess I'm asking whether there's anything in changing the file system that would necessitate a change in the parity data held?

No - when you format a drive to xfs then Unraid treats this as just another write operation and updates parity in realtime to reflect the new format.

[Roscoe62]

Sorry it's taken so long to return to this thread, but I have been busy! I changed my main share (the one I'm having an issue with) to use the highwater allocation method. I've also switched out my 1 parity drive to a new lager one - went from a 4TB drive to a 12 TB drive. UnRAID rebuilt the parity data on the new drive and found no issues...so far, so good.

I'm expecting the first of my new 12TB data drives to turn up later today. I'm going to use the "mirror method" as described here...

 

https://legacy.wiki.unraid.net/index.php/File_System_Conversion

 

I've read through this method several times now, and I'm satisfied I understand what each of the steps does. In my case it's a little easier in that each of my drives will be replaced with a new, larger drive and once the new drive (in xfs format) becomes the new drive, the old one will just be retired, as opposed to reformatting to xfs and starting on the next drive. Changing all my data drives over will likely take a good 6-8 months (money for drives is limited). I've still got 4 questions about the procedure I'm not sure of, and would appreciate it if someone could confirm the answers before I dive into this next step...

 

1. Once I've physically installed the new 12TB 'swap drive', and I've powered up the server, does unRAID automatically clear the drive? If so, at what point does that happen, or what extra step do I need to do to clear the drive?. If I've understood the procedure correctly, clearing the new disk should immediately precede formatting it to xfs, correct?

 

2. The main "tool" the procedure author uses is rsync. Is 'rsync' a part of the standard unRAID software package - so I can just run the command lines he states as part of the procedure, or do I need to add it in first? If so, where can I find it?

 

3.  Can I run the rsync command lines using a terminal session from within the webgui, or should they be run from a keyboard attached to the server, or does it not matter?

 

4. Once I've completed the mirroring procedure for this first new data drive, and I've switched the drive assignments, there should be no need to physically swap the drives over - doing the re-assignment steps should take care of this. However, what if I wanted the new larger disk to be in the same 'slot' as the older smaller drive? Is it simply a matter of completing the procedure as written, powering down the server, swapping the drive slot, powering the server up, and ensuring the new drive is still allocated correctly before starting the array again? I currently have all the drives mapped in a document, and I'd like to leave the drives in the same positions in between upgrading each of the drives. I could just update my document after each drive upgrade, but if keeping the drive numbering and slot positioning the same is easy enough to do...that would be my preference.

 

[itimpi]

Note that the link you give puts to legacy documentation - not the current online documentation accessible via the ‘Manual’ link at the bottom of the GUI or the DOCS link at the top of each forum page. The Unraid OS->Manual section in particular covers most features of the current Unraid release.

 

You may find this section useful as it describes the process in much simpler terms.     You may also find the Dynamix File Manager to be of use when moving the data around.    It did not exist when the link you give was created so not mentioned there.

[Roscoe62]

Hi itimpi,

 

Thanks for pointing that out. It hadn't occurred to me that there might be an updated (simpler?) way of doing this. I'm waiting for my final parity check to complete before starting this, so there's still time to change the process.

 

Having said that - there are still a number of steps that remain common and, reading through the Storage Management document you linked to - I couldn't find equivalent steps anywhere, so maybe they haven't changed?

 

Here's a high-level task list of the things I need to do...

 

1. Install new drive, assign it to 'Disk 9' (the next free disk number in my array)

2. Before starting array ensure disk9 is excluded in Global Share Settings

3. Start array - ensure unRAID clears Disk 9

4. When Disk 9 clear has completed, format Disk 9 to XFS format

5. Once Disk 9 formatting is complete, use Dynamix File Manager to copy all files from Disk 1 (my first 'source' candidate) to Disk 9 - ensuring, if possible, all file/folder permissions/settings have also been copied across

6. Stop the array

 

This next section is crucial to ensure UnRAID handles the new drive correctly, and I couldn't find an updated process for this - maybe because it's so specific.

 

7. In Tools/New Config ensure Preserve Current Assignments is set to ALL. Click Apply and Done

8. On Main page select dropdown for Disk 9 and unassign it

9. Click on the dropdown for Disk 1 and reassign it to disk 9

10. Go back to the dropdown for Disk 9 and assign it to disk 1

11. Click on drive name of disk 1 & change file system from ReiserFS to XFS, click on drive name of disk 9 & change file system from XFS to ReiserFS

 

Not 100% sure what to do in this step. For me, I'll be retiring disk 9 (the old disk 1 in Reiser format).

 

12. All array disks should be blue with warning "All data on the parity drive will be erased when the array is started". Make sure to click the check box "Parity is already valid". Then click the Start button to start the array.

 

Do the above steps seem reasonable to accomplish the task? Is anything missing or outdated?

Looking back at step 2 where Disk 9 was excluded from the share, does this still stand, or does it need to be fixed? I must admit I lost track among the reassignments.

 

Sorry! So many questions!

 

 

[JonathanM]

4 minutes ago, Roscoe62 said:

10. Go back to the dropdown for Disk 9 and assign it to disk 1

Why? If you are retiring the disk, why not just leave slot 9 blank, and NOT check "parity is already valid" so it calculates parity from the disks you wish to use?

 

The only other gotcha I see is if files are added or modified while the copy process is running you may not have the full up to date copy. You can either stop all the things that could write to the disk, or use rsync to verify the copy and list differences until the verify is perfect. It's not a bad idea to verify the copy anyway, but not the end of the world since you are keeping the old disk(s) intact as you go.

[JonathanM]

10 minutes ago, Roscoe62 said:

8. On Main page select dropdown for Disk 9 and unassign it

9. Click on the dropdown for Disk 1 and reassign it to disk 9

10. Go back to the dropdown for Disk 9 and assign it to disk 1

 

Maybe when you finalize your directions to yourself, use Slot instead of Disk when referring to the logical positions, and fill out the last 4 of the drive serial numbers so you keep it straight in your head and can cross check things. For example, select slot 1 and assign disk A54E or something like that.

 

Also, I suggest actually committing a document to print after verifying all your steps, then physically cross things off the list as you do them.

[Roscoe62]

 

 

47 minutes ago, JonathanM said:

Why? If you are retiring the disk, why not just leave slot 9 blank, and NOT check "parity is already valid" so it calculates parity from the disks you wish to use

 

Yes, I guess so, and it was worded that way in the original instructions as well.

 

49 minutes ago, JonathanM said:

The only other gotcha I see is if files are added or modified while the copy process is running you may not have the full up to date copy. You can either stop all the things that could write to the disk, or use rsync to verify the copy and list differences until the verify is perfect. It's not a bad idea to verify the copy anyway, but not the end of the world since you are keeping the old disk(s) intact as you go.

 

Got this one covered. During the copy process I will be the only one using the unRAID, there are no dockers or VM's running, and the ONE job I had running (to back up my main PC) I've deactivated for the time being. The system will be completely static.

 

49 minutes ago, JonathanM said:

 

Maybe when you finalize your directions to yourself, use Slot instead of Disk when referring to the logical positions, and fill out the last 4 of the drive serial numbers so you keep it straight in your head and can cross check things. For example, select slot 1 and assign disk A54E or something like that.

 

Yes, I think this is where some confusion has crept in - is the process step referring to the physical disk, or the disk slot? I do currently have a disk map document, which lists the physical disk data (make/model/serial number) for each disk and (currently) which slot it's in.

 

53 minutes ago, JonathanM said:

Also, I suggest actually committing a document to print after verifying all your steps, then physically cross things off the list as you do them.

 

That's a really good idea. Using that - along with the disk map - should help me to keep track of my progress.

 

Many thanks!

[Roscoe62]

OK,

 

I've followed your advice and documented ALL the steps to upgrade ONE disk (increase in disk capacity and changing to use XFS file system) and I've attached the resulting document to this post. I've tried to be as detailed as I could be - to reduce the risk of misunderstanding/misinterpretation. I've included, at the end, a section of steps to reposition the new drive into the position occupied by the old drive. Please bear in mind I've never done any of this before, so some of the things that seemed logical to me may NOT be correct. I've had to take a guess at a lot of these steps - hopefully I got them right.

 

I would be VERY grateful if someone who has done this kind of thing before, or who is confident they know the steps required to do them, could take a look at the document and see if the steps are correct or not. Like, any unRAID user, I'd hate to get part-way through this procedure only to find some steps are missing, or are incorrect. There are 48 steps in total covering 6 separate high level tasks, and accomplishing 4 goals (listed at the top). Don't worry though - it's only a 2 page document.

 

Again, thank you for your guidance, time, and advice!

UnRAID Server - Disk Replacement steps_v1.docx

[JorgeB]

It seems to me you may be overcomplicating this, also don't understand why you would rebuild parity with disk1 and 9 swapped to then remove old disk1 and rebuild parity again.

 

IMHO you could just format the new disk xfs with the UD plugin, still using UD copy everything from disk1 to the new disk, as a bonus it will be faster since the new disk won't be slowed by parity, once the copy is done do a new config with the new disk assigned as disk1 and without the old disk, then start array to resync parity. 

[Roscoe62]

13 hours ago, JorgeB said:

It seems to me you may be overcomplicating this, also don't understand why you would rebuild parity with disk1 and 9 swapped to then remove old disk1 and rebuild parity again.

 

IMHO you could just format the new disk xfs with the UD plugin, still using UD copy everything from disk1 to the new disk, as a bonus it will be faster since the new disk won't be slowed by parity, once the copy is done do a new config with the new disk assigned as disk1 and without the old disk, then start array to resync parity. 

 

You make it sound so simple when you put it that way....😉

 

The simple answer is - because I don't know HOW. Since building the unRAID server about 7-odd years ago (actually, REbuilding is more accurate, but I won't elaborate) I've been using it pretty much as a "plug and play' device. Do a parity check each month, when wanting more space, swap the disks for larger ones starting with the parity drive, if a disk goes faulty, shut down the server and replace the faulty disk with a new one. Using plug-in's is almost completely foreign to me, so it's all stuff I need to come up to speed with now that I'm dealing with issues outside of what I've known before.

 

So, to answer your question - I simply don't know HOW to use UD (I even had to look it up to find out that 'UD' refers to the 'Unassigned Devices' plug-in) to do the things you're suggesting. It's something I'll have to learn.

 

What you suggested in your last post sounds WAY easier than my process document, and it's also likely to contain a lot less unnecessary duplication. I'd much rather do it your way, but I'm going to have to figure out HOW to use UD to format the new drive to xfs. Then I'm going to have to figure out HOW to use UD to copy all the data from disk 1 to the new drive. Then I guess I can use the steps in my document to set up the new config (leaving the old disk out of the picture completely). And after parity has been rebuilt - based on the new config - will doing my final set of steps work to allow me to physically relocate the new disk 1 to the old disk 1 location without screwing anything up? Your summary seems to have excluded that.

 

In the meantime, I'm browsing through the UD support thread in an attempt to learn how to use UD to format the new drive, and then how to use UD to copy all the data from the old drive to the new one.

 

Yes, your high level process sounds way simpler than what I'd documented, and I'd much rather use it, than mine. But I'm paranoid about doing something wrong and losing data in the process. I'm sorry if this post sounds "snippy" - it's just the frustration of not being able to move forward because of my own lack of knowledge.

[Roscoe62]

Actually, I'm going to try this from a different angle. Please bear with me, and forgive me if I tread on toes....

 

I'm here posting in this forum because I need help. I know what I want to do (see the goals listed at the top of my procedure document)....actually, I'll list them here as well.

 

Goal 1 - Replace Disk 1 in my array with a larger, new disk

Goal 2 - Ensure the new disk is formatted as xfs (all data disks in my array are currently Reiserfs)

Goal 3 - Remove/retire old Disk 1 from the array

Goal 4 - Ensure the new Disk 1 is re-positioned to occupy the same physical location in the server as the old Disk 1

 

That's WHAT I want to do. What I DON'T currently know is HOW to accomplish these goals as simply as possible, with as little risk as possible.

 

Can someone please give me a list of steps showing me how to proceed? JorgeB gave me a pretty good high-level way of doing most of this in his post above, but I need this broken down a little further as I lack both knowledge and experience.

 

THIS is what I want. I'm probably guilty of not being direct enough with my questions. I'm trying to fix that now. If someone is kind enough to provide the steps I will be very grateful. If I need more detail, I'll ask.

[JorgeB]

I can suggest a slightly easier way with UD, it would not require using UD to format a disk or doing a new config, but it would involve a little more risk, since until disk1 is copied back top the array, it would exist only outside, and unprotected:

 

• Replace disk1 with the new disk and rebuild, you can install the new disk in the old disk1 physical location now
• Once the rebuild is done change the filesystem on disk1 to xfs and format it
• Connect old disk1 to the server
• Mount old disk1 with UD
• Copy data from old disk1 to new disk1
• Disconnect old disk1 and you're done

 

 

[Roscoe62]

Hi JorgeB,

 

Thank you so much for this. I can see this working, and it ensures all 4 of my goals are achieved. There's a few specifics I don't know how to do yet (using UD to mount a disk & copying the data from old disk 1 over to new disk 1 (now in xfs format)), but I'm hopeful this will become more obvious once I actually have an unassigned device attached.

 

I'm going to start this process today. Thanks again!

[Roscoe62]

I'm finally circling back around to this. I've now completed the Disk 1 upgrade and it's all gone well.

 

I also got to re-run the test that caused the initial issue that lead me to raise this thread in the first place....I transferred a (roughly) 25GB file from my PC to the server. This time the file transferred without any issues. It did slow down a little - which was expected given I'm not using a cache drive (yet), but the transfer speed never dropped below 50MB/s, which is WAY faster than it ever went before.

 

So, I'll mark this thread as solved. I believe - based on many of the discussions that have gone into this thread - that the cause of this issue is one of, or a combination of, these 3 things...

• The problem share used the 'most space available' allocation option. I'm now using the 'high-water' option
• My server was running a bit low on storage space. I've updated one of the drives the share was using (from 4TB to 12TB)
• My server still uses the 'reiserfs' file system. The drive I updated now uses 'xfs' instead

Thank you to all who contributed to this conversation, helped me to learn a lot, and finally resolve the issue. I'm grateful.