Drive rebalancing

[garycase]

r.e. pre-clearing your old parity drive => If you want to pre-clear it as a "sanity check" that the drive is okay; that's fine.    But pre-clearing it will make NO difference in the time it takes to add it to the array, since you're not adding it as an additional drive, but are going to replace a drive ... which will then result in a rebuild of that drive.    Pre-clearing only saves time if you're ADDING a drive to an array ... not if it's going to be a parity drive or a replacement for an existing data drive.

 

[JustinChase]

so, some progress, but some frustration too.

 

I finished all the preclears and have the old 3TB parity installed in place of a 1TB drive, the drive has been rebuild and parity has been checked.  All is well there.  But, back to the original point of this post, I'm trying to get some of the data off of the full drives to put on the 2TB of free space I have now.  However, when I try to copy the files with windows from disk1 to disk5, it says I need permission from linux/nobody to move the file.  Why would it tell me that?  I ran the newperms script, but it didn't help, I still get this message.

 

What do I have to do to get files moved?  I tried using midnight commander, but I can only move one directory at a time, and have to wait for it to finish, which will take me several more days to finish rebalancing that way.  i was hoping I could just tell windows to move several folders at once, and let it run overnight for a few nights, but it's not working.

 

Arrrggghhh!

[dgaschk]

Use teracopy. It will queue the copy jobs. Copying in parallel will be slower than copying serially.

[JustinChase]

thanks, I'll try that.  On the other hand, someone mentioned that having drives completely full doesn't affect playback for them, so maybe I'll just leave the other drives full.  i use a cache disk anyway, so that should mitigate any writing to the array slowness, since it does it at 4am with the mover script.

 

With that said, I have been getting long pauses while playing back videos, so there's something going on here

 

too many moving parts (including me moving to another country in a week!)

 

thanks again for all the help everyone.

[pengrus]

For the permissions from nobody, you probably only need to

chmod -R 777 /directoriesyouwanttomove/

 

, with possibly

 

chown -R windowsuser:users /directoriesyouwanttoown

 

to make your windows box own and have the permission to modify those files.  (#$%ing Windows...  )

 

With MC, use ctrl-t to select multiple directories and F6 to move them over.  I have often waited until my array was basically full to add new drives, leaving me with a rather unbalanced system such as yours.  I just start MC in a screen session and eyeball it. 

 

You might be able to use multiple screen sessions to do it all at once, but that would probably not do much for your read/write speed, maybe leading to a frozen box.  (done this too)

 

Good luck!

[garycase]

someone mentioned that having drives completely full doesn't affect playback for them, so maybe I'll just leave the other drives full.

 

"someone" was me 

... and there's NOTHING wrong with full disks => over half of my disks are over 99.9% full, and it is NOT an issue at all.  I can stream 5 simultaneous streams from the same full disk with no issues.

 

WRITES are a very different story => you do NOT want to be writing to really full disks, as that's very slow.

 

 

I have been getting long pauses while playing back videos, so there's something going on here ...

 

This could be from a variety of causes.  Are you sure the entire video is on the same disk?    [Or is there some other array access going on while you're watching the movie?]    Any drive spin-up activity can cause pauses.    There are ways to resolve this (e.g. spin-up groups), but first you need to isolate just what's happening.  It is NOT, however, due to a "full" disk 

[drawz]

Seems doubtful, but could be a heavily fragmented disk, which might be more likely if full (and not filled sequentially).

[garycase]

Very unlikely it's due to fragmentation => modern disks are far faster than a video stream ... even if a very high percentage of the file's nodes required a seek => especially with the drive's buffer and AHCI "elevator seeking".

 

I suppose anything's possible, but I'd put that very low on the list of possibilities.

 

It IS possible if a file was exceptionally fragmented that this could be the case, but with ReiserFS's dynamic tree structure (which doesn't use traditional fixed-size allocation blocks), the likelihood of this is VERY small.

 

 

 

[JustinChase]

Most/all my videos are mkv files, so I don't think fragmentation could be the cause.  i suspect that either unpacking downloaded files, or other access causing disks to spin up is to blame.  one of these days, i'll try to figure out how to find the cause for sure, then work towards a long-term fix.  i wish there was a way to just tell unRAID (or my router) that media playback takes precedence over all else, and to make that an uninterrupted process, but I don't suspect it's quite possible.  oh well, not everything can be perfect.

 

I'm preclearing the last 2TB drive from an older external case, and will replace one more 1TB drive hopefully tomorrow evening, then rebuild that disk to the new drive, and I think all my maintenance will be done (just) in time for my to move Wednesday morning.

 

I'll report back from Mexico once we're settled in and I have the server connected again

[fysmd]

WRITES are a very different story => you do NOT want to be writing to really full disks, as that's very slow.

 

What makes writes to full disks slow

Also, at what point does this happen?

[garycase]

I don't know the point at which it becomes really noticeable ... I'd estimate around 95% or so for TB size drives.    Nor do I know exactly why, but I suspect it's due to "node pruning" to free up as much contiguous space as possible for the next file.    Think of this as putting food in a big pantry.  You probably put things in very quickly until it gets nearly full;  then you do some rearranging before adding the last few things, so you can maximize the space.    If you don't want to do things slowly for the last few bits, you can just leave some empty space   

 

... but since most of my drives are media that, once stored, is essentially static, I manually write for the last dozen or so DVDs to fill them up.    I consider a drive "full" when there's less than one DVD's worth of space left.