Jump to content

Allocation method seems to change


Recommended Posts

I'm trying to copy a large amount of data (7TB+), from a current unRaid server (6 x 2TB), to 2 new UnRaid servers (5 x 2TB disks), but then run into a problem where a single disk will fill up, even with high water set.

 

Setup is as follows for all three servers:

 

media (user share)

->Video

-->Movies

---> 01 Movies

----> 'Movies Directories' - one movie per directory

-----> 'Movie Files' - multiple files

---> 02 Movies

----> 'Movies Directories' - one movie per directory

-----> 'Movie Files' - multiple files

---> 03 movies

----> etc...

 

Split level is set to 4 so that (if I've got it right) the 'Movies Directories' can be split across disks but the files in each 'Movie Directory' will be on one disk.

 

Allocation method is set to 'high water'.

 

When I start the bulk copy (from the 1st unRaid Server with exactly the same setup), the disks are filled up as expected, 1st disk goes to 50%, then the 2nd to 4th also fills up to 50%, the 1st disk to 75%, etc. I monitor this via unmenu. If I check the split on the actual disks, it seems correct per disk; a batch of movies per disk.

 

Here's the problem:

 

If I start a new copy to load the next batch of movies (still from the unRaid server with the same directory structure), it suddenly starts to fill up the 1st disk till it hits 100% and then fail further copying with a 'disk full error' in the copying program.

 

This is still with the same split level (4) set and the same directory structure.

 

I've seen this on 3 different occasions and the only way I could get the system to copy more files was to change the split to 999, even with about 50GB left per disk.

 

I've searched the forum but did not see a similar post, but with all 3 mu unRaid servers doing this, it's either common or I'm being a noob. :)

 

Any idea what causing this? Is it something to do with the stopping and restarting file copies?

Link to comment

How much space on each disk and how big are the movie files. It sounds like the High Water level has gone through the 50gig level filling each disk until it has under 50gig free. This would mean the high water is at the 25gig level. So, if the movies are around 25gig in size then what you describe could easily happen.

 

Here's an example. You copy a 22gig movie to the server and it goes to say disk3 leaving 27gig free on disk3. Now, disk3 still has over 25gig free so unRAID will keep using it. However, if you copy a 30gig movie to the server and it goes to disk3 then the move will fail.

 

Try using the most free method when filling the server that full. Or else you have to manually place the files on the disks with space.

 

The min free space really doesn't help when trying to cram the drives right full.

 

Peter

 

Link to comment

How much space on each disk and how big are the movie files. It sounds like the High Water level has gone through the 50gig level filling each disk until it has under 50gig free. This would mean the high water is at the 25gig level. So, if the movies are around 25gig in size then what you describe could easily happen.

 

Here's an example. You copy a 22gig movie to the server and it goes to say disk3 leaving 27gig free on disk3. Now, disk3 still has over 25gig free so unRAID will keep using it. However, if you copy a 30gig movie to the server and it goes to disk3 then the move will fail.

 

Try using the most free method when filling the server that full. Or else you have to manually place the files on the disks with space.

 

The min free space really doesn't help when trying to cram the drives right full.

 

Peter

 

 

Thanks Peter,

 

Let's maybe seperate the two issues. The first was when there were little space left per disk (1st server), but the real problem I have is when trying to copy files to a new, clean server with 7TB space (2nd and 3rd server). I do think the symptoms are probably related but let me rather not make that assumption for now.

 

So, regarding servers 2 and 3, don't think it's a file size issue. There are still hundreds of GB left when I see the problem.

 

The disks are all 2TB and when I start the copy, it'll fill up 50% (1TB) of each and then go around and start to fill up the remaining 50% of each, i.e. up to 75% (1.5TB). But if the copy process stops there and I start a new copy (at the same directory level), it'll just fill up the remaining 500GB of disk 1 till it returns a disk full error.

 

It's as if the new copy is running at a split of somewhere between '0' and '3' instead of '4' causing unRaid to not split anymore at per-movie directory level but putting all the movie directories one one disk (i.e. not splitting any more).

 

Could the way I copy the files trigger the "non-splitting" in any way?

Link to comment

Are you doing the 500gig copy in one pass? If so, I bet the software is creating all the empty directories first and then filling them.

 

Peter

 

I definitely am copying more than 500GB in one batch, could be up to 2TB or so.

 

Can you explain your statement in more detail, please. I'm not following. :)

 

Even if I copy a large amount of data, unRaid should still follow the directory structure and designated split level?

 

I used a program called folderclone to replicate the directory structure from the 1st unRaid to the 2nd unRaid server, via my desktop where both shares are mapped as drives. No concrete reason, but I have a suspicion it could possibly be something to do with the copy program?

Link to comment

Some more feedback.

 

As described above, one copy would distribute the files according to the split level but a second copy would stick all the files on one disk. As if the split level is suddenly smaller.

 

I deleted all the files, that got dumped on one single disk during the second copy, from the share and then restarted the copy. Now it's filling up the disks correctly according to the split level and the high-water method.

 

I can see no difference between the two copy processes. Both were done at share level (i.e. not disk level) and they are exactly the same files in the same directory structure. But one filled up only a single disk up to 100% (while the other disks were between 50 and 75%) while the second copy is working fine.

 

Any ideas?

Link to comment

Did you change the program or any settings in the program used for the copying?

 

Did you look for empty directories before deleting everything?

 

Peter

 

Nope, I intentionally tried to ensure everything stayed the same. I basically just deleted the files that filled up drive-1 to 100% to ensure there's enough space for the high water method and since then it's been behaving itself. It's got all the drives up to 75%, just filled drive-1 to 88% and is now busy with drive-2, nearly at 88%. So it should switch to drive-3 soon.

 

It's now working a treat, which in itself is worrying as there is this uncertainty when it might act up again.

 

Unfortunately I never looked for empty directories before I deleted the batch.

Link to comment

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...