Video Editing, Force spread all files over all drives in a share

[nukedeath]

Hello!

I am currently editing my videos over Unraid and it is very slow reading the files. And that is because of the High Water share setting.

I tough about making a new share just for video editing, to keep high water on the other shares.

But I cannot see any setting that allows me to force it to spread all the files over all the drives.

 

That would massively increase the read performance for video editing.

Importing videos on a timeline in Resolve on files spread over 10 drives would max out a 10gbe connection.

But as of now, resolve spends ages scanning the files, and rescanning on timeline zoom changes.

Heck it hasn't even finished scanning the media library.

 

Add this and I'll send you guys a bottle.

 

Edited September 5, 2018 by nukedeath

[trurl]

You seem to have some misconceptions about unRAID. Allocation method can't have any effect on speed.

 

The simple fact is that unRAID is not RAID. There is no striping. Each file is written and read from a single disk, regardless of settings you can make. NO STRIPING.

 

You could work with unRAID files in a cache pool and take advantage of one of the btrfs raid configurations, perhaps with SSDs.

 

But even there you might to run into network bottlenecks.

 

The usual method is to edit videos and photos on the storage of the machine doing the editing, and use a NAS such as unRAID for archiving.

[nukedeath]

2 minutes ago, trurl said:

The simple fact is that unRAID is not RAID. There is no striping. Each file is written and read from a single disk, regardless of settings you can make. NO STRIPING.

Yes, that it what I want.

I want each separate file, to be placed on separate drives.

So when my editor starts scanning, I don't want it to fetch all the files from one drive, I want it to fetch the separate video files from as many drives as possible.

When editing, one larger file would be as slow as one drive yes. That I can see, and survive.

But there are so many files that are scanned when going through a timeline, it would be a huge performance boost.

[trurl]

unRAID decides where to put a new file based on Split Level, Allocation Method, and Minimum Free. How do you have your Split Level set?

 

Of course it has no way to know which files will be requested for reading, so how is it to decide where to put a new file?

 

If all your disks are the same size then Allocation Method = Most Free might make it switch to a different disk more often when writing a new file since presumably the disk with the most free space will change somewhat frequently if they are filling up at the same rate.

[nukedeath]

3 minutes ago, trurl said:

unRAID decides where to put a new file based on Split Level, Allocation Method, and Minimum Free. How do you have your Split Level set?

 

Of course has no way to know which files will be requested for reading, so how is it to decide where to put a new file?

 

If all your disks are the same size then Allocation Method = Most Free might make it switch to a different disk more often when writing a new file since presumably the disk with the most free space will change somewhat frequently if they are filling up at the same rate.

Most Free could work If I didn't already use the server for other things.

The feature i'm asking for is a new Allocation method that spreads the files over all the drives to "possibly" speed up video editing.

 

 

 

[1812]

maybe try the unbalance plugin and scatter.... but you'd have to do it for all files which would be tedious, if it worked as you wanted it to.

 

I edit 4k on off my server from a single disk via 10be and only run into issues with multi-stream 60fps footage...

 

and because of this, I have asked for a while now for multiple cache pools instead of what I have one in the past on bigger project and have the OS create a raid for a couple disks passed through to it.

 

how many MB/s read are you needing to achieve?

 

 

[trurl]

This thread has mostly been about read speeds, but I assume when you say edit you intend to also write back to the server. As you should be aware, writes to the parity array are going to be even slower.

[trurl]

My wife does photoshop on this PC I am currently typing on. I built it for her and it has a 1TB SSD for her working files. We just use unRAID for archiving.

[nukedeath]

8 hours ago, 1812 said:

maybe try the unbalance plugin and scatter.... but you'd have to do it for all files which would be tedious, if it worked as you wanted it to.

 

I edit 4k on off my server from a single disk via 10be and only run into issues with multi-stream 60fps footage...

 

and because of this, I have asked for a while now for multiple cache pools instead of what I have one in the past on bigger project and have the OS create a raid for a couple disks passed through to it.

 

how many MB/s read are you needing to achieve?

 

 

Ey thanks man, I will try the Unbalance plugin.

With that I can test If I really would get a better editing experience by spreading all the files.

I do not use a cache pool for editing as of now.

Right now my problem with editing in Resolve seems to be related to it wanting to scan ALL THAT IT CAN at the SAME TIME, and when all the files is on one disk it boggs down a lot.

 

In current project I have only the camera h264 files at 100 mbit so thats not much reads.

But the other project I only used DNXHR HQ transcodes to speed up editing, and they ranged from 83 MB/s to 103 MB/s.

 

Let's say I used a cache pool for editing.

What if I wanted to "Promote" a 2 year old project from the archive to the cache pool for editing again

If that possible?

[testdasi]

For your use case, I think a RAID-0 SSD cache pool will work way better (with regular back up into the array to mitigate RAID-0 shortcomings).

 

Even if you spread your files out, there's no guarantee of better speed if the software "scan ALL THAT IT CAN at the SAME TIME". When it tries to scan more files than you have disks then there will be 1 disk that has more than 1 file being scanned. And speed penalty for simultaneous access on a HDD is very bad.

[trurl]

3 hours ago, nukedeath said:

What if I wanted to "Promote" a 2 year old project from the archive to the cache pool for editing again

If that possible?

There is a User Share setting for Use cache.

 

If it is set to Yes, files/folders from the share will be moved from cache to array when mover runs.

If it is set to Prefer, files/folders from the share will be moved from array to cache when mover runs.

 

So, if the project is in its own user share, you could set it to cache-prefer and run mover.

 

This link to the FAQ explains this setting in a lot more detail. There are ways to have shares on cache that don't get moved, shares on cache that get moved to the array, shares on the array that get moved to cache, and even shares with some data on cache and some on array that stay where they are until you change that setting.

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?page=2#comment-537383

 

There are many other topics about cache pools in the FAQ.

 

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?page=2&tab=comments#comment-554741

 

[nukedeath]

Hey guys I figured it out.

I copied over my old DNXHR encoded project and it ran without any problems so I had to do some digging on Resolve.

The problem it seems is that the free version of Davinci Resolve is a heroin addict on withdrawal when it comes to video files that are h264.

I spent 4 hours transcoding all my files with Davinci Resolve's Media Manager to DNXHR HQ and saw a staggering 10 times increase in disk usage.

Now everything is works fine and shows bandwidth usage of around 800Mbit and spikes up to 1.2Gbit when working over the network.

 

So I don't need a feature to spread the files anymore, I just need to buy larger hard drives.

Thank you everyone for chiming in on solutions.