DVR VM Setup advice?

[stanger89]

So unRAID 6 has me pretty excited, I'm planning to upgrade the hardware in my current unRAID box and combine it with my SageTV system by putting SageTV in a VM.  To that end I've successfully got 6.0b15 running on the new hardware, albeit with minimal hardware while I'm testing.

 

With the help of the info on the forum I've got a Windows 7 VM up and running, and all relatively painlessly.  But before I go too much farther and make any silly mistakes, I'm looking for some advice on how best to setup my hardware/unRAID/VM for my recording.

 

I know the pool isn't the fastest thing in the world, and I don't care if/need my recordings to be parity protected.  Also I don't care to have the whole array spun up whenever I'm recording something.  So what's the best way to setup my drives for recording?  Right now I've got 2TB of recordings so I'm not sure it's practical or smart to put all that in the VM on a cache drive.  Any suggestions would be appreciated.

 

My setup is (or will be) as follows:

unRAID 6.0 beta 15

i7 3770s

16GB RAM

Asus P8Q77-M/CSM

SUPERMICRO AOC-SASLP-MV8

 

[CHBMB]

I'm not familiar with SageTV and I'm not aware of anyone here using it.  TVHeadEnd is the main PVR software run here, normally with a xbmc/kodi frontend.

 

What DVB hardware are you using?  Unless it's some sort of HDHomeRun type device then you will need to passthrough your tuner card. 

 

I was using ArgusTV in a Windows 7 VM and just recording onto an Unraid share that was cache drive only.  This worked well enough and didn't require my array to be spun up.

 

Post a few more details and I for one will be interested in how things work out.

[dmacias]

I use a mythtv vm. I don't save any of my recordings. I mount a drive that is not part of the array in the vm for recording storage. Don't forget you have to think about live tv buffer also. I think for performance not just the vm but unRAID, it's best to use a separate drive for recording and live tv.  When I used to run WMC  or Mythbuntu, I always used at least a 2 drive system. One for operating system and one for live tv and recordings.

[CHBMB]

Like dmacias says, I agree with the two drive principle and my VM is on a non array drive but I've always used the cache drive for buffering live tv and recordings, but I don't really archive any recordings. 

 

Do the 2TB of recordings need to be in the same folder location as your new recording?

[johnodon]

Same here...I'm running MythBuntu in a VM and it has been rock solid!

 

John

[JM2005]

Me too I'm running MythBuntu in a VM and it has been rock solid as well.  Thanks to dmacias for all his help! I do like to keep the recordings but need to figure out way to auto remove the commercials and recompress to smaller file size.

 

JM

 

[stanger89]

Hey guys, thanks for the responses, let me try to get you the info.  SageTV is a multiplatform DVR/PVR system, it was purchased by Google a few years ago and is the basis for their Google Fiber DVR, but it is being Open Sourced in the near future (a SageTV Docker is an exciting idea, but that's a different discussion).  One of the best things about SageTV, and the primary reason I'm not moving, are the extenders, hardware boxes that run the full SageTV experience (think WMC Extenders but not sucky), but I digress, I don't intend this to be a debate about DVRs.

 

Anyway, all my tuners are HDHomeRuns and I've already got them setup in a Windows 7 VM.  I've also got SageTV setup in the VM, none of that is really a question/problem for me.  I do have a USB tuner, but it's analog and actually hasn't been connected for a while so while I believe it should be easy to pass that through if I want, it's not really an issue for me.

 

dmacias, your thinking mirrors my own, but you have more unRAID experience with this which is what I'm looking for.  Right now my SageTV machine has Windows and Sage on an SSD, with two 1TB Caviar Blacks for my recordings.  I plan to maintain something like this when I migrate (recording drives separate from OS drive), but while I've run unRAID for a while, I've not dug deep into it.  Do you use something like SNAP for your recording drives?

 

I too don't archive any recordings, anything I want that bad I buy on Blu-ray (better quality, no recording glitches, no network banners, etc).  One difference with SageTV is there's no liveTV "buffer" per se, everything is a recording, LiveTV is just a recording of something that's currently airing.  So there's no such thing as a special buffer, LiveTV and recordings all go to/share the same drives.

 

I suppose I could use a regular HDD for Cache.  SSDs are too expensive/GB to record to.

[kegler]

Looks to me we are both heading toward the same goal/end point.  I'm just coming from a slightly different starting point.  Like you I have been using SageTV for a long time (8 years) and have endured this Google induced situation.  Going almost four years without an update to SageTV has been trying.  If I didn't like the system and the extenders so much I would have given up a long time ago.  As it is, there exists a big unknown as to where this open sourcing will lead to.  That said there is as much talent over there in the SageTV world as there is around here.  So the potential for good things to happen is very good.  I will be watching developments closely and give it sufficient time to bear fruit.

 

Currently I have the SageTV server running in an ESXi Win 8.1 vm.  My recording disk is configured as an independent disk on a hd vmdk.  The  windows vm resides on a ssd vmdk.  I also use HDHomeRun tuners exclusively, both for OTA and cable.  Unlike you I do a fair amount of archiving to Sage import directories residing on unRAID shares.  unRAID is also virtualized on ESXi.  I access the windows vm using RDP.  This works well enough that I am able to edit videos on it using VideoReDo  (I do not use comskip).  I am not sure this (VideoReDo editing) will be possible over RDP under KVM.  It may be neccessay to go to GPU passthrough.  ESXi is a type 1 hypervisor where KVM is not.

 

I am not married to ESXi.  I would prefer to move to KVM as it would preclude having to possess knowledge of one less world (ESXi).  In addition VMware has never made free users feel very welcome. 

 

All of that aside, I currently have a very stable system.  Granted it is still on ESXi and unRAID 5.0.  I am in no need or hurry to change this.  I will wait until I see where I am going, will be both stable and mature enough to fulfill my needs.  Fortunately I invested in identical hardware for my testbed, so any new configuration can be thoroughly tested.  I am open to help test out new configurations supporting SageTV on it, if needed.  Just give the word.

 

 

[dmacias]

Hey guys, thanks for the responses, let me try to get you the info.  SageTV is a multiplatform DVR/PVR system, it was purchased by Google a few years ago and is the basis for their Google Fiber DVR, but it is being Open Sourced in the near future (a SageTV Docker is an exciting idea, but that's a different discussion).  One of the best things about SageTV, and the primary reason I'm not moving, are the extenders, hardware boxes that run the full SageTV experience (think WMC Extenders but not sucky), but I digress, I don't intend this to be a debate about DVRs.

 

Anyway, all my tuners are HDHomeRuns and I've already got them setup in a Windows 7 VM.  I've also got SageTV setup in the VM, none of that is really a question/problem for me.  I do have a USB tuner, but it's analog and actually hasn't been connected for a while so while I believe it should be easy to pass that through if I want, it's not really an issue for me.

 

dmacias, your thinking mirrors my own, but you have more unRAID experience with this which is what I'm looking for.  Right now my SageTV machine has Windows and Sage on an SSD, with two 1TB Caviar Blacks for my recordings.  I plan to maintain something like this when I migrate (recording drives separate from OS drive), but while I've run unRAID for a while, I've not dug deep into it.  Do you use something like SNAP for your recording drives?

 

I too don't archive any recordings, anything I want that bad I buy on Blu-ray (better quality, no recording glitches, no network banners, etc).  One difference with SageTV is there's no liveTV "buffer" per se, everything is a recording, LiveTV is just a recording of something that's currently airing.  So there's no such thing as a special buffer, LiveTV and recordings all go to/share the same drives.

 

I suppose I could use a regular HDD for Cache.  SSDs are too expensive/GB to record to.

Mythtv works the similarly.  Live TV is a recording. So unlike WMC if you had paused a show for 15min and the dog stepped on the remote and changed channels you won't throw that remote at the dog. Not tryin to debate which is better. Setting it up should be easier since you have network tuners. I use a 750GB 7200RPM laptop drive for recordings. It's not mounted with snap or anything.  I just mount the whole disk by id in the vm.  You can create a partition and mount that but I just mount the whole disk and then partition inside the vm.

I have a 10GB raw image for the OS and the 750GB for recordings

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/domains/myth/myth.raw'/>
      <backingStore/>
      <target dev='hda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/disk/by-id/ata-WDC_WD7500BPKX-00HPJT0_WD-WXL1E84AK1MK'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
    </disk>

 

I don't believe the new vm manager supports adding a disk but you can just use the manger's xml editor and add the drive manually.

 

[stanger89]

Looks to me we are both heading toward the same goal/end point.  I'm just coming from a slightly different starting point.  Like you I have been using SageTV for a long time (8 years) and have endured this Google induced situation.  Going almost four years without an update to SageTV has been trying.  If I didn't like the system and the extenders so much I would have given up a long time ago.  As it is, there exists a big unknown as to where this open sourcing will lead to.  That said there is as much talent over there in the SageTV world as there is around here.  So the potential for good things to happen is very good.  I will be watching developments closely and give it sufficient time to bear fruit.

 

Not to go too OT, but I understand where you're coming from, it's been "tough" psychologically, not having updates.  I like to see updates/improvements.  But at the same time, my SageTV system has been essentially flawless all that time, without any updates, so I was very glad to see it going open source, if for no other reason than it should help to maintain/re-grow interest.  There's not really anything "important" I want to see addressed in Sage....

 

All of that aside, I currently have a very stable system.  Granted it is still on ESXi and unRAID 5.0.  I am in no need or hurry to change this.  I will wait until I see where I am going, will be both stable and mature enough to fulfill my needs.  Fortunately I invested in identical hardware for my testbed, so any new configuration can be thoroughly tested.  I am open to help test out new configurations supporting SageTV on it, if needed.  Just give the word.

 

I almost did what you did, put unRAID and Sage/Windows onto ESXi.  I'd even set it up as a test at one point, I ran into some trouble with tuners though, it might have been my HD PVRs at the time, oh, no, it was my R5000, for some reason, despite being USB it just wouldn't work, the HD PVR worked fine.  Go figure.

[stanger89]

Setting it up should be easier since you have network tuners. I use a 750GB 7200RPM laptop drive for recordings. It's not mounted with snap or anything.  I just mount the whole disk by id in the vm.  You can create a partition and mount that but I just mount the whole disk and then partition inside the vm.

I have a 10GB raw image for the OS and the 750GB for recordings

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/cache/domains/myth/myth.raw'/>
      <backingStore/>
      <target dev='hda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/disk/by-id/ata-WDC_WD7500BPKX-00HPJT0_WD-WXL1E84AK1MK'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
    </disk>

 

I don't believe the new vm manager supports adding a disk but you can just use the manger's xml editor and add the drive manually.

 

I'll have to look into that, in theory with that I could just mount my existing recording drives.  Although I'm sort of short on space (ports) at the moment so I think I'd rather consolidate my recordings to a single drive.

 

On a side note, I was trying (in this experimentation phase) to mount drives using my USB 3.0 HDD dock.  I see it listed in the USB devices, and selected it, but I see nothing show up in my VM.

 

Though I've run into a strange issue.  I had my "array" running without parity, I only configured one drive in the array, now after rebooting the unRAID machine (to see if that helped with the USB passthrough showing up) unRAID won't let me start the array, it says invalid configuration.  I'm sort of at a loss for either what changed, or how I managed to start the array the first time...  Nevermind, apparently it was a number of devices thing.  This is the Trial version since I haven't moved my key over and I had too many devices connected.

[stanger89]

I never figured out why the USB passthrough doesn't work (maybe unRAID "claims" it first?) but I did get a drive passed through "by-id", formatting now.  For some reason I had to resave/reselect the disk image though before the VM would start.

 

But, the adventure continues.

[dmacias]

 

 

I never figured out why the USB passthrough doesn't work (maybe unRAID "claims" it first?) but I did get a drive passed through "by-id", formatting now.  For some reason I had to resave/reselect the disk image though before the VM would start.

 

But, the adventure continues.

 

You may be able to pass the whole usb3 controller through.

[kegler]

 

 

I never figured out why the USB passthrough doesn't work (maybe unRAID "claims" it first?) but I did get a drive passed through "by-id", formatting now.  For some reason I had to resave/reselect the disk image though before the VM would start.

 

But, the adventure continues.

 

You may be able to pass the whole usb3 controller through.

 

That is not fully supported yet.  You need to be careful that you don't passthru the controller containing the unRAID usb device.  See the below post from jonp.

 

http://lime-technology.com/forum/index.php?topic=39322.msg368127#msg368127

 

[stanger89]

Yeah, I got the drive passed through successfully, still not sure why I had to resave the xml in the "wizard", as opposed to the XML editor.  And long term, any drives will be on SATA/SAS not USB, but this is just an easy way for me to mess around while I get comfortable and test the stability of everything.  Up side of this way is I should be able to move this disk to an SATA port and I shouldn't have to change anything in the config, if I understand how this "by-id" identification works.

 

I may have to start saving for a nice rackmount case and one of those 16-port LSI controllers, or one of the M1015's with an expander.  Or maybe I should just get another card and put it in the PCIe x1 slot on the board.  Maybe just for the DVR drive(s).

 

-edit

 

Looks like I could get a AOC-SAS2LP-MV8 and use that in the x16 slot, that might help performance a bit.  And then I could try my SASLP in the x1 slot.  That would be pretty cheap, get me more ports and save me from the comparatively expensive option of getting a 16 port card or replacing drives with larger ones....

[MRA45]

Hi everyone,

 

I am in the process of setting up a mythbuntu VM backend. I have passed through a 750gb non array drive to use for recordings, but I am not sure how to have the system mount the drive on startup. Any help will be appreciated thanks.

[kegler]

Made a bit of progress last evening.  I had previously set up a Win7 Prof 64bit vm on my testbed system.  But RDP performance (also noted elsewhere) has been poor.  RDP is very important for my use case.

 

Last evening I found where to download a Win8.1 Prof 64bit ISO and patch it so it doesn't require a product key before installation (run in trial mode).  RDP performance on it is excellent.  It is every bit as good as what I am used to seeing on EXSi.  Next, I will install the SageTV server on it for further testing.  This SHOULD work without  issue.  My big concern was RDP performance and this is no longer an issue for me,  as I was not going to be using Win7 as a host anyway.

[dmacias]

Hi everyone,

 

I am in the process of setting up a mythbuntu VM backend. I have passed through a 750gb non array drive to use for recordings, but I am not sure how to have the system mount the drive on startup. Any help will be appreciated thanks.

My drive is formatted to jfs. I mount it to the same location that is the default for mythtv so I don't have to change any directories in mythtv backend setup and can easily replace. This is what I have in /etc/fstab

/dev/vdb1 /var/lib/mythtv        jfs    rw,noatime,nodiratime,errors=remount-ro 0    1

[MRA45]

Thanks for the help dmacias. I tried your suggestion as well as various other mount points, but was not able to get the drive to work. It constantly fails to mount on startup. Currently the drive is formatted as xfs. Perhaps that is the problem? What is the preferred drive format? Are there any system dependencies I should have installed that I might be missing? Thanks again for the assistance.

 

[dmacias]

Thanks for the help dmacias. I tried your suggestion as well as various other mount points, but was not able to get the drive to work. It constantly fails to mount on startup. Currently the drive is formatted as xfs. Perhaps that is the problem? What is the preferred drive format? Are there any system dependencies I should have installed that I might be missing? Thanks again for the assistance.

 

You may need to add something for xfs. I had to add jfstools to format and mount it. I think xfs is fine. It was between xfs and jfs and I don't remember why I pick jfs. You just want a file system that's better with large sized files. I think jfs deletes faster or something.

 

Also you should make sure it shows up in the vm first before editing fstab.  It should show up on the desktop. So if it's not, check the /dev name and the depends.