ironicbadger Posted June 22, 2014 Share Posted June 22, 2014 As part of my series of KVM compatible VM images I present to you my Arch Linux appliance. The basic premise is that you boot unRAID into 'KVM (NOT XEN) mode' as I show in my video series below, then download my VM image and run it! Once you've run the image you're free to customise the installation to your hearts content with any application you can find for Arch (that's almost all of them!). Bear in mind that these videos tell you to boot in the Xen mode, ignore this and just boot into 'standard' unRAID mode which has KVM enabled by default. What is Arch Linux and why do I need it in a VM? Arch Linux (or Arch /??rt?/)[3] is a Linux-based operating system for i686 and x86-64 computers.[4] It is composed predominantly of free and open source software' date='[5'] and supports community involvement.[6] The design approach of the development team focuses on elegance, code correctness, minimalism, and simplicity, and expects the user to be willing to make some effort to understand the system's operation.[7] A package manager written specifically for Arch Linux, pacman, is used to install, remove and update software packages. Arch Linux uses a rolling release model, such that a regular system update is all that is needed to obtain the latest Arch software; the installation images released by the Arch team are simply up-to-date snapshots of the main system components. I chose Arch Linux to base this particular VM on because of the way in handles software / packages in addition to it's tiny footprint on disk and in RAM. Arch Linux, just like every other Linux distro, makes use of things called 'repositories' that store packages ready for you to download. The main Arch repo's have a huge array of packages already, but they notably don't include 'those kind of apps' that we use heavily on a media server - this is where the Arch User Repository (AUR) comes in. Members of the Arch community are free to create packages for their favourite apps and Arch provides an easy way to download and compile them on a full install. To make your life easier what I've done is gone ahead and compiled the majority of media apps ahead of time and put them into my own 'unraid6repo' which is preconfigured into your VM, making installing non standard apps a breeze but more on that later. Ok I get it, Arch is awesome. How do I get going?! See the bottom of this post for download links, and post #2 for a series of YouTube videos that take you through the process from start to finish if you prefer that method over a text guide. Youtube Playlist for the setup guide - http://www.youtube.com/playlist?list=PLgUdkYSObfxotSl4c4KJlQKNyQbPAduLl ---------------- How to download and start this appliance 1. Login via SSH or Telnet to your unRAID box using Putty on Windows or Terminal on a Mac. 2. Download the .zip file to the location you want to run the VM from and unzip it. cd /mnt/cache <------ *** wget http:// <insert download link from bottom of this post> unzip ArchVM_KVM_v*.zip <------ this will take approx 5 mins *** this is up to you, you can place the .zip wherever you like. If you change this location, you will need to edit the .cfg file accordingly. If you want to do this use mcedit after you've unzipped the download. nano ArchVM/arch.cfg 3. Run your VM. virsh create ArchVM_KVM/arch.xml Once you've booted the VM and checked everything is setup correctly in the XML file you should 'define' your machine using virsh which effectively 'registers' it. virsh define ArchVM_KVM/arch.xml virsh list virsh start ArchVM 4. Login to your VM via either SSH or Console (working on this) and find the IP address of your Arch Linux VM. User / Passwords root / unraid6 **xl console ArchVM** <-- working on console access, can someone help me fix this OR ssh root@archvm Once you're logged in, to find the IP address of the VM issue ip addr 5. Once you're in you should be presented with the screenshot from above with the pretty Ascii art in the shape of the Arch logo. I compiled this image probably several days, if not weeks before you downloaded it so first we need to update the repositories as they will be stale. If there are any package upgrades, do them. pacman -Syu <-- this should be done as often as possible (weekly minimum ideally) 6. We are now ready to start installing packages! w00t! Arch uses something called 'pacman' as it's package manager and it couldn't be simpler. Here are the basics. All commands are CASE SENSITIVE!!! pacman -Sy <insert app name> <--- eg pacman -S sickbeard-git pacman -Ss <insert app name> <--- searches the repositories the VM can access for a package pacman -R <insert app name> <--- removes a package 7. Access unRAID data from the VM This is where KVM really shines, it's support of block-based protocols. VirtFS 9p is blazing fast and is not a network based protocol. There is an example line in the fstab of the VM ready for you to edit as you see fit, but it's a two step process with this VM. 7a. Edit the XML file to point your unraid shares (here is the section below as an example) <!-- Virtfs 9p shares --> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/Media/'/> <!-- This is the path of your unRAID share you want to access --> <target dir='Media'/> <!-- This will be referenced in your fstab --> </filesystem> 7b. Now edit the /etc/fstab file 'nano /etc/fstab' from within the VM to include this line. Note the 'Media' at the start which is a reference to the XML file where you 'passed through' the unRAID share. This isn't a network based protocol and gives the VM almost direct access to the drives so it is BLAZING fast. Media /mnt/Media 9p auto,x-systemd.automount,trans=virtio,version=9p2000.L,rw 0 0 There is still about a 30-60 second delay when the VM boots up whilst the unRAID gives it an IP. There is nothing I can do about this, it is not specific to this VM. Sorry! It annoys me too! Samba Add this entry (and change the IP of course) to /etc/fstab. Edit fstab with ' nano /etc/fstab '. nano /etc/fstab -------------- //10.0.0.2/Media /mnt/unraid cifs auto,x-systemd.automount,guest,noperm,noserverino,uid=nobody,gid=users 0 0 --------------------- How to create a data .img to act as a datastore for Plex and others... ** These steps aren't modified from my Xen VM thread but are almost identical so I'm sure you can figure it out ** The base .img for this install is only 50G, for some of you this won't be enough! Here are the steps to add more storage to your VM as some of you have ridiculous Plex libraries!! It is also useful as scratch space for Sabnzbd and/or Tranmission incomplete downloads too... 1. Log in to unRAID (not your VM) via SSH, telnet or the console 2. Power off the ArchVM if it isn't already xl shutdown ArchVM xl top <--- wait for ArchVM to disapear 3. cd to the directory of your VM for example: cd /mnt/user/cache_only/ArchVM 4. Create an .img file of the required size truncate -s 250G data.img <--- adjust the size as needed, as for as I know there's no limit. 5. Edit your arch.cfg with mcedit mcedit arch.cfg ----------- disk = [ 'file:/mnt/user/cache_only/ArchVM/arch.img,xvda,w', 'file:/mnt/user/cache_only/ArchVM/data.img,xvdb,w' <--- add this line ] 6. Boot up your ArchVM xl create arch.cfg -c 7. Check the data.img file is seen with fdisk -l ------------- Disk /dev/xvdb: 250 GiB 8. Create a partition structure and filesystem on the .img fdisk /dev/xvdb command: o command: n partition type: default p partition num: default 1 First sector: default Last sector: default command: w mkfs.ext4 /dev/xvdb1 9. Mount the new partition in /etc/fstab mkdir /mnt/data nano /etc/fstab -------------- /dev/xvdb1 /mnt/data ext4 defaults 0 1 Ctrl + X to save a quit the fstab file. Reboot. --------------------- The IronicBadger repository ! The reason for using Arch is two fold. First, it has an incredibly tiny footprint for a full distro and uses almost no RAM. Second, it's what I use and I host a repository full of my favourite apps which are not in the default repositories. The VM already has it added so needs no user interaction. I take requests on adding apps to my repository, but you must first do just a little homework. You must find out first whether the package already exists in the mainline repositories and if it doesn't you must check to see if it exists on the Arch User Repository. I will only add packages from the AUR to my repo! Arch Linux Main Repos - https://www.archlinux.org/packages/ AUR - https://aur.archlinux.org/packages/ For more information visit http://unraidrepo.ktz.me/. Download Links Personal hosting - http://repo.ktz.me/archVM/KVM/ArchKVM_v1.zip If you find this VM useful please consider making a donation. Link to comment
ironicbadger Posted June 22, 2014 Author Share Posted June 22, 2014 Reserved for future use. *Probably some more videos* Link to comment
MTA99 Posted June 22, 2014 Share Posted June 22, 2014 Thanks Ironic! Your work is very much appreciated. I was really excited to try Docker but I don't think its ready for unRAID yet (to be discussed elsewhere). For now your Arch VMs are so fast, lightweight and easy to use that I'm going to stick with Arch for my application needs. This new VirtFS feature sounds great. Actually looking forward to migrating my XEN VMs to KVM #geek Link to comment
jumperalex Posted June 23, 2014 Share Posted June 23, 2014 50GB!!!! Any chance of something more like what you had for Xen? Or a tad smaller? I'm only using 1.5GB of the 14GB image you had there. I'd liek to keep this on my SSD but it is only a 55GB (effective) drive Link to comment
peter_sm Posted June 23, 2014 Share Posted June 23, 2014 50GB!!!! Any chance of something more like what you had for Xen? Or a tad smaller? I'm only using 1.5GB of the 14GB image you had there. I'd liek to keep this on my SSD but it is only a 55GB (effective) drive Look here, it's a KVM image that are smaller :-) and almost same features on it. http://lime-technology.com/forum/index.php?topic=31623.0 This should be moved in to the VM Appliances! //Peter Link to comment
ironicbadger Posted June 23, 2014 Author Share Posted June 23, 2014 50GB!!!! Any chance of something more like what you had for Xen? Or a tad smaller? I'm only using 1.5GB of the 14GB image you had there. I'd liek to keep this on my SSD but it is only a 55GB (effective) drive Try this? https://mariadb.com/kb/en/resizing-a-virtual-machine-image/ Link to comment
jumperalex Posted June 23, 2014 Share Posted June 23, 2014 Ahhhh I see what you did there ? many thanks. Sent from my XT1060 using Tapatalk Link to comment
jumperalex Posted June 23, 2014 Share Posted June 23, 2014 But wait? That is for you expanding not shrinking... And my guess is I'd have to first shrink my partition before I could shrink the image? Sent from my XT1060 using Tapatalk Link to comment
ironicbadger Posted June 23, 2014 Author Share Posted June 23, 2014 Some people are never happy. When I released it at 15GB people moaned it wasn't enough. 50GB and it's too big. Ugh. I'll upload another version later perhaps that is smaller. Link to comment
jumperalex Posted June 23, 2014 Share Posted June 23, 2014 haha sorry man. It is no rush yet, I'm going to be slow to move ATM until some other things on my end settle down. All I can say is that it does seem like growing is easier than shrinking (based on my research "back in the Xen days ), and being small is "easily fixed" for most use cases by going with a data.img. But I know there are some people out there who oddly think using a data.img is some sort of complicating "hack" instead of a good practice to isolate /system and /data [shrug]. Link to comment
MTA99 Posted June 23, 2014 Share Posted June 23, 2014 I'm just happy to have the image but............. If a smaller image could be made available, I'm comfortable with adding a data.img to extend the storage. Thanks again Link to comment
grumpybutfun Posted June 23, 2014 Share Posted June 23, 2014 I'm just happy to have the image but............. If a smaller image could be made available, I'm comfortable with adding a data.img to extend the storage. Thanks again My advice... 1. Make the image as small as possible but leave enough room for app updates and the configuration settings for the apps. 2. Take advantage of qcow2 thin provisioning. Meaning... if you set the VM to a maximum size of 300MB and only use 150MB it will only take up 150MB of space on your hard drive and will grow when needed to 300MB. 3. Do not use a separate "virtual hard drive" to store your data before it is moved to unRAID or Cache drive via Samba or NFS. Use VirtFS (9p with Virtio) instead. With VirtFS you have a paravirtualized driver that can directly access block devices ( unRAID / cache drives) on the Host instead of using NFS, Samba, etc. Old Way 1. Sickbeard uses SABnzbd to download a TV Show into a Download folder in a separate "virtual hard drive" (data.img). 2. When the TV Show is finished downloading Sickbeard moves it from the separate "virtual hard drive" (data.img) via Samba or NFS (through the network) to where it belongs on the host (unRAID / cache drive). New Way 1. Sickbeard uses SABnzbd to download a TV Show into a Download folder directly using VirtFS on unRAID (or cache drive). 2. When the TV Show is finished downloading Sickbeard moves it from the Download folder on unRAID (or cache drive) via VirtFS (which is A LOT faster than NFS or Samba and without the network traffic) from the Download folder on unRAID (or cache drive) to where it belongs on the host (unRAID / cache drive). Link to comment
ironicbadger Posted June 24, 2014 Author Share Posted June 24, 2014 Cheers G. The reason most people wanted a larger image was for their Plex library. As you say, just use virtfs 9p and put the library directly on the cache drive (or array as you see fit). Link to comment
jumperalex Posted June 24, 2014 Share Posted June 24, 2014 I'm sure that makes sense for Sickbeard (I don't use it) but for Plex we aren't using it for temp storage before moving elsewhere. Plex still has the "issue" of constantly writing to its logs (even with it turned "off" and verbose "off"). The logs are part of the Library. Keep in mind the "library" in this case is not the library of media. It is where Plex stores the media's index, metadata, cover art, logs, and the thumbnails created to provide a filmstrip like interface during scanning. Those thumbnails especially take a fair bit of space. So where to store a large constantly accessed database? An SSD of course. If the systems SSD is the array cache and the Library has been put there, then using VirtFS to access it is of course the way to go. On the other hand if the SSD is outside the array, then the best option is to store the Library there regardless if you put it in the VM image or a data.img. TL;DR, for Plex we aren't using the data.img to store temp data, we are using it to store a large constantly accessed index which keeps HDDs spinning. Link to comment
grumpybutfun Posted June 24, 2014 Share Posted June 24, 2014 I'm sure that makes sense for Sickbeard (I don't use it) but for Plex we aren't using it for temp storage before moving elsewhere. Plex still has the "issue" of constantly writing to its logs (even with it turned "off" and verbose "off"). The logs are part of the Library. Keep in mind the "library" in this case is not the library of media. It is where Plex stores the media's index, metadata, cover art, logs, and the thumbnails created to provide a filmstrip like interface during scanning. Those thumbnails especially take a fair bit of space. So where to store a large constantly accessed database? An SSD of course. If the systems SSD is the array cache and the Library has been put there, then using VirtFS to access it is of course the way to go. On the other hand if the SSD is outside the array, then the best option is to store the Library there regardless if you put it in the VM image or a data.img. TL;DR, for Plex we aren't using the data.img to store temp data, we are using it to store a large constantly accessed index which keeps HDDs spinning. You got it all correctly. Again with qcow2 you can have thin provisioned VMs. Ironic could make the available size of the VM 1TB (but the download still remains small). The actual hard drive space it uses will only be as large as the amount of data you have in your VM and grows by itself up too 1 TB if that what you guys decide needs to happen and set up the image that way. Link to comment
thebigslicksuitd Posted June 25, 2014 Share Posted June 25, 2014 Ironic - any chance you'll re-create that owncloud appliance under KVM? Under Xen, had mine setup as an nginx reverse proxy and would love to get back to that. Thanks! Link to comment
steini84 Posted June 25, 2014 Share Posted June 25, 2014 You can skip the 30-60 sek network delay if you only have one NIC and choose "Enable STP: = NO" in unraid Link to comment
dougnliz Posted June 30, 2014 Share Posted June 30, 2014 Ok so I've got the ArchVM_KVM up and running, at least according to the virsh list command. How do I access the vm? I tried the ssh command in your guide here but it doesn't work. ssh root@archvm ----> Times out ssh root@archVM_KVM ----> Times out I also tried: virsh console ArchVM_KVM ----> that give me this: root@TowerB:/mnt/cache# virsh console ArchVM_KVM Connected to domain ArchVM_KVM Escape character is ^] error: internal error: cannot find character device (null) root@TowerB:/mnt/cache# ssh root@ArchVM_KVM I even looked up the IP my router gave me and tried to ssh to that, but it still just times out. Please help, Doug Link to comment
ljm42 Posted June 30, 2014 Share Posted June 30, 2014 Thanks Ironic! I had huge success with Docker this weekend so I thought I'd try KVM too. I've got the basics going, but I thought I'd mention... In step 2, it should say: nano ArchVM/arch.xml (not cfg) Also, as a step 0 you might want to make sure br0 is enabled in unRAID. I found some instructions over here: http://lime-technology.com/forum/index.php?topic=32733.0 But it is also worth mentioning that if you get an IP address via static DHCP, br0 has a different mac address than eth0 so it will change your unRAID ip. Found that out the hard way Link to comment
ironicbadger Posted June 30, 2014 Author Share Posted June 30, 2014 Good evening all, I am writing this post with a view toward winding up my active involvement in the maintenance of the Xen / KVM / repo images and projects I've been running since January. There are numerous reason for this, some of them personal which I won't bore you with but others are more publishable. With the roadmap published by Limetech this week it is quite clear to me that headless Linux VMs, such my ArchVM, are not long for this world. Docker is the future for LT it seems, I'm not fully convinced yet but maybe I'll be proven wrong. Also, with the talk of potentially dropping Xen and the flip flopping over support various hypervisors I've decided to just call it quits as it takes up too much of my time to keep up with all the changes. I'm in the middle of writing a dissertation and am therefore a little pushed for time. Lately, and by that I mean the last 3-4 months, donations have all but dried up (bar the odd generous $5 here and there, thank you to those who did donate) yet my bandwidth usage is getting higher and higher, it approached 300gb last month. This is not sustainable or fair. As previously mentioned I am an MSc student and here in the UK we get no funding from the government so have to pay our own way and the few ££ a month matter. I'm also just a little bored of what unRAID has to offer. It is rather behind the times, admittedly better lately, but I'm seeing much, much better performance with AUFS and SnapRAID on a proper Linux distro base OS (not this Slackware crap). I enjoy being able to type 'package-manger install package' far more than what unRAID has to offer. I've been an unRAID user for 3+ years and know so much more now than I did when I setup my unRAID system, so if you like my training wheels are coming off. So in summary, I'm not going away or deleting my account or posts or anything but I will with immediate effect stop maintaining my ArchVM images (both Xen and KVM) and the ArchVMrepo. With that said, I use the archVMrepo for friends and family so you may get lucky and piggy back of that for a while but it will go away one day. I wish LT all the best and all that. Ciao for now. Link to comment
Recommended Posts