Jump to content
ironicbadger

[KVM VM IMG] ArchVM <--- deprecated 01/07/2014

21 posts in this topic Last Reply

Recommended Posts

As part of my series of KVM compatible VM images I present to you my Arch Linux appliance.

 

The basic premise is that you boot unRAID into 'KVM (NOT XEN) mode' as I show in my video series below, then download my VM image and run it! Once you've run the image you're free to customise the installation to your hearts content with any application you can find for Arch (that's almost all of them!). Bear in mind that these videos tell you to boot in the Xen mode, ignore this and just boot into 'standard' unRAID mode which has KVM enabled by default.

 

What is Arch Linux and why do I need it in a VM?

 

Archlinux-official-fullcolour.svg

 

Arch Linux (or Arch /??rt?/)[3] is a Linux-based operating system for i686 and x86-64 computers.[4] It is composed predominantly of free and open source software' date='[5'] and supports community involvement.[6]

The design approach of the development team focuses on elegance, code correctness, minimalism, and simplicity, and expects the user to be willing to make some effort to understand the system's operation.[7] A package manager written specifically for Arch Linux, pacman, is used to install, remove and update software packages.

Arch Linux uses a rolling release model, such that a regular system update is all that is needed to obtain the latest Arch software; the installation images released by the Arch team are simply up-to-date snapshots of the main system components.

 

I chose Arch Linux to base this particular VM on because of the way in handles software / packages in addition to it's tiny footprint on disk and in RAM. Arch Linux, just like every other Linux distro, makes use of things called 'repositories' that store packages ready for you to download. The main Arch repo's have a huge array of packages already, but they notably don't include 'those kind of apps' that we use heavily on a media server - this is where the Arch User Repository (AUR) comes in.

 

Members of the Arch community are free to create packages for their favourite apps and Arch provides an easy way to download and compile them on a full install. To make your life easier what I've done is gone ahead and compiled the majority of media apps ahead of time and put them into my own 'unraid6repo' which is preconfigured into your VM, making installing non standard apps a breeze but more on that later.

 

Screen%20Shot%202014-06-22%20at%2022.56.03.png

 

Ok I get it, Arch is awesome. How do I get going?!

 

See the bottom of this post for download links, and post #2 for a series of YouTube videos that take you through the process from start to finish if you prefer that method over a text guide.

 

Youtube Playlist for the setup guide - http://www.youtube.com/playlist?list=PLgUdkYSObfxotSl4c4KJlQKNyQbPAduLl

 

----------------

 

How to download and start this appliance

 

1. Login via SSH or Telnet to your unRAID box using Putty on Windows or Terminal on a Mac.

 

2. Download the .zip file to the location you want to run the VM from and unzip it.

 

cd /mnt/cache     <------ ***
wget http:// <insert download link from bottom of this post>
unzip ArchVM_KVM_v*.zip    <------ this will take approx 5 mins

 

*** this is up to you, you can place the .zip wherever you like. If you change this location, you will need to edit the .cfg file accordingly. If you want to do this use mcedit after you've unzipped the download.

 

nano ArchVM/arch.cfg

 

3. Run your VM.

 

virsh create ArchVM_KVM/arch.xml

 

Once you've booted the VM and checked everything is setup correctly in the XML file you should 'define' your machine using virsh which effectively 'registers' it.

 

virsh define ArchVM_KVM/arch.xml
virsh list
virsh start ArchVM

 

4. Login to your VM via either SSH or Console (working on this) and find the IP address of your Arch Linux VM.

 

User / Passwords

root / unraid6

 

**xl console ArchVM** <-- working on console access, can someone help me fix this 
OR
ssh root@archvm

 

Once you're logged in, to find the IP address of the VM issue

 

ip addr

 

5. Once you're in you should be presented with the screenshot from above with the pretty Ascii art in the shape of the Arch logo. I compiled this image probably several days, if not weeks before you downloaded it so first we need to update the repositories as they will be stale. If there are any package upgrades, do them.

 

pacman -Syu  <-- this should be done as often as possible (weekly minimum ideally)

 

6. We are now ready to start installing packages! w00t! Arch uses something called 'pacman' as it's package manager and it couldn't be simpler. Here are the basics. All commands are CASE SENSITIVE!!!

 

pacman -Sy <insert app name>  <--- eg pacman -S sickbeard-git
pacman -Ss <insert app name>  <--- searches the repositories the VM can access for a package
pacman -R <insert app name>   <--- removes a package

 

7. Access unRAID data from the VM

 

This is where KVM really shines, it's support of block-based protocols. VirtFS 9p is blazing fast and is not a network based protocol. There is an example line in the fstab of the VM ready for you to edit as you see fit, but it's a two step process with this VM.

 

7a. Edit the XML file to point your unraid shares (here is the section below as an example)

 

		
<!-- Virtfs 9p shares -->
<filesystem type='mount' accessmode='passthrough'>
<source dir='/mnt/user/Media/'/> <!-- This is the path of your unRAID share you want to access -->
    	<target dir='Media'/> <!-- This will be referenced in your fstab -->
</filesystem>

 

7b. Now edit the /etc/fstab file 'nano /etc/fstab' from within the VM to include this line. Note the 'Media' at the start which is a reference to the XML file where you 'passed through' the unRAID share. This isn't a network based protocol and gives the VM almost direct access to the drives so it is BLAZING fast.

 

Media /mnt/Media  9p  auto,x-systemd.automount,trans=virtio,version=9p2000.L,rw    0   0

 

There is still about a 30-60 second delay when the VM boots up whilst the unRAID gives it an IP. There is nothing I can do about this, it is not specific to this VM. Sorry! It annoys me too!

 

Samba

 

Add this entry (and change the IP of course) to /etc/fstab. Edit fstab with ' nano /etc/fstab '.

 

nano /etc/fstab
--------------
//10.0.0.2/Media    /mnt/unraid     cifs         auto,x-systemd.automount,guest,noperm,noserverino,uid=nobody,gid=users 0 0

 

---------------------

 

How to create a data .img to act as a datastore for Plex and others...

 

** These steps aren't modified from my Xen VM thread but are almost identical so I'm sure you can figure it out ;) **

 

The base .img for this install is only 50G, for some of you this won't be enough! Here are the steps to add more storage to your VM as some of you have ridiculous Plex libraries!! It is also useful as scratch space for Sabnzbd and/or Tranmission incomplete downloads too...

 

1. Log in to unRAID (not your VM) via SSH, telnet or the console

 

2. Power off the ArchVM if it isn't already

 

xl shutdown ArchVM
xl top <--- wait for ArchVM to disapear

 

3. cd to the directory of your VM for example:

 

cd /mnt/user/cache_only/ArchVM

 

4. Create an .img file of the required size

 

truncate -s 250G data.img    <--- adjust the size as needed, as for as I know there's no limit.

 

5. Edit your arch.cfg with mcedit

 

mcedit arch.cfg
-----------
disk = [
        'file:/mnt/user/cache_only/ArchVM/arch.img,xvda,w',
        'file:/mnt/user/cache_only/ArchVM/data.img,xvdb,w'  <--- add this line
        ]

 

6. Boot up your ArchVM

 

xl create arch.cfg -c

 

7. Check the data.img file is seen with

 

fdisk -l
-------------
Disk /dev/xvdb: 250 GiB

 

8. Create a partition structure and filesystem on the .img

 

fdisk /dev/xvdb
command: o
command: n
partition type: default p
partition num: default 1
First sector: default 
Last sector: default
command: w

mkfs.ext4 /dev/xvdb1

 

9. Mount the new partition in /etc/fstab

 

mkdir /mnt/data

nano /etc/fstab
--------------
/dev/xvdb1    /mnt/data ext4 defaults 0 1

 

Ctrl + X to save a quit the fstab file.

 

Reboot.

 

---------------------

 

 

The IronicBadger repository !

 

The reason for using Arch is two fold. First, it has an incredibly tiny footprint for a full distro and uses almost no RAM. Second, it's what I use and I host a repository full of my favourite apps which are not in the default repositories. The VM already has it added so needs no user interaction.

 

I take requests on adding apps to my repository, but you must first do just a little homework. You must find out first whether the package already exists in the mainline repositories and if it doesn't you must check to see if it exists on the Arch User Repository. I will only add packages from the AUR to my repo!

 

Arch Linux Main Repos - https://www.archlinux.org/packages/

AUR - https://aur.archlinux.org/packages/

 

 

For more information visit http://unraidrepo.ktz.me/.

 

Download Links

Personal hosting - http://repo.ktz.me/archVM/KVM/ArchKVM_v1.zip

 

If you find this VM useful please consider making a donation.

 

btn_donate_SM.gif

Share this post


Link to post

Thanks Ironic! Your work is very much appreciated.

 

I was really excited to try Docker but I don't think its ready for unRAID yet (to be discussed elsewhere). For now your Arch VMs are so fast, lightweight and easy to use that I'm going to stick with Arch for my application needs.

 

This new VirtFS feature sounds great. Actually looking forward to migrating my XEN VMs to KVM #geek  :P

Share this post


Link to post

50GB!!!!  :(

 

Any chance of something more like what you had for Xen?  Or a tad smaller?  I'm only using 1.5GB of the 14GB image you had there.  I'd liek to keep this on my SSD but it is only a 55GB (effective) drive

Share this post


Link to post

50GB!!!!  :(

 

Any chance of something more like what you had for Xen?  Or a tad smaller?  I'm only using 1.5GB of the 14GB image you had there.  I'd liek to keep this on my SSD but it is only a 55GB (effective) drive

Look here, it's a KVM image that are smaller :-) and almost same features on it.

http://lime-technology.com/forum/index.php?topic=31623.0

 

This should be moved in to the VM Appliances!

 

//Peter

 

Share this post


Link to post

Ahhhh I see what you did there ???? many thanks.

 

Sent from my XT1060 using Tapatalk

 

 

Share this post


Link to post

But wait?  That is for you expanding not shrinking...  And my guess is I'd have to first shrink my partition before I could shrink the image?

 

Sent from my XT1060 using Tapatalk

 

 

Share this post


Link to post

Some people are never happy. When I released it at 15GB people moaned it wasn't enough. 50GB and it's too big. Ugh.

 

I'll upload another version later perhaps that is smaller.

Share this post


Link to post

haha sorry man.  It is no rush yet, I'm going to be slow to move ATM until some other things on my end settle down.  All I can say is that it does seem like growing is easier than shrinking (based on my research "back in the Xen days  :o ), and being small is "easily fixed" for most use cases by going with a data.img.  But I know there are some people out there who oddly think using a data.img is some sort of complicating "hack" instead of a good practice to isolate /system and /data [shrug].

Share this post


Link to post

I'm just happy to have the image but.............

 

If a smaller image could be made available, I'm comfortable with adding a data.img to extend the storage.

 

Thanks again  :D

Share this post


Link to post

I'm just happy to have the image but.............

 

If a smaller image could be made available, I'm comfortable with adding a data.img to extend the storage.

 

Thanks again  :D

 

My advice...

 

1. Make the image as small as possible but leave enough room for app updates and the configuration settings for the apps.

 

2. Take advantage of qcow2 thin provisioning. Meaning... if you set the VM to a maximum size of 300MB and only use 150MB it will only take up 150MB of space on your hard drive and will grow when needed to 300MB.

 

3. Do not use a separate "virtual hard drive" to store your data before it is moved to unRAID or Cache drive via Samba or NFS. Use VirtFS (9p with Virtio) instead. With VirtFS you have a paravirtualized driver that can directly access block devices ( unRAID / cache drives) on the Host instead of using NFS, Samba, etc.

 

Old Way

 

1. Sickbeard uses SABnzbd to download a TV Show into a Download folder in a separate "virtual hard drive" (data.img).

 

2. When the TV Show is finished downloading Sickbeard moves it from the separate "virtual hard drive" (data.img) via Samba or NFS (through the network) to where it belongs on the host (unRAID / cache drive).

 

New Way

 

1. Sickbeard uses SABnzbd to download a TV Show into a Download folder directly using VirtFS on unRAID (or cache drive).

 

2. When the TV Show is finished downloading Sickbeard moves it from the Download folder on unRAID (or cache drive) via VirtFS (which is A LOT faster than NFS or Samba and without the network traffic) from the Download folder on unRAID (or cache drive) to where it belongs on the host (unRAID / cache drive).

 

Share this post


Link to post

Cheers G.

 

The reason most people wanted a larger image was for their Plex library. As you say, just use virtfs 9p and put the library directly on the cache drive (or array as you see fit).

Share this post


Link to post

I'm sure that makes sense for Sickbeard (I don't use it) but for Plex we aren't using it for temp storage before moving elsewhere. 

 

Plex still has the "issue" of constantly writing to its logs (even with it turned "off" and verbose "off").  The logs are part of the Library.  Keep in mind the "library" in this case is not the library of media.  It is where Plex stores the media's index, metadata, cover art, logs, and the thumbnails created to provide a filmstrip like interface during scanning.  Those thumbnails especially take a fair bit of space.

 

So where to store a large constantly accessed database?  An SSD of course.

 

If the systems SSD is the array cache and the Library has been put there, then using VirtFS to access it is of course the way to go. 

 

On the other hand if the SSD is outside the array, then the best option is to store the Library there regardless if you put it in the VM image or a data.img.

 

TL;DR, for Plex we aren't using the data.img to store temp data, we are using it to store a large constantly accessed  index which keeps HDDs spinning.

Share this post


Link to post

I'm sure that makes sense for Sickbeard (I don't use it) but for Plex we aren't using it for temp storage before moving elsewhere. 

 

Plex still has the "issue" of constantly writing to its logs (even with it turned "off" and verbose "off").  The logs are part of the Library.  Keep in mind the "library" in this case is not the library of media.  It is where Plex stores the media's index, metadata, cover art, logs, and the thumbnails created to provide a filmstrip like interface during scanning.  Those thumbnails especially take a fair bit of space.

 

So where to store a large constantly accessed database?  An SSD of course.

 

If the systems SSD is the array cache and the Library has been put there, then using VirtFS to access it is of course the way to go. 

 

On the other hand if the SSD is outside the array, then the best option is to store the Library there regardless if you put it in the VM image or a data.img.

 

TL;DR, for Plex we aren't using the data.img to store temp data, we are using it to store a large constantly accessed  index which keeps HDDs spinning.

 

You got it all correctly.

 

Again with qcow2 you can have thin provisioned VMs. Ironic could make the available size of the VM 1TB (but the download still remains small). The actual hard drive space it uses will only be as large as the amount of data you have in your VM and grows by itself up too 1 TB if that what you guys decide needs to happen and set up the image that way.

 

Share this post


Link to post

Ironic - any chance you'll re-create that owncloud appliance under KVM? Under Xen, had mine setup as an nginx reverse proxy and would love to get back to that. Thanks!

Share this post


Link to post

You can skip the 30-60 sek network delay if you only have one NIC and choose "Enable STP: = NO" in unraid

Share this post


Link to post

Ok so I've got the ArchVM_KVM up and running, at least according to the virsh list command.

 

How do I access the vm?  I tried the ssh command in your guide here but it doesn't work.

 

ssh root@archvm          ---->  Times out

ssh root@archVM_KVM  ---->  Times out

 

I also tried:

 

virsh console ArchVM_KVM  ---->  that give me this:

 

root@TowerB:/mnt/cache# virsh console ArchVM_KVM
Connected to domain ArchVM_KVM
Escape character is ^]
error: internal error: cannot find character device (null)

root@TowerB:/mnt/cache# ssh root@ArchVM_KVM

 

I even looked up the IP my router gave me and tried to ssh to that, but it still just times out.

 

Please help,

 

Doug

Share this post


Link to post

Thanks Ironic!  I had huge success with Docker this weekend so I thought I'd try KVM too.  I've got the basics going, but I thought I'd mention...

 

In step 2, it should say:

  nano ArchVM/arch.xml (not cfg)

 

Also, as a step 0 you might want to make sure br0 is enabled in unRAID.  I found some instructions over here:

  http://lime-technology.com/forum/index.php?topic=32733.0

But it is also worth mentioning that if you get an IP address via static DHCP, br0 has a different mac address than eth0 so it will change your unRAID ip.  Found that out the hard way :)

 

Share this post


Link to post

Good evening all,

 

I am writing this post with a view toward winding up my active involvement in the maintenance of the Xen / KVM / repo images and projects I've been running since January. There are numerous reason for this, some of them personal which I won't bore you with but others are more publishable.

 

With the roadmap published by Limetech this week it is quite clear to me that headless Linux VMs, such my ArchVM, are not long for this world. Docker is the future for LT it seems, I'm not fully convinced yet but maybe I'll be proven wrong. Also, with the talk of potentially dropping Xen and the flip flopping over support various hypervisors I've decided to just call it quits as it takes up too much of my time to keep up with all the changes. I'm in the middle of writing a dissertation and am therefore a little pushed for time. Lately, and by that I mean the last 3-4 months, donations have all but dried up (bar the odd generous $5 here and there, thank you to those who did donate) yet my bandwidth usage is getting higher and higher, it approached 300gb last month. This is not sustainable or fair. As previously mentioned I am an MSc student and here in the UK we get no funding from the government so have to pay our own way and the few ££ a month matter.

 

I'm also just a little bored of what unRAID has to offer. It is rather behind the times, admittedly better lately, but I'm seeing much, much better performance with AUFS and SnapRAID on a proper Linux distro base OS (not this Slackware crap). I enjoy being able to type 'package-manger install package' far more than what unRAID has to offer. I've been an unRAID user for 3+ years and know so much more now than I did when I setup my unRAID system, so if you like my training wheels are coming off.

 

So in summary, I'm not going away or deleting my account or posts or anything but I will with immediate effect stop maintaining my ArchVM images (both Xen and KVM) and the ArchVMrepo. With that said, I use the archVMrepo for friends and family so you may get lucky and piggy back of that for a while but it will go away one day.

 

I wish LT all the best and all that.

 

Ciao for now.

Share this post


Link to post
Guest
This topic is now closed to further replies.