limetech Posted June 18, 2014 Share Posted June 18, 2014 Download Please see readme.txt in the release zip file for installation/upgrade instructions. Disclaimer: This is beta software. While every effort has been made to ensure no data loss, use at your own risk! unRaid OS has undergone major development and there comes a time when you release what you got so far, and then carry on. With this release we have three different means of "virtualization": Xen, KVM, and docker. Of course you can run normally without using any virtualization features as well. I was hoping to release after more webGui refinements, but docker is such a great feature that we felt it would be worth it to release what we have in place for it now. Thanks to Jon Panozzo and Eric Schultz for their many long hours of development and testing. Jon will modify this post in the next 24 hours with some links to docker guides. Summary of changes from 6.0-beta5a to 6.0-beta6 ----------------------------------------------- - docker: added support for docker version 1.0.0 - emhttp: support btrfs-formatted cache disk - linux: update to version 3.15.0 - linux: added more features: CONFIG_NR_CPUS: changed from 8 to 16 (number of CPU's supported) CONFIG_NUMA: Enable NUMA (Non Uniform Memory Access) support. CONFIG_SCSI_MVSAS_TASKLET: Support for interrupt tasklet (improves mvsas performance) CONFIG_ALX: Qualcomm Atheros AR816x/AR817x support CONFIG_I40E: Intel(R) Ethernet Controller XL710 Family support CONFIG_AMD_IOMMU: AMD IOMMU support CONFIG_INTEL_IOMMU: Support for Intel IOMMU using DMA Remapping Devices CONFIG_INTEL_IOMMU_DEFAULT_ON: Enable Intel DMA Remapping Devices by default CONFIG_IRQ_REMAP: Support for Interrupt Remapping CONFIG_VFIO: VFIO Non-Privileged userspace driver framework CONFIG_VFIO_PCI: VFIO support for PCI devices CONFIG_VFIO_PCI_VGA: VFIO PCI support for VGA devices CONFIG_CIFS_SMB2: SMB2 network file system support CONFIG_NET_9P: Plan 9 Resource Sharing Support (9P2000) CONFIG_NET_9P_VIRTIO: 9P Virtio Transport CONFIG_9P_FS: Plan 9 Resource Sharing Support (9P2000) CONFIG_9P_FS_POSIX_ACL: 9P POSIX Access Control Lists CONFIG_MAGIC_SYSRQ - slack: updated packages with security advisories: curl-7.36.0 gnutls-3.1.25 ntp-4.2.6p5 openssh-6.6p1 openssl-1.0.1h openssl-solibs-1.0.1h shadow-4.1.5.1 libtasn1-3.6 php-5.4.29 samba-4.1.6 - slack: added packages: btrfs-progs libcgroup sqlite - slack: minor fixes in rc.S, rc.M, rc.sshd - qemu: version 2.0.0 with SeaBIOS 1.7.5 - xen: update to version 4.4.0, with security patches through 6-13-2014. Eliminated qemu-system-i386 altogether and now just builds qemu-system-x86-64 (qemu 2.0.0) Correct typo in /etc/default/xendomains. BEGIN JONP EDITS Boot Modes In Beta 6, users will have two options for booting into unRAID, just like before: Basic Mode or Xen Mode. However, there are now more differences in these two modes: Basic (Docker + libvirt) Basic mode in beta 6 comes with the same features and capabilities that beta 5a had before it. In addition, basic mode is what we call "KVM Ready." This is similar to TVs that were sold back in the day with being "HD Ready." Essentially this means that if you want VMs in this mode, you can use KVM to create them, but you don't have to. And unlike Xen-mode, there is no "extra Linux kernel" required to load before unRAID Server OS. The lack of this extra step makes this mode easier to support. That said, KVM is only usable via command-line with libvirt (virsh) tool in beta 6. Xen (Docker + Xen/xl) Selecting the Xen Mode boot option will load the Xen hypervisor before booting into unRAID mode (see http://wiki.xenproject.org/wiki/Hypervisor). If upgrading from a previous unRAID 6 Beta making use of the Xen hypervisor, your system should automatically boot into this mode. Xen was the first hypervisor that Lime Tech implemented in the unRAID 6 beta and has undergone some upgrades for this release. NOTE: Booting into Xen mode also includes libvirt for VM management. Existing Xen users can choose to use EITHER tool when booting into Xen mode to create / manage their VMs. That said, it is not recommended to have a mix of VMs created by both tools running simultaneously (and this is not supported). One or the other people ;-). Docker: Requirements and Setup One of the biggest new improvements to unRAID 6 is the inclusion of Docker. Docker allows us to safely use applications that were not built to run on unRAID Server OS. These applications run in a secure, compartmentalized fashion with minimal consumption of hardware resources. Docker is available in all three Boot Modes (Basic, KVM, and Xen) in unRAID Beta 6d. Docker does not require any type of virtualization support on the hardware. Docker performs well when installed on either array or cache drive devices (SATA or SSD). Instructions for setup and configuration can be found in the Docker Quick-Start Guide in our Lime Tech Forums. Docker for Plugin Authors When writing an unRAID Plugin, authors have a painful process of development and testing they have to go through in order to both build and maintain the plugin properly. We believe Docker offers a solution to many of the common problems with plugin design and management. We believe that improving this platform is a key goal to our success in growing as a company and as a community. Experiment with Docker in Beta 6d and let us know what you think in our Community Forums by imagining the possibilities and sharing your views! unRAID 6 Beta 6 Supplemental Content Throughout the course of this week, guides and other details will be posted by JonP on how to best utilize this beta and take advantage of everything baked into it. As they are posted, this post will be updated here to add links directly to these guides. In addition, forum changes will occur in the weeks ahead to better arrange/organize content. Docker Quick-Start Guide Btrfs Quick-Start Guide VM Quick-Start Guide (KVM / Xen) Converting Xen CFG to libvirt XML New VM-Specific Updates (KVM / Xen) Link to comment
NAS Posted June 18, 2014 Share Posted June 18, 2014 It is a monster ! Congratulations I think this is the biggest change in a single beta in the history of the project. Link to comment
jamescarlos Posted June 18, 2014 Share Posted June 18, 2014 Thanks Tom and Team! This is loaded... Link to comment
alikuenkano Posted June 18, 2014 Share Posted June 18, 2014 Amazing beta release !!! Good work guys !!! Link to comment
sacretagent Posted June 18, 2014 Share Posted June 18, 2014 Hi Tom, guess you need to add a 5 at this Upgrade Instructions ==================== If you are currently running 6.0-beta1/2/3/4 -------------------------------------------- It is only necessary to copy these files from the zip file to the root of your USB Flash device: bzimage bzroot xen readme.txt no need to replace syslinux folder i guess as it is identical as the 5a release but without having installed the new beta yet how are we going to select docker for example ? syslinux only chooses between these default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode gues that is only unraid and xen ? how are we going to choose kvm or docker ? Link to comment
itimpi Posted June 18, 2014 Share Posted June 18, 2014 Do not see anywhere in the release notes that support for KVM has been added even though it was mentioned in this threads opening post! I guess the thing to do is to get this beta up and running to see what it actually provides! Just downloaded it so off to try that Link to comment
peter_sm Posted June 18, 2014 Share Posted June 18, 2014 IF KVM is added, what version of libvirt are used ? 1.2.5? that are the latest. KVM also need to have a own section in syslinux, since that are a different way than xen to assign PCI pass through. //Peter Link to comment
PeterB Posted June 18, 2014 Share Posted June 18, 2014 gues that is only unraid and xen ? how are we going to choose kvm or docker ? My guess is that kvm and docker are both permanently available in the image, or are loaded at run time. Enabling either/both must be via the 'Extensions' page in the web GUI. Link to comment
dalben Posted June 18, 2014 Share Posted June 18, 2014 Nice work. I've got a weeks leave booked within the next month and now I'll know what I will be doing during that time. Link to comment
itimpi Posted June 18, 2014 Share Posted June 18, 2014 One strange thing I have immediately noticed is that if I boot into Xen mode I can no longer log in to the console. The last message displayed on the console is "loading libvirtd". Telnet and webGUI appear to be OK so the system is successfully loaded. It is almost as if GPU pass-thru has been activated despite the fact I have not set up any specific settings for this. Anyone else seeing this or have any idea of something I should check? Link to comment
itimpi Posted June 18, 2014 Share Posted June 18, 2014 My first 'buglet' report on beta 6! If you try and start the docker service and the location which docker is to use (defaults to /mnt/cache/docker) is not in btrfs format then an error message is flashed up and then disappears before you really have time to work out what it says. I only managed to find out what the problem was by looking in the syslog. I think that ideally two things should happen: The error message should persist longer The option to start the service should be disabled if the target location is not in btrfs format if this is a requirement (ideally with a message in the GUI stating this requirement). Off now to look at clearing current contents of my cache drive so that I can reformat it (or convert it) to btrfs format Link to comment
sacretagent Posted June 18, 2014 Share Posted June 18, 2014 My first 'buglet' report on beta 6! If you try and start the docker service and the location which docker is to use (defaults to /mnt/cache/docker) is not in btrfs format then an error message is flashed up and then disappears before you really have time to work out what it says. I only managed to find out what the problem was by looking in the syslog. I think that ideally two things should happen: The error message should persist longer The option to start the service should be disabled if the target location is not in btrfs format if this is a requirement (ideally with a message in the GUI stating this requirement). Off now to look at clearing current contents of my cache drive so that I can reformat it (or convert it) to btrfs format LOL there goes docker... i have no space in my box for 2 cache drives in btrfs Link to comment
itimpi Posted June 18, 2014 Share Posted June 18, 2014 My first 'buglet' report on beta 6! If you try and start the docker service and the location which docker is to use (defaults to /mnt/cache/docker) is not in btrfs format then an error message is flashed up and then disappears before you really have time to work out what it says. I only managed to find out what the problem was by looking in the syslog. I think that ideally two things should happen: The error message should persist longer The option to start the service should be disabled if the target location is not in btrfs format if this is a requirement (ideally with a message in the GUI stating this requirement). Off now to look at clearing current contents of my cache drive so that I can reformat it (or convert it) to btrfs format LOL there goes docker... i have no space in my box for 2 cache drives in btrfs As I understand it, it is quite possible to have single drive in btrfs format. I guess I will be finding out Having said that if I use 2.5" or SSD drives then I could squeeze in two drives into spare space in the box. I have the SATA connections available, just no space for another 3.5" drive. Link to comment
butlerpeter Posted June 18, 2014 Share Posted June 18, 2014 My first 'buglet' report on beta 6! If you try and start the docker service and the location which docker is to use (defaults to /mnt/cache/docker) is not in btrfs format then an error message is flashed up and then disappears before you really have time to work out what it says. I only managed to find out what the problem was by looking in the syslog. I think that ideally two things should happen: The error message should persist longer The option to start the service should be disabled if the target location is not in btrfs format if this is a requirement (ideally with a message in the GUI stating this requirement). Off now to look at clearing current contents of my cache drive so that I can reformat it (or convert it) to btrfs format Would be interesting to know why the docker location has to be on a btfs formatted drive? It's been a while since I had a (brief) play with docker but I'm pretty sure I ran it in an Ubuntu VM on a disk formatted as ext4. Link to comment
itimpi Posted June 18, 2014 Share Posted June 18, 2014 Would be interesting to know why the docker location has to be on a btfs formatted drive? It's been a while since I had a (brief) play with docker but I'm pretty sure I ran it in an Ubuntu VM on a disk formatted as ext4. I suspect it is to do with the ability for multiple docker containers (or VM's) to only store differences to their base. It would have a big effect on reducing storage space required. I could see docker being the basis of providing some of the new monitoring/recovery features that are planned without the risk of de-stabilizing the core system. It looks like it would be more robust than the current plugins that live in core unRAID and would also make it easier to provide updates outside major unRAID releases. However that is just a 'guess' rather than any real knowledge of what is planned. However if docker is going to demand a btrfs device that might become an issue? I guess resolving this sort of issue is the purpose of a beta program. Link to comment
jumperalex Posted June 18, 2014 Share Posted June 18, 2014 Seems like a lot of jumping to conclusions. It is just as likely a bug since I strongly doubt Docker itself has that requirement. A quick read of the Docker install says nothing about needing a specific FS for docker (http://docs.docker.com/installation/ubuntulinux/). I'm not ruling out some unraid-ism, but really, we're already seeing posts in other threads stating this ("docker requires btrfs") as near fact EDIT: or it could be incompatible with RFS, but again ... waiting for official answers is probably best. Link to comment
itimpi Posted June 18, 2014 Share Posted June 18, 2014 Seems like a lot of jumping to conclusions. I am often good at that It is just as likely a bug since I strongly doubt Docker itself has that requirement. A quick read of the Docker install says nothing about needing a specific FS for docker (http://docs.docker.com/installation/ubuntulinux/). I am sure that is correct. I'm not ruling out some unraid-ism, but really, we're already seeing posts in other threads stating this ("docker requires btrfs") as near fact It appears to be a requirement in beta 6. EDIT: or it could be incompatible with RFS, but again ... waiting for official answers is probably best. Why take all the fun out of it Link to comment
peter_sm Posted June 18, 2014 Share Posted June 18, 2014 My Ubuntu and Arch KVM machine is running under B6 :-) Needed to remove this line from my xml file <input type='keyboard' bus='ps2'/> But that shall not be necessary for a functional VM. I though I should find a KVM Manager like Xen Manager :-) Continue play with this ..... GREAT JOB on B6 LT team!! Link to comment
JustinChase Posted June 18, 2014 Share Posted June 18, 2014 Mover still isn't working for me. I have a user share called "download", which is set to use Cache Disk: Only When I start mover manually (because it's not working, and my cache drive is 1/2 full again), I see this in the syslog... Jun 18 06:35:40 media logger: mover started Jun 18 06:35:40 media logger: skipping JRiver/ Jun 18 06:35:40 media logger: skipping VM/ Jun 18 06:35:40 media logger: moving download/ It sees the first 2 shares as cache only, but not the download share. Not sure why that is. With that said, it tries to move files from this directory to someplace, and I get the following lines in the syslog, indicating I'm out of space, but that's not correct either. Each drive in my array has a mimimum of 100GB free space right now... Jun 18 06:35:41 media logger: ./download/incomplete/House.of.Cards.S02E02.1080p.BluRay.x264-DEiMOS/__ADMIN__/SABnzbd_nzf_AA7E0I Jun 18 06:35:41 media shfs/user0: shfs_mkdir: assign_disk: download/incomplete (28) No space left on device Jun 18 06:35:41 media logger: rsync: recv_generator: mkdir "/mnt/user0/download/incomplete" failed: No space left on device (28) Jun 18 06:35:41 media logger: *** Skipping any contents from this failed directory *** Jun 18 06:35:41 media logger: .d..t...... download/ Jun 18 06:35:41 media logger: cd+++++++++ download/incomplete/ Jun 18 06:35:41 media logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0] Jun 18 06:35:41 media logger: ./download/incomplete/House.of.Cards.S02E02.1080p.BluRay.x264-DEiMOS/__ADMIN__/SABnzbd_nzf_D88b0C Jun 18 06:35:41 media shfs/user0: shfs_mkdir: assign_disk: download/incomplete (28) No space left on device Jun 18 06:35:41 media logger: rsync: recv_generator: mkdir "/mnt/user0/download/incomplete" failed: No space left on device (28) Jun 18 06:35:41 media logger: *** Skipping any contents from this failed directory *** I have a folder under another share called download, but it's located at /mnt/user/video/download I can't see why this subfolder having the same name as a share should cause an issue, but I don't know what is causing the problems with mover, so this is just a stab in the dark. Full syslog attached. Also attached are my share.cfg and the download.cfg for the download share I really want to get this resolved, so I can clean up my cache drive, and format it to btrfs and try out Docker. ** I just noticed that my download share shows an odd amount of free space. The other cache only shares show the amount of free space on the cache drive, and the regular shares show the amount of free space on the whole array, but the download share shows something in the middle of these 2 values. Weird. User Shares Name Free JRiver 269 GB VM 269 GB backup 5.36 TB documents 5.36 TB download 3.23 TB music 5.36 TB photos 5.36 TB video 5.63 TB In looking further, I see there is a compute... option for the size column, and when I click it, I see that the download share points to 3 discs for some reason... Name Size Free download 33.7 GB 3.23 TB Cache: 33.7 GB 269 GB Disk 10: 0 B 1.79 TB Disk 5: 0 B 1.17 TB Why is it wanting to use these other 2 discs, and why is mover telling me there is no room when I have over 1TB free on each drive? syslog.zip share.cfg download.cfg Link to comment
PeterB Posted June 18, 2014 Share Posted June 18, 2014 Why is it wanting to use these other 2 discs, and why is mover telling me there is no room when I have over 1TB free on each drive? Just a stab in the dark ... are you sure that there are no /mnt/disk5/download and /mnt/disk10/download folders? 269GB + 1.79TB + 1.17TB = 3.23TB At a guess, it's using the existence of download folders on array disks to make the decision that download is not a cache-only share. It then attempts to move files from cache drive to array drives and finds that it is not permitted to use the array drives - hence 'no space ...'. Link to comment
autumnwalker Posted June 18, 2014 Share Posted June 18, 2014 Docker support? Did not see that coming. Nice work guys! Link to comment
prostuff1 Posted June 18, 2014 Share Posted June 18, 2014 Mover still isn't working for me. I have a user share called "download", which is set to use Cache Disk: Only When I start mover manually (because it's not working, and my cache drive is 1/2 full again), I see this in the syslog... Jun 18 06:35:40 media logger: mover started Jun 18 06:35:40 media logger: skipping JRiver/ Jun 18 06:35:40 media logger: skipping VM/ Jun 18 06:35:40 media logger: moving download/ It sees the first 2 shares as cache only, but not the download share. Not sure why that is. With that said, it tries to move files from this directory to someplace, and I get the following lines in the syslog, indicating I'm out of space, but that's not correct either. Each drive in my array has a mimimum of 100GB free space right now... Jun 18 06:35:41 media logger: ./download/incomplete/House.of.Cards.S02E02.1080p.BluRay.x264-DEiMOS/__ADMIN__/SABnzbd_nzf_AA7E0I Jun 18 06:35:41 media shfs/user0: shfs_mkdir: assign_disk: download/incomplete (28) No space left on device Jun 18 06:35:41 media logger: rsync: recv_generator: mkdir "/mnt/user0/download/incomplete" failed: No space left on device (28) Jun 18 06:35:41 media logger: *** Skipping any contents from this failed directory *** Jun 18 06:35:41 media logger: .d..t...... download/ Jun 18 06:35:41 media logger: cd+++++++++ download/incomplete/ Jun 18 06:35:41 media logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0] Jun 18 06:35:41 media logger: ./download/incomplete/House.of.Cards.S02E02.1080p.BluRay.x264-DEiMOS/__ADMIN__/SABnzbd_nzf_D88b0C Jun 18 06:35:41 media shfs/user0: shfs_mkdir: assign_disk: download/incomplete (28) No space left on device Jun 18 06:35:41 media logger: rsync: recv_generator: mkdir "/mnt/user0/download/incomplete" failed: No space left on device (28) Jun 18 06:35:41 media logger: *** Skipping any contents from this failed directory *** I have a folder under another share called download, but it's located at /mnt/user/video/download I can't see why this subfolder having the same name as a share should cause an issue, but I don't know what is causing the problems with mover, so this is just a stab in the dark. Full syslog attached. Also attached are my share.cfg and the download.cfg for the download share I really want to get this resolved, so I can clean up my cache drive, and format it to btrfs and try out Docker. ** I just noticed that my download share shows an odd amount of free space. The other cache only shares show the amount of free space on the cache drive, and the regular shares show the amount of free space on the whole array, but the download share shows something in the middle of these 2 values. Weird. User Shares Name Free JRiver 269 GB VM 269 GB backup 5.36 TB documents 5.36 TB download 3.23 TB music 5.36 TB photos 5.36 TB video 5.63 TB In looking further, I see there is a compute... option for the size column, and when I click it, I see that the download share points to 3 discs for some reason... Name Size Free download 33.7 GB 3.23 TB Cache: 33.7 GB 269 GB Disk 10: 0 B 1.79 TB Disk 5: 0 B 1.17 TB Why is it wanting to use these other 2 discs, and why is mover telling me there is no room when I have over 1TB free on each drive? If I had to guess, the download folder at some point made it onto an array disk. While it is set to cache only now since the download folder does exist on the array it likely is "parsing it" into the folders that need to be moved. Inspect all your disk shares and see if you have a folder called download hanging out on any of them. Link to comment
JustinChase Posted June 18, 2014 Share Posted June 18, 2014 ... are you sure that there are no /mnt/disk5/download and /mnt/disk10/download folders? If I had to guess, the download folder at some point made it onto an array disk. While it is set to cache only now since the download folder does exist on the array it likely is "parsing it" into the folders that need to be moved. Inspect all your disk shares and see if you have a folder called download hanging out on any of them. Bing, bing, bing!!! both correct. I have now eliminated this folder on both drives. Will report back later. I would like to suggest that this may still be a 'bug' and that changing a share to cache only would move these folders off of any array drives, thus preventing this in the future, and not require me to be as smart as you 2 Thanks to you both for your help. Link to comment
jumperalex Posted June 18, 2014 Share Posted June 18, 2014 Careful with that suggestion. Doing what you ask means unraid is responsible for knowing you want to move data from the array into the cache. I'm not saying that is impossible, but some things are better left to the user to do manually. Personal Opinion: unraid just shouldn't let you change a share to be cache only. It should block that option once the share is created. Instead you would have to make a new cache-only share, move the data, then delete the old array share. And that could be documented in the webgui "help" when it won't let you change from an array share to a cache-only share. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.