Compiling a custom kernel with unRAID 4.2.1

[bubbaQ]

Why compile a custom kernel?

 

Well, the standard distro of unRAID is very conservative -- it has some features left out of the kernel because they may be new and untested, or may not be stable in all circumstances.  The standard distro has to be generic enough to work on a wide range of hardware.

 

There are also kernel features and software that would only be of interest to a small subset of users, and which may be prone to support issues.  So they are left out.

 

Now I know what my hardware is, so I can optimize the kernel for it.  I can enable features, such as CPU Frequency Scaling, in the kernel that I need and that I know work well on my system.

 

And it proves to the nay-sayers that Tom has complied with the GPL... all the code is there to compile your own system.

[dertbv]

"It is not for the feint of heart, but if anyone wants the directions to repeat this process and get a development system u p and running to build a custom kernel with unRAID, let me know.  The hard part is the initial config file which I can give you.  After you get it set up, it is just one script to compile, build the initramfs, and copy it all to the usb stick... so it is pretty easy to test out new kernels."

 

I would love to get a copy of the initial config file.  I have been trying to build a custom kernel for a few days.  I am rather new at linux so I need all the help i can get..

 

thanks

[bubbaQ]

If you have a development system with Slackware 12 with gcc 4.2.1, grab the kernel 4.6.22.5 source from  here:

 

  http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.5.tar.gz

 

You should work with the identical source as was used for the release of unRAID you are modding (which is 2.6.22.5).

 

I'll clean up the config file and get it posted.  But remember, it is tailored for my machine, and I have enabled only the drivers I need, as well as optimizations for my HW (such as AMD k8 specific features, instrumentation support (for lm_sensors), and cpu frequency support).  You'll have to edit the config to make it work for your hardware.

 

You will also have to uncompress your own bzroot file, copy the recompiled modules and module config files to it, then recompress it back to the USB stick.

 

 

[dertbv]

Thanks for the help.  I am running it on a K8N Neo4.  I have already gotten the bzoot file uncompressed and copied over into a temp directory.  And I have gotten the temp file back on to the stick and running it with a few modifications.  Just trying to add my raid cards 2320 and 2314 rocket raid  to the kernel. 

Thanks again

 

 

[bubbaQ]

What a coincidence... I'm running it on a K8N-E.

 

Attached is my .config file for my latest build on the K8N-E.

 

I've tried several times to upload the config file (as txt and as a zip) but the forum software hangs.  I'll try again tomorrow.

[limetech]

Starting with the next release (4.3) the .config file will be included in the /usr/src/linux directory on the flash - we should have included it all along - just an oversight.

 

One reason the kernel is "conservative" is because there is no swap space allocated.  Once you start running out of memory the kernel randomly kills processes.  May not make very next release, but soon we'll be adding kernel support for more memory, and swap space on an array disk or external usb/sata disk/flash.

[cmhardwick]

yeah, swap space. Would be great, as someone did with a custom script I think, small drive in the box, not part of the array, swapdisk.

 

Great to see you back, BTW!

[bubbaQ]

Thanks Tom.  That'll be nice.

[Suse User]

As far as custom Kernels go I have very little (and I mean very little) linux knowledge, I don't even know how to compile really, let alone how to script it.  I've done a few compiles (mainly for Tivo apps) by blindly following step-by-step instructions that generally seem to work, but this is because of excellent instructions which have also provided include files (whatever they are).

 

Now I obviously don't want a tutorial here, but can anyone point to a good resource for someone who's pretty PC literate but wants to get an understanding of how to compile custom Kernels under Linux?  I write my own extremely complex VBA apps that I use in my business, but I think I'll experiment on a Kubuntu MCE install I'm trying out rather than mess up my unraid.

 

Starting with the next release (4.3) the .config file will be included in the /usr/src/linux directory on the flash - we should have included it all along - just an oversight.

 

One reason the kernel is "conservative" is because there is no swap space allocated.  Once you start running out of memory the kernel randomly kills processes.  May not make very next release, but soon we'll be adding kernel support for more memory, and swap space on an array disk or external usb/sata disk/flash.

 

You've done it now!  Wait for the... "When will it be?!  "Will it include xxx?" etc.  The moment you mention a new release I want it yesterday!

 

Glad to know something is on the horizon.

 

Mark.

 

PS. NFS?   Don't answer this point, it's the wrong thread, it either will be in 4.3 or it won't.

 

 

[bubbaQ]

First, install a complete Slackware development system.

1) Get the kernel source for 4.6.22.5 (see above post)

2) Copy the unRAID source mods

3) Get a config file... either from someone else or mod your own

4) Build new kernel and modules

5) uncompress bzroot filesystem from USB

6) copy new kernel and modules to uncompressed bzroot filesystem.

7) recompress new bzroot filesystem back to usb.

edit syslinux config on flash to give you boot option to run custom unRAID or standard.

 

This biggest bear is step 3 -- the config file.... you either have to edit it by hand or use "make menuconfig" and know what to change and how to change it.  The best thing to do is wait for unRAID 4.3 so you can get Tom's actual config file to start with rather than starting from scratch like I did.

[WeeboTech]

What a coincidence... I'm running it on a K8N-E.

 

Attached is my .config file for my latest build on the K8N-E.

 

I've tried several times to upload the config file (as txt and as a zip) but the forum software hangs.  I'll try again tomorrow.

 

Can you try again or would you mind sending it directly to my email?

Thanks.

[jcolo]

 

2) Copy the unRAID source mods

 

 

Which of the unRAID directories/files need to be copied over the linux-2.6.22.5 extracted kernel source directories ?

 

 

[WeeboTech]

 

Look in /usr/src/linux on your unraid system

 

Bring over the whole md directory.

 

What I did was mv the original md directory out of the way on my development vmware system.

Then mv this md directory in it;s place.

 

I took the old config and did a

make oldconfig

Then did a

make menuconfig

and customized what I need.

 

I did not do the bzroot step yet, but plan to in the next day or so.

got sidetracked with mt-daapd.

 

Here's a screen capture of the location.

 

 

root@unraid:/boot/custom/bin# cd /usr/src
root@unraid:/usr/src# ls -l
total 0
lrwxrwxrwx 1 root root 12 Feb 23 12:01 linux -> linux-2.6.20/
drwxr-xr-x 3 root root  0 Feb 23 12:01 linux-2.6.20/
root@unraid:/usr/src# cd linux
root@unraid:/usr/src/linux# ls -l
total 688
-rw-r--r-- 1 root root 701078 Feb 23 12:01 System.map
drwxr-xr-x 3 root root      0 Feb 23 12:01 drivers/
root@unraid:/usr/src/linux# cd drivers
root@unraid:/usr/src/linux/drivers# ls -l
total 0
drwxr-xr-x 2 root root 0 Feb 23 12:01 md/
root@unraid:/usr/src/linux/drivers# cd md
root@unraid:/usr/src/linux/drivers/md# ls -l
total 124
-rw-r--r-- 1 root root   346 Feb 23 12:01 Kconfig
-rw-r--r-- 1 root root   118 Feb 23 12:01 Makefile
-rw-r--r-- 1 root root 55140 Feb 23 12:01 md.c
-rw-r--r-- 1 root root 10172 Feb 23 12:01 md_private.h
-rw-r--r-- 1 root root 43372 Feb 23 12:01 unraid.c
-rw-r--r-- 1 root root  3627 Feb 23 12:01 xor.c

 

 

[bubbaQ]

WeeboTech, you have the wrong source.  You MUST get kernel source for 4.6.22.5... not 4.6.20 that is shown in your post.

[jcolo]

I think the 4.6.20 directory he is referencing is from the unRAID uncompressed bzroot.  That's the only directory I find which has the unRAID source files like unraid.c

[WeeboTech]

I think the 4.6.20 directory he is referencing is from the unRAID uncompressed bzroot.  That's the only directory I find which has the unRAID source files like unraid.c

This is correct.

I downloaded the correct kernel source and use the md driver source from the supplied ram disk for unraid.

That source just happens to have a  2.4.6.20 directory.

 

I had some issues compiling it initially, I ended up removing some other drivers and it compiled thereafter.

 

 

 

[BLKMGK]

Curious, how hard would it be to setup and distribute a VMware image of the development environment? Perhaps via orrent so that we would all be working from a common environment when attempting to add functionality?

[xbit]

I'd be interested in a VMWare image too!

[WeeboTech]

We just need some extra space somewhere.

I can make one easily.

 

[xbit]

Here's a site that lists several file hosting providers (I think most have free options).  I haven't used any of them so I don't have any specific recommendations.

 

http://www.freewebspace.net/guide/diskstorage.shtml

[BLKMGK]

Actually, VMware has "appliances" that can be downloaded on their site. It might be possible to upload one there to share with everyone. I do not think that there's anything too strict about what you upload so long as there's no infringement.

 

For that matter if someone were to create one, create a torrent, post the Torrent link here, and then the rest of us could download it. Now, we would all need to be nice and leave the image up for others so the first person didn't get slammed but DHT and refrences by the person who puts it up ought to allow everyone to share. Worse comes to worse once it's spread a bit we could put up links round-robin to share the load. I leave my Torrent client up fulltime but right now it's not directly reachable so I couldn't be a primary seed. I intend to fix that but I could certainly lend bandwidth to this if it's deemed the way to go. <shrug>

[WeeboTech]

If I knew more about a torrent server I might be able to do this.

In the meantime, I'll see about making a development appliance.

 

My idea is to build a disk image with two partitions.

 

One for the FAT SYSLINUX parition.

One for the Slackware dev environment.

 

I was considering adding nodes or hooks for 3 ide drives, so that you could boot the development environment or unraid in a test mode.

 

perhaps I'll just start off with one drive, 2 partitions.

At least that is my thought for now.

 

 

 

[BLKMGK]

Honestly I'd try to set it up as simple as possible for a test run, particularly if larger images result from adding bells and whistles.

 

As for a server, I think you misunderstood me perhaps - no server needed. All that's needed is a reachable address and a torrent linked here. Once posted others could begin downloading it and your client will be able to tell the other clients about one another or DHT may link them. At some point the full load of the downloads should no longer be on your machine. I honestly do not know how feasible that is long term and having others post links to their machines with the same download would be a good idea as they receive the file but eventually we ought to be able to come up with something.

 

For that matter - post it to a public tracker like PirateBay. They will take a legal Linux ISO as easily as they will accept links to anything else. thinking about it - that's probably way better than my initial torrent suggestion!

 

FWIW, I've had friends create Torrents and send them to me via email so that I can DL straight from them. So long as their machine is addressable on the port specified it works like a charm! Moves in chunks so any disruption is no big deal...

[toonz]

If someone does setup a torrent for this I'll grab it and seed. I've got a 24/7 torrent box I'll put it on.

[eeking]

I'm currently trying to setup a minimal Slackware 12 development environment using coLinux. If I get it working, I'll try to share a fs image.