April 18, 200818 yr I'm not at the moment, I'm waiting to see how the final comes out in regards to the cache drive. Once the dust settles, then we may have a place to install a mysql server without it affecting memory too much. Another thing I've been looking at is the gigabyte ramdisk. http://www.gigabyte.com.tw/Products/Storage/Products_Overview.aspx?ProductID=2180 With this we can have a 4GB memory based disk that survives a power reset. I know there will be allot of balkers about this. My concern is another constantly spinning drive on the network. So with this I plan to move my mysql server to it, plus a bunch of other services thereby shutting down a few spindles in my network. Also exploring root over NFS and using this chunk of memory to hold the remote roots. So we'll see how the next few weeks flush out.
April 19, 200818 yr Another thing I've been looking at is the gigabyte ramdisk. http://www.gigabyte.com.tw/Products/Storage/Products_Overview.aspx?ProductID=2180 The info says it acts like a regular SATA drive at 1.5 Gb/s. Does this mean that a 3.0 Gb/s "real" SATA drive would be faster? Or would this ramdisk get closer to the theoretical limit of the interface and maybe be faster that a 3.0 Gb/s drive? I guess another question would be whether you're looking at this for speed performance or for energy efficiency? Also, NewEgg lists it as a "Deactivated Item." What's up with that?
April 19, 200818 yr The ramdrive itself saturates at very close to the 1.5gb/s rate. It's very fast, but it's all dependant on the the type of writes going to the drive. My goal is to make this ram drive available for a mysql server and mountable segments for NFS. I plan to migrate some of my machines to use their root filesystems over NFS. One reason NFS is considered slow is writes are done synchronously. They say for NFS mounted volumes having a caching controller helps NFS out tremendously. So having a NFS mounted root system that is a ram drive should provide enough of a performance boost to make this worth while for me. With the mysql server in my log host environment, messages are getting written to the DB and filesystem very often. I have all the syslogs on my machines forward thier messages to once central log host, which then also inserts the messages into a DB for review later. In addition NAGIOS logs allot of information constantly to the hard drive over and over. By using this ram disk for these services I can minimaze spindles and writes while improving effeciency of the services. It's really only for specialized applications, temp disk, game storage, pagefile disk. swap file or swap filesystem, etc, etc. I'm also considering putting 8GB into my ABIT motherboard and recompiling the kernel to support a large amount of memory. If I can get it to work then I can create a TMPFS filesystem in ram and put these services there. The only caveat is persistance, with the ramdisk I have more persistance, of which I can periodically rsync the filesystem to a real spindle for backup. Hope this sheds more light.
April 30, 200818 yr Author ...so to get back to the topic (now that I am using unRAID full time *AND* found out that indeed I would love to move torrent jobs to this machine too)... Any real (and viable for a Linux "basics" man like me) proposal? Tonight I leave two machines on, where I could only leave one.
May 4, 200818 yr The issue is not the basic install of this. It's the whole "REINSTALL" of it every time you boot up. Man that could be allot of work. I'm going to work on another ftw tool after ftwd is done. The second one will be ftwdup. It will take a source tree and a destination tree. It will then make directores on the destination that do not exist but exist on the source (duplicate the directores). It wil then make symlinks to the source files on the source tree on the destination tree. This will allow you to install applications to a disk, then link them back into root Thereby saving some of the config/bin and setups of the applications you install. Eventually you will be able to install perl,python,php,apache,mysql to one of your disks and leave it there. I plan to use this with my gigabyte ramdisk (then periodically rsync it to a compact flash). This is backwards from what JosetAnn has done where he dropped unraid onto a slackware distro. I'll work on symlinking an installed package from outside of / into / so that they will work. Not sure how far I'll get. In the meantime would a console/telnet application suffice?
May 4, 200818 yr Author Yes. It will have to suffice. It is the difference of leaving one or two PC on, all the time. Thank you man.
May 17, 200818 yr Author ...please? (have various problems using a windows client on an unRAID shared torrent folder - must move the whole thing locally, either way and of course I prefer unRAID-way)
May 18, 200818 yr Well I have ctorrent and rtorrent compiled. I was able to make ctorrent a static binary so many of the source libs are not mandatory. I was almost able to do this with rtorrent but it came out to almost 26MB. Ugh! In any case with rtorrent you need a number of llibraries plus screen so you can detach. -rw-r--r-- 1 root root 376583 May 18 13:26 ctorrent-1.3.4.dnh3-i486-1.static.tgz oot@unraid:/var/log/packages# ldd /boot/custom/bin/ctorrent /usr/bin/ldd: line 102: lddlibc4: command not found not a dynamic executable root@unraid:/var/log/packages# ls -l /boot/custom/bin/ctorrent -rwxr-xr-x 1 root root 876619 May 18 10:44 /boot/custom/bin/ctorren -rw-r--r-- 1 root root 99700 May 18 13:26 ctorrent-1.3.4.dnh3-i486-1.tgz This is the shared library version and it's dependencies root@unraid:/var/log/packages# ldd /usr/bin/ctorrent linux-gate.so.1 => (0xb7fb2000) libssl.so.0 => /usr/lib/libssl.so.0 (0xb7f6e000) librt.so.1 => /lib/librt.so.1 (0xb7f66000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7e82000) libm.so.6 => /lib/libm.so.6 (0xb7e5b000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xb7e4f000) libc.so.6 => /lib/libc.so.6 (0xb7d0d000) libcrypto.so.0 => /usr/lib/libcrypto.so.0 (0xb7bce000) libdl.so.2 => /lib/libdl.so.2 (0xb7bca000) libpthread.so.0 => /lib/libpthread.so.0 (0xb7bb3000) /lib/ld-linux.so.2 (0xb7fb3000) These are the standard packages that will have to be installed to satisfy dependencies. -rw------- 1 root root 694999 May 6 2007 curl-7.16.2-i486-1.tgz -rw-r--r-- 1 root root 903354 Jun 27 2007 cxxlibs-6.0.8-i486-4.tgz -rw-r--r-- 1 root root 402093 Feb 26 16:58 libidn-1.5-i486-1.tgz These We'll have to work it out so I can provide them. Not sure how to seed, so if someone can tutor me. -rw-r--r-- 1 root root 1896281 May 18 13:07 libsigc++-2.0.18-i686-1as.tgz -rw-r--r-- 1 root root 3256928 May 18 13:07 libtorrent-0.11.9-i686-1as.tgz -rw-r--r-- 1 root root 344139 May 18 13:06 rtorrent-0.7.9-i486-1_SBo.tgz These are so you can detach from the screen -rw------- 1 root root 574993 Apr 10 2007 screen-4.0.3-i486-1.tgz -rw-r--r-- 1 root root 15394 Feb 7 2007 utempter-1.1.4-i486-1.tgz root@unraid:/var/log/packages# ls -l /usr/bin/rtorrent -rwxr-xr-x 1 root root 827644 May 18 12:42 /usr/bin/rtorrent* Here is rtorrent's dependancies for working in a shared library environment root@unraid:/var/log/packages# ldd /usr/bin/rtorrent linux-gate.so.1 => (0xb7f5d000) libncursesw.so.5 => /lib/libncursesw.so.5 (0xb7f0d000) libcurl.so.4 => /usr/lib/libcurl.so.4 (0xb7ed4000) libidn.so.11 => /usr/lib/libidn.so.11 (0xb7ea4000) libssl.so.0 => /usr/lib/libssl.so.0 (0xb7e64000) libdl.so.2 => /lib/libdl.so.2 (0xb7e5f000) libz.so.1 => /usr/lib/libz.so.1 (0xb7e4b000) libtorrent.so.10 => /usr/lib/libtorrent.so.10 (0xb7dac000) libcrypto.so.0 => /usr/lib/libcrypto.so.0 (0xb7c6d000) libsigc-2.0.so.0 => /usr/lib/libsigc-2.0.so.0 (0xb7c67000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7b83000) libm.so.6 => /lib/libm.so.6 (0xb7b5b000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xb7b50000) libc.so.6 => /lib/libc.so.6 (0xb7a0e000) /lib/ld-linux.so.2 (0xb7f5e000) I was able to get libtorrent, libsig as static, but the executable was huge. Now I always install openssl libs so I cannot comment if others have them installed or not. this could be a dependency also. I cannot get ctorrent to run in the background no matter how hard I try. So if you want to detach, you will need screen or the dtch command.
May 18, 200818 yr Author I will most probably need to detach - except if I "lock" my main terminal (easily accessible through KVM switch) with this task, which is what will happen if the ability to detach is too complex. Why seed? Don't you have an FTP? (else I can give you an account on one of mine) Ah you mean so you can test? Better download a .torrent file (choose something small or here is a chance to download a whole other linux distro you may have missed ), wait for it to download fully and hopefully the default action of the torrent program will be to seed it from then on (at least for some hours). Shoot me with testing material.
May 18, 200818 yr Send me a pm with an ftp account and I'll upload a few modules for you to work with. Read up on these http://kmandla.wordpress.com/2007/05/02/howto-use-rtorrent-like-a-pro/ http://www.rackaid.com/resources/linux-tutorials/general-tutorials/linux-screen.cfm I'm also playing wit BitFlu which is in perl. Only required one module which was only a slight pain to install. the real pain is configuring the daemon. What I like about this one is it has a telnet accessible port AND an http accessible port.
May 18, 200818 yr Before you guys put too much work into making clients work there is one serious consideration to take into account. There are 2 kind of torrent sites namely public and private. On a public swarm you can use whatever you want client wise but on most private sites you probably wont be able to use anything but rTorrent or Azureus (and many wont allow that one). The reasons why are pretty off topic but just keep this potential limitation in mind.
May 19, 200818 yr Following up with this. rTorrent has some pretty cool features and seems like it may be the way to go until we have a more advanced webserver for the environment From what I'm seeing, rTorrent has an SCGI interface which allows remote programs to make calls to it. Furthermore there are a number of web and gui apps out there. Some java based for the local browser. Most seen to need an advaned webserver lighthttp/apache with SCGI and PHP (some need python). In any case there is also a "watch" mode. So you can set a share to watch a directory for new .torrent files. You drop a file there and it will queue it up. Phew I'm tired, just spent the last few hours recompiling everything to minimze lib dependancies and building a SlackBuild package for XMLRPC-C. I searched all over but had to write my own I'm so lazy.. More to come as I learn.
May 19, 200818 yr Author thanks man - ftp pm in a couple of hours (ah the time difference - you mostly work when I sleep and the opposite ) (btw, how easy is to setup ftp included in unRAID to be actually usable as an externaly accessible ftp?)
June 23, 200818 yr Revisiting this thread. Although I have not been around much lately, I've been working with libtorrent, rtorrent, ntorrent and wtorrent. These are. libtorrent - The basic library for torrent communciation. Many tools use this. rtorrent - The nCurses / console torrent client. ntorrent - A JAVA front end to talk with rtorrent wtorrent - a php front end to talk with rtorrent. I choose rtorrent because it has the best of 3 worlds of control for users. console(screen), an XMLRPC api for other applications to grab data and communicate with it. These are ntorrent and wtorrent (there is also rtgui and a few others). Point being here, thecus uses this also. So far, rtorrent has been working well for me. It takes a bit of getting used to, but once you master it, it has allot of information and control I have 2 packages so far. rtorrent with XMLRPC 1.06.27 in a semi static state. Note the dependencies for rtorrent communication is a bit hefty. This provides a working console package with a "few" required libraries. The one issue with this version of xmlrpc is... it just doesn't work. I could not get it to function no matter what I did this weekend. So I had to recompile and rebuild with XMLRPC-1.11.00. This has to be used in a shared library configuration. This expands the library requirements. Ugh.. In all there are 7-10 packages required to get this all running. This is without any gui interface. This weekend I was able to get the gui's working, adding even more dependancies. 1. web server with SCGI ability. - I choose lighttpd because scgi is already built in. apache has the ability with an external module. I did not bother at the moment. 2. nTorrent - The cool thing with this program is it's java based, so your front end can run anywhere. It communites over http with SCGI/XMLRPC to rtorrent. It can also communicate over an SSH channel. (I believe this eliminates the need for the webserver). 3. wTorrent - this is a php app which communicates to rtorrent over http/scgi. it works. My only issue with this is, initialization on install. I have not given too much effort on this part. (actually none). I just installed it, ran the install.php, filled in the blanks and was done. The only issue here is that it may need to be done upon every reboot. I have not looked further into this. For those wondering why I bothered posting this information... Well.. We need to come up with some form of agreed upon directory structure. Right now I have a torrent share (Which I think will be the minimum requirements). I plan to require an rtorrent user and a torrent group. (this will be installed by the pacakge). It's home directory should be the torrent share. It will probably have to use the /mnt/disk? naming as the user filesystem does not support ownership management. I plan to use screen to start up rtorrent upon boot up. The cool thing about this is, you can fire up the console program, it's screen is virtualized. Then you can telnet in from anywhere and do a screen -x rtorrent and get a copy of the screen while it's running. I may even set it up so the screen is automatically displayed on one of the virtual consoles. So far things have been reliable with the console application. I'll update more as I can. For those wanting an even simpler approach. I've compiled ctorrent in a static mode. This is a command line torrent downloader that handles a single torrent file. It could not be any simpler then this. I'm using rtorrent, What I like about the rtorrent is it has these scheduler controls which let you drop a .torrent file in a directory and it will automatically start picking it up. After seeding for your ratio, it can remove the torrent file and you will be done. I looked at allot of different torernt clients and chose this one for it's multifaceted control. Another I was able to get running with minimal fuss was bitflu. it only depended on Perl and the CPAN module SHA1. This is another option for those that have to have a gui. One of the things that will have to be discussed is how to share these files among us. We need a repository or tracker to begin with.
June 23, 200818 yr I thought I'd chime in here because I'm basically trying to do the same thing as you guys are. WeeboTech, are you running rTorrent on user shares? The problem that I'm running into is that whenever it tries to load a torrent I get: Storage error: [File chunk write error: No such device] After doing some digging, I found http://libtorrent.rakshasa.no/ticket/226, which basically says that rTorrent will not work until FUSE supports mmap (which has been added in kernel 2.6.26). When I get home tonight, I'm thinking I'll try booting my unRAID box with that kernel to see if rTorrent will work for me. I really think it's the best client for Linux there is.
June 23, 200818 yr I had and saw the same issue. I do a chdir directly to /mnt/diskx/bittorrent. Then spawn my rtorrent client. How far have you gotten with building the rtorent package? I got it working with XMLRPC/lighttpd and a few clients. it was a bit of a bear getting all the packages together. The wtorrent interface will be the hardest part.
June 23, 200818 yr Building it wasn't actually too bad. I have a Slackware 12 environment that I used to build my own libtorrent and rtorrent packages. As for prerequisites, I installed the following: curl-7.16.2-i486-1.tgz cxxlibs-6.0.8-i486-4.tgz libidn-1.5-i486-1.tgz libsigc++-2.0.18-i486-1.tgz I was able to find all of those packages from the various Slackware package sources. I've also got screen installed too. For me, I'm not too concerned about having a web interface, so I didn't try compiling with XMLRPC support. If anyone is interested, I can post the rtorrent/libtorrent packages I built on the interweb.
June 23, 200818 yr Author I really hope you guys take it as far as needed for us "basic" users to be able to install it and use it. Sorry Weebo for not trying your first build (things really got messy the last few weeks here). I even have a shortcut on my desktop with the files you sent me. I guess I shouldn't bother as you have most probably gotten way further. So, both you guys have a much more complete Slackware environment than normal unRAID. Need some guy to tests things out on a basic one? Or should we give you some more time to make this more "press button and it works"?
June 23, 200818 yr It's actually not that easy to make it easy for basic users. There's a bunch of piieces to set up. i.e. user, directories, shares, .rtorrent.rc The rtorrent client itself is quite powerful and has a larger the average learning curve. The ctorrent client is so very basic anyone could use it in a split second. however this does only one at a time so you must multiplex it with screen. Then you have the screen learning curve. Right now I'm just testing for reliability. Which is going well. If I release anything it's in console only format, Which would be as planetscott has set up. The whole xmlrpc/lighttpd/php/scgi interface makes it much more complex. I don't want to go to far until I figure out how to set up lighttpd to proxy. In the meantime people can review the following url for keyboard commands http://libtorrent.rakshasa.no/wiki/RTorrentUserGuide
June 23, 200818 yr I've used torrentflux for years. The recent version is the easiest to set up yet. I like it.
Archived
This topic is now archived and is closed to further replies.