Help Use Cron and rsync to synch XBMC Thumbnails

[hernandito]

Hi Gang,

 

It has been a long time since I visited. I have also been away from configuring my unRAID and my multiple XBMC Live  machines. SO long that I feel like I am a complete newbie.

 

A while back I create what I think has been a popular tutorial on how to centralize your XBMC database and your thumbnails into the unRAID machine.

 

Tutorial here:

http://lime-technology.com/forum/index.php?topic=11473.0

 

Now that XBMC Eden is officially out, I am looking into upgrading my XBMC Live machines to XBMCbuntu. I have done some testing and I am liking the results. There are portions of my guide above that are now not applicable to Eden, and I want to update my tutorial.

 

BUT, I think there is one improvement to make. In my original guide I outline how to create a centralized folder for Thumbnails on the unRAID server and have all the HTPCs use it. The main complain from other is the load time once you click on Movies. For me it is about 3~4 seconds with around 750 movies (if unRAID drives are fully spinning). Not a horrible thing in a living room browsing setting, but a good place to make improvements.

 

To avoid this delay, I know that the right thing is to keep the Thumbnails folder local to each HTPC. In order to keep all those Thumbnail folders synchronized, I believe I will need rync to do the synch task, and for crontab to execute the rsync task at given intervals.

 

I need help from the real experts to achieve this. I don't have a clue how to setup the crontab task. I saw some samples on getting crontab to shutdown unRAID. I cannot make the jump of applying this example to what we need to do.

 

I THINK the command for rync is something like:

 

## local to livingRoom
rsync -avr ~/.xbmc/userdata/Thumbnails xbmc@xbmc:~/.xbmc/userdata/
## livingRoom to local
rsync -avr xbmc@xbmc:~/.xbmc/userdata/Thumbnails ~/.xbmc/userdata/

 

Can someone please help me piece this together so I can update my guide. 

 

Thank you.

 

H.

 

[Influencer]

You can make a little script with the rsync commands in it, and then add a command to crontab to run it however often you want it ran:

 

crontab -l > /tmp/cronbackup.txt ;echo "#Add Description of Cron here" >> /tmp/cronbackup.txt ;  echo "* * * * * /path/to/your/script" >> /tmp/cronbackup.txt ; crontab /tmp/cronbackup.txt

 

Thanks to speeding_ant for that, it was in an install file, much cleaner than my way of doing it.

[hernandito]

Thank you Influencer...

 

I guess the big pieces I am missing are:

 

1) How do I set up crontab to automatically execute the script every 60 minutes or so.

 

2) Is there anything I need to do to the "go" script so it launches this every time I reboot the server

 

3) In my rsync script, is there anything special I need to do so that unRAID can see each of my XBMC client machines. For example, willl this work?

 

## Bedroom to livingRoom
rsync -avr xbmc@xbmc-bedroom:/home/myname/.xbmc/userdata/Thumbnails xbmc@xbmc-livingroom: /home/myname/.xbmc/userdata/
## livingRoom to bedroom
xbmc@xbmc-livingroom:/home/myname/.xbmc/userdata/Thumbnails xbmc@xbmc-bedroom:/home/myname/.xbmc/userdata/

 

Probably using IP address is better, but, in other words, will unRAID and rsynch see the path to each of the XBMC clients as:

[email protected]:/home/myname/.xbmc/userdata/Thumbnails/?

 

4) The actual script in 3) above, can I just name it /boot/custom/thumbsynch.txt? Or do I need to make it something executable. If so, how do I do that?

 

I know some of these are basic newbie questions, I wish I did not have to ask. 

 

If there are more than 2 machines on a network, I would revise the rsync script, that one I can figure out.

 

Thanks.

 

 

[Influencer]

Actually, if you want it to run every hour, you can just save the script anywhere on the flash drive, I use /boot/scripts/ to hold mine.

 

Then edit the go file to copy the script to /etc/cron.hourly

cp /boot/scripts/myscript.sh /etc/cron.hourly/

 

I'm not sure if unraid will see the path as that, I currently do not run XBMC.

 

You will need to make it executable, if you are editing the script on windows, use an editor like notepad ++ or notepad2 to edit it so it will be properly formatted. Both will have an option to save as SH.

 

 

[Influencer]

Once you get everything running properly, you should possibly partner with someone that has had experience making plug-ins for unraid, or its entirely possible to tackle it yourself, a plug-in would be nice to point unraid/your script to xbmc, allow to enable or disable the sync, or adjust how often to run rsync.

[bigdog66]

I have not had a need to set this up yet but will in the future

 

just want to say THANKS for keeping this up!

 

and hopefully you will get all the help you need and knowing this place i have no worries about that

[hernandito]

Thank you Influencer. You pointed me in the right direction.

 

I have managed to get rsync working from XBMC Live to XBMC Live machine. For me it was not an easy feat. I believe in order to get rsync to work with unRAID, I have to enable ssh in unRAID, and it looks complicated to me.

 

What I am struggling now is in developing a strategy for the thumbnails process. It looks straight forward copying from one XBMC machine to the other via rsync. This means that Sickbeard and CP process the downloads, and adds all pertinent thumbs and metadata to library via XBMC Live Machine A. Then on XBMC Machine B, I use an rsync script to copy the Thumbnails from Machine A. If I add a Machine C, this one will also rsync from Machine A. This all excludes unRAID from the process.

 

I was considering Dropbox for my Thumbnails folder, but it is bigger than 2GB and I do not really want to pay $10/month to centralize my Thumbnails.

 

Now back to my basic command questions....

 

I managed to get ssh and credentials stuff working so the rsync command does not ask for a password.

 

I created a test file called syncthumbs.sh. I put this in my /etc/cron.hourly directory. The contents of the file are:

 

#!/bin/sh
rsync -e ssh -avr [email protected]:/home/hernando/thumbs ~/snarf/

 

I then ran the command:

sudo chmod +x /etc/cron.hourly/syncthumbs.sh

 

The command ran fine, but the command is not running hourly as I would expect. Is there anything I need to do to make the script auto execute every hour?

 

Actually, if you want it to run every hour, you can just save the script anywhere on the flash drive, I use /boot/scripts/ to hold mine.

 

Then edit the go file to copy the script to /etc/cron.hourly

cp /boot/scripts/myscript.sh /etc/cron.hourly/

 

I'm not sure if unraid will see the path as that, I currently do not run XBMC.

 

You will need to make it executable, if you are editing the script on windows, use an editor like notepad ++ or notepad2 to edit it so it will be properly formatted. Both will have an option to save as SH.

[Influencer]

Is there any output in your syslog regarding the script?

[hernandito]

Thanks Influencer,

 

I think it was having issues because the cron.hourly scripts were being run as "root" which is NOT the username used to rsynch from another machine. As root, the other machine asks for the password. So instead I simply edited /etc/crontab file and added the following line:

 

04 * * * * hernando rsynch -e ssh [email protected]:/home/hernando/Thumbnails/ ~/Thumbnails/

 

So this is a good schema for unRAID with XBMCbutu or XBMC Live HTPCs. Basically unRAID hosts Sabnzbd, Sickbeard and Couch potato, it hosts the MySQL database for XBMC, and all the media files. Sickbeard and Couch Potato update a single XBMC machine on the network, and the other XBMC machines rsynch the Thumbnails from that one machine. unRAID is not involved at all with the Thumbnails management isssue.

 

I now want to be able to help others that have a mix of XBMCs running on Win, Mac, and AppleTV. I am simply not sure how one would get these synchronized. I would love to get feedback suggestions on this so we could have an informative tutorial for others.

 

Is there any output in your syslog regarding the script?

[Influencer]

Everything but windows should be pretty straight forward, all can use rsync. Windows you have to use some combination of cygwin and rsync, there are packages with minimal install of cygwin and rsync, but you could always write up a quick howto on setting it up if you'd wish. There are plenty out there already!

 

The XBMC wiki gives the path to thumbnails for all OS's. Shouldn't be too hard to compile the information, and I'm sure there are users here that would be willing to give it a try.

[aptalca]

in windows cwrsync works pretty well for me

[Influencer]

I've heard good things about cwrsync, you just have to stick to version 4.0.3 on sourceforge as recently the developer monetized the program and is selling it on his website.

 

Atleast he was gracious enough to leave it up the latest version for free while making any updates a paid solution. Kudos to him

[hernandito]

in windows cwrsync works pretty well for me

 

Thank you guys,

 

I will test it tonight....

[iceman1000]

Hi there!

 

I am on that same path! I tried the thumbnails synchronization with symlinking the thumbnails cache on a network share to share thumbnails from win 7 to xbmcbuntu, it worked perfectly!

 

Thanks to hernandito for that great tutorial!

 

What I have found though is that it is a little slow on my client machines and even on my win 7 pc which holds the mysql database. I think it is due to the thumbnails cache being on the network share and not locally.

 

So I searching through the boards to find an automatic synching system that syncs my local thumbnails cache from my "master machine" to a network share and from the network share to the clients. What I have seen rsync and cwrsync seem to be a good solution. Though I didn't have the time to try it yet. Did you guys make any progress on this? Especially the sync from xbmcbuntu to a win 7 machine?

[brimnac]

I know this is a *real* old thread in the world of computers, but did anyone get anywhere with this?

 

I have the shared thumbnails working now, and on three out of the four XBMCs I have running it works fine (they're HTPCs or full-fledged computers, so they have horsepower to spare). I have an Apple TV 2 I'd love to get off the shared thumbs, though, if possible.

 

Thanks for any info, no biggie (with Frodo coming out) if there isn't a reply; just wanted to check it out.