trurl

In v5, if you make the folder part of a cache-only share it won't be moved even if it doesn't start with a dot (.)

unRAID has a RAM filesystem where all of its operating system files are kept. When unRAID boots it unpacks all of its operating system files from bzroot and keeps them in RAM. Since bzroot is not altered to reflect any changes to these operating system files, they will all be restored to the original from bzroot when you reboot. This is why anything stored in any of the usual linux operating system folders does not survive a reboot. In general, anything not in /boot (your flash drive) or /mnt (actual disks or shares on those disks) is in RAM. Anything stored in these locations will persist after reboot. This is why some people add code to the go script to alter operating system files every time they reboot. I have code in my go script which alters /etc for example. You can put those packages in /boot/extra and they will be automatically installed on boot. Add the other script to the script in /boot/config/go (like I have just done. Thanks for this)

Not sure I understand your problem. If you want guests to be able to read/write a share make it Public. If you want guests to only be able to read a share make it Secure. If you don't want guests to have any access to a share make it Private. You can also make shares hidden so it can't be seen when you browse. The only way to get to these is to specify the path. Are you anticipating guests accessing your server from: One of your computers using your Windows user? Using one of your computers using a separate Windows user? Using their own computer using their own Windows user? Some other scenario I haven't imagined? Where is a workaround required?

I have used every scenario Tom describes here, including referring to the server by IP to get a different login to my server. I can confirm that it all works just as described for me.

I've been using this for pre-clears since my case is full and I have one sata port left over: eSATA enclosure This particular one even came with an eSATA bracket

Go back to page 73 of this thread for more discussion of these parameters. I was trying to do 3TB. Increased -b parameter is what you need. Did that but did not find anything concerning a -b parameter... Since there does appear to be a relation with these parameters (I conclude this from your reply) I will take my chances and do the preclear without the extra parameters, if only to see if that makes the difference.. Sorry, wrong thread. Try this instead. When I tried it on 3TB with the defaults (no parameters) I ran out of memory. The originally suggested -b 200 parameter was too conservative and caused it to be very slow. -b 256000 was suggested and I stopped mine during post-read and restarted it like this.

Go back to page 73 of this thread for more discussion of these parameters. I was trying to do 3TB. Increased -b parameter is what you need.

The emails are sent to the same address the system notification emails are sent. This can be setup in the simplefeatures email notify plugin. There is also an unMenu package but I don't use that method anymore since I upgraded from 4.7

Working for me. At first I tried to increase the size but I closed the page before it finished and that didn't work so I tried again and waited for it and it works. Thanks

What Joe said. Linux "screen" lets you reconnect if the session dies. Preclearing With Screen There is also a plugin.

Longer answer: Because the OS is running completely in RAM (this includes any "files" in the usual places, such as /etc/sudoers) any changes you make in the running OS is lost when you reboot. To make a change that will survive a reboot, you need to make the system make those changes for you each time it boots. This is usually done in /boot/config/go script file which is run automatically after the system boots.

How would it anticipate a file being opened?

I ran the new version twice this afternoon and can report that log packaging and cleanup is working as intended. Both times rsync completed without errors so I haven't really tested what happens when it fails but at least it looks like I managed to get all of my filenames cleaned up. After the first run I didn't get an email so I looked at the script again and noticed that there was no email for success (I just love getting notifications from my unRAID) so I put in an email on success before the second run and that worked too. One thing that didn't work was Dropbox. After each run I waited for the log to show up in my Dropbox but it was stuck on the unRAID side. I think Dropbox had probably crashed due to OOM or something. A lot of OOM going around these days as people try to squeeze more apps into their server. I tried to restart Dropbox from the webGUI but that resulted in SAMBA crashing as well and I had to reboot. I didn't bother to capture the syslog. I don't know if the size of the rsync log (about 16MB on each run) had anything to do with this or not since I am not really clear on all this Linux memory management, low memory, swap, etc. I do have a swap file and 4GB RAM. I will investigate further if it continues to be a problem.

OK. I have been playing with this some more today. Ran it a few times while trying to fix the filenames. The first thing I tried was fixing the filenames in my source files. Then I noticed that it seemed to have created some of the bad filenames on my destination so I deleted all of those. The last run it seemed to complete without error. I think at least some of the errors may have been due to trying to do the compare on the NTFS volume rather than the copy from the array. I tweaked the script a little with some copy/paste so I could get an email and a log package on both success and fail. I am running it again to test. I also set my logdir to my Dropbox (thanks for the plugin) so I could access it after I got the email. It is appending the log rather than creating a new one and that was a little confusing at first as I was trying to fix some things I had already fixed because they were coming up in a search of the log. Also, the log is still hanging around in /tmp after completion and it can get large. I deleted it from /tmp and the old logs in my logdir. Might need some more work to get it to quit appending and clean up the old log in /tmp. rsync_backup_to_external.sh.txt

What version of unRAID are you using? 5.0rc8a What is your unRAID locale set to? (issue "locale" from command line) That command gives a bunch of lines but they are all set to "en_US.UTF-8" What is the method/command you're using to mount your Offsite disk to /mnt/user/Backups/Offsite? It has been mounted for several days so I don't have that command line handy anymore but I took it from this wiki page: Mounting an external USB drive having an existing NTFS file system in READ/WRITE mode to transport files from/to unRaid server so it must have been something like: mkdir /mnt/user/Backup/offsite mount -t ntfs-3g -o umask=111,dmask=000 /dev/sdc1 /mnt/user/Backup/offsite Thanks for the new version. The previous one did indeed only do dry-run. I don't entirely understand the code but it looks like the log only gets packaged if rsync is successful. In any case I had to go find it in /tmp instead of $logdir/$logname. Anyway... After staying up way too late beating my head against the wall I have determined that there are only a few files affected by this problem (thanks to the log). There are plenty of other files (my wife has some "world" music in her iTunes) with "foreign" characters that are not affected by this problem. It seems the main thing the problem files all have in common is that they were all downloads and so were originally created on some other system. I have been trying to rename them from Windows but that doesn't seem to work very well. I even tried to retag one of the files in MediaMonkey and that didn't work either. Even though these filenames are displayed "correctly" (whatever that means in this context) in Windows, Windows does not seem to be able to tell unRAID what file it is talking about when it asks unRAID (or SAMBA) to manipulate the file. I am going to try renaming these few files from the unRAID (linux) command line in PuTTY, then I will try the new version of the script. Thanks

I ran your new script with dry-run="" verbose="true" email="true" I go the error email again with the return code this time: Error: rsync exited with non-zero return code (23), please check backup status. The log file did not appear in the place I had specified. I found it instead in /tmp Looking through the logfile I found several files were not copied. It is the same problem I had before. 2012/12/10 21:12:43 [5846] rsync: recv_generator: failed to stat "/mnt/user/Backup/offsite/Rick/Music/Lossy/Steve Hackett/Momentum/12 ~ Bourée.mp3": Invalid or incomplete multibyte or wide character (84) When I try to ls this file in PuTTY it comes out as 12 ~ Bour?e.mp3 Apparently some of the filenames with special characters are copying OK and others are not. When I thought my problem had gone away it was just because I had not chosen the right example to test. Don't know how these got named in the first place. I will see if I can rename them to something that works and try again.

My offsite backup using your script completed. It looks like it successfully did all 830GB in two of my user folders, which is what I intended for it to do. For some reason it sent me the "rsync exited with a non-zero return code" email and I don't see any log file. Here is the "user variables" section in my script. This is the only section I made any changes to. ###################################################################### # DEFINE USER VARIABLES ###################################################################### # set global date var logdate=$(date +%d-%m-%y) # define the source location. must end in slash to work correctly source=/mnt/user/ # define target backup location target=/mnt/user/Backup/offsite # define rsync include file name / location rsyncincludes=/boot/scripts/rsync_include_list # define log dir logdir=/mnt/user/Backup/logs # define log name as composite of static text and logdate variable logname=rsync_offsite_log_$logdate.tgz rsync_include_list is Mary/*** Rick/*** I'm not up to speed yet on bash scripting. /mnt/user/Backup is a share where I put backup images. This is where I did my ntfs-3g mount and also where I pointed my logdir. It is on a separate disk from the user shares in my rsync_include_list. None of these shares are cached. Let me know if you would like me to try anything else. Thanks

That is the ntfs-3g I have always used as well. I just tried this with some of my existing files that had been giving me problems and it worked for me too. It's been a while since I last tried this and I have made a lot of changes but none that I thought should have fixed the problem I used to have. I upgraded unRAID from 4.7 to 5.0rc8a and installed a lot of plugins but it's not obvious that has anything to do with it. The only other change that seems like it might be relevant is I changed the translation in PuTTY to utf-8 from its default of ISO-8859-1. I say this because translating these was exactly what I was coming up with in my research of this problem, but relative to ntfs-3g and rsync, not relative to PuTTY. I thought translation setting in PuTTY would only affect the way things got displayed, not the way they worked. I don't know but it looks like my problem might be solved. I will try a full offsite backup using your script and see what happens. Just started your script and it appears to be working beautifully. My offsite backup will be about 500GB so won't know until later if my problem files get transferred or not. Thanks again.

Thanks. I have been using rsync to backup critical files on eSATA mounted ntfs-3g which I rotate offsite monthly. Your script will help make this better. Look forward to trying it and any further development. One problem I have run into trying to use ntfs-3g for this is incompatible characters in filenames. I have some music files with accented or other "foreign" characters in them that were put on unRAID from Windows and they work just fine with Windows but rsync won't copy them. I have done some research on this and it looks like there may be some options in rsync or possibly mount that might address this but I have not actually gotten around to trying anything out yet. If anyone has any experience with this please respond.

I had even put the -? output in a file so I could study it. Don't know how I missed that option. Post-read 2% complete in <18 minutes, so about 15 hours to go instead of 30. Thanks

Thanks. I'll keep that in mind for next time. Finished writing zeros and am now on the post-read, 41+ hours so far. I think 30+ of that was all pre-read so I guess I can expect another 30+ for post-read. Doesn't seem to be an option to let me start over and only do post-read using the larger -b, but I guess I will just count it as further burn-in/testing.

Well I am currently preclearing the disk again using preclear_disk.sh -w 65536 -r 65536 -b 200 /dev/sdX and it took more than 24 hours just for the pre-read. Now it is writing zeros (11% done). Interestingly, this step seems to be going a lot faster than the pre-read. I did not disable any of my plugins so we'll see if it crashes again but so far I can still get to the webgui. Here is fdisk -l output: Disk /dev/sde: 3000.6 GB, 3000592982016 bytes 255 heads, 63 sectors/track, 364801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sde1 1 267350 2147483647+ 0 Empty Partition 1 does not end on cylinder boundary. Partition 1 does not start on physical sector boundary. I guess with 16GB anything is possible.

There was a discussion over on another thread about running out of memory while running preclear. This happened to me yesterday. It is my first time trying preclear on a 3TB disk. I think it probably finished the pre-read but sometime later the webgui was unresponsive. I couldn't get powerdown to finish until I killed dd. After rebooting everything seems fine so I started preclear again using the parameters in Joe L.'s post above but it is only 35% finished with pre-read after nearly 12 hours. I am sure the real problem is all the plugins I had running. I have 4GB memory. If I disable my plugins should I be able to preclear 3TB using the default parameters so I can get this done sooner? Are other users able to preclear 3TB using the defaults without running out of memory?

Would a swap file help with this? I have 4GB memory and don't usually run with a swap file because creating it seems to be one of the longest parts of boot time. But it might be worth it for those rare occasions when I need to do a preclear.

I went ahead and registered at yamaha just so I could read your receiver's manual. On page 53 of the English pdf it mentions playing from a NAS with DLNA. There is a DLNA package for unRAID. You won't get the iTunes user interface this way, but it should work. On page 58 of the English pdf it shows that you can play music from an iPad with iOS 4.3.3 or later if it is connected to the same router as your receiver. So, if you can work out how to play music on your iPad from your unRAID then you probably can play it on the receiver using the iPad as the player.