How does unRaid handle duplicte files?

I ask because I seem to be getting many of them.

I always transfer files to the share ie /movies but allow the cache disk to be used on that share.

 

I am currently moving all my files on my servers onto unraid, but every now and then I get duplicate file errors, and then my transfers seem to slow down to a crawl.

 

Here is the syslog.

I don't use User Shares myself, so I'm not sure I can help.  It does look like the User Share system was overwhelmed here, plus you swamped the Cache drive.  I would reboot so that the User Share system can reset itself.

 

When loading a large number of files, more than can fit on the Cache drive, it would be better to temporarily turn off the Cache drive.

 

If this is a first time load of your unRAID server, it is fastest to disable the Parity and Cache drive, the once loaded, re-assign the Parity drive and build parity.

I dumped 1.5TB of data across my network and I wish I would've turn off parity check the first time. It took me 12-14hrs to do this on a 100mb network. See ROBjs advice if this is your initial load.

 

You won't lose any of your data, you'll be just fine.

 

I thought of doing that, but since I want to do a move and forget, I need the security.

I guess for now I will set all the files to access 777 through chmod and force overwrites.

I know I have some duplicate files. This way the transfers should keep going.

 

BTw I tried using beyond compare in the past to clean up the dupes, as well as terracopy.

I even tried some freeware sync software, all with mixed results.

 

Thanks for the info.

 

Did you guys spot any other errors in the logs?

Did you guys spot any other errors in the logs?

 

No, not really.  There was evidence of an improper shutdown on the previous boot, so a few transactions were replayed.  Disk 9 was added and cleared, then Disk 7, Disk 8, and Disk 9 were formatted, then a parity check started, also because of the previous shutdown.  There are a few of the 'too many iterations' message, but that is typical of the forcedeth driver, used by nVidia onboard NIC's, and usually happens under heavy network traffic, has always appeared to be harmless in mine and other systems.  The Cache drive is too small for your current usage, but as I recommended, just disable it for now.

Yes, I have a 750G and a 1TB drive that were RMA'ed. It should be back by friday!

Are all my sata drives seen as sata 2 3Gb/s?

 

No, but they ARE configured as best as they can be.  The 2 Seagate 400GB's (ST3400832AS) and the Maxtor are older SATA150 drives, so all they can do is 1.5Gbps.

Check the size of you syslog:

 

ls -alh /var/log/syslog

-rw-r--r-- 1 root root 6.8M Jan 19 12:14 /var/log/syslog

 

When unRAID detetcst dupes and your doing alot of file listings it is not smart enough not to stop adding the error. This also happens ALOT when using mc which copys files then deletes original. If you move 10,000 files using mc and do say 5 "ls" while doing it unRAID will add 50,000 log messages.

 

I am rationalising my drives this now and my syslog grew very quickly to nearly 2GB and things started to crawl

Yeah, the log is at 771K.

 

root@Tower:/mnt# ls -alh /var/log/syslog

-rw-r--r-- 1 root root 771K Jan 19 23:54 /var/log/syslog

 

I think as time goes on, I will slowly adopt a system that is 100% compatible (no nvidia chipsets).

For now it works, and once all my data is moved over, I will not need to do bulk loads from many sources at the same time.