Dupe Files

[JarDo]

I have ls -R running every minute and I have it wrapped in a script that stops the per-minute job once duplicate files are identified and I also have it sending me an SMS when this occurs.

 

Well, yesterday I added some files to a disc (I didn't realize at the time) that duplicated some files on another disc (in the same share).

 

I'd think that it would be minutes before the dupe's were identified and I was notified, but it wasn't until today (a day later) after my server crashed... a hard reboot... and the start of a parity check, that the dupes were identified and I was notified.

 

I'm trying to imagine why the lag.  Did it have something to do with the reboot or the parity check?  Does anyone have any ideas?

[RobJ]

Could you attach the scripts that are involved, perhaps someone will be better able to identify the issue.

[JarDo]

This is the script that installs the cron job.  Notice that the cron will execute another script every minute:

 

#!/bin/bash
crontab -l >/tmp/crontab
grep -q "frequent ls" /tmp/crontab 1>/dev/null 2>&1
if [ "$?" = "1" ]
then
    echo "# frequent ls to keep directory blocks in memory:" >>/tmp/crontab
    cp /boot/custom/bin/frequent_ls.sh /tmp/frequent_ls.sh # Copy script to ramdrive
    echo "* * * * * /tmp/frequent_ls.sh 1>/dev/null 2>&1" >>/tmp/crontab
    crontab /tmp/crontab
    echo "frequent_ls - CRON Job Added to "ls -R" every minute"
fi

 

This is the script that is executed once per minute:

 

#!/bin/bash
#http://lime-technology.com/forum/index.php?topic=2306.msg17809#msg17809
# Edit as needed, if user shares not used, then list /mnt/disk1, /mnt/disk2, etc.
SHARED_DRIVE="/mnt/user/Media /mnt/disk6"
DATESTRING=`date '+%b %d %X'`
if ! grep "DUPLICATE FILE Email Sent" /var/log/syslog >/dev/null # Only send one notification then halt ls
then
    DUPECOUNT=`grep "duplicate object" /var/log/syslog | cut -d" " -f8- | sed -e "s/^\/\([^\/]*\)\/\([^\/]*\)\/\(.*\)/ls -l \/*\/*\/'\3'/" | sort -u | wc -l`
    DUPEOBJECT=`grep -m1 "duplicate object" /var/log/syslog | sed -e 's/^.*: \///'` # We're sending and SMS so only show the first duplicate found.
    if [ "$DUPECOUNT" != "0" ]
    then
        # Send SMS Notification
        echo "unRaid Share Contains Duplicate File: " $DUPEOBJECT | /boot/custom/bin/sms.sh --rcpt [email protected]
        echo "$DATESTRING UNRAID frequent_ls: DUPLICATE FILE Email Sent" >> /var/log/syslog 
    else
        ls -R $SHARED_DRIVE 1>/dev/null 2>&1
    fi
fi

 

NOTE: sms.sh is a modified version of mikep's email script that sends a shortened email because SMS messages have size restrictions.