File List Generation - By me!

[MadsRC]

Inspired from the other thread about File List Generation.

 

NOTE: Please note that this is an alpha! - Use at your own risk!

Version: 0.1

 

Future plans

• Support for cache drive (Write list to cache instead of flash)
  
• Support for all drives
  
• Proper plugin - Control the script from plugin
  
• Scheduled runs through plugin
  

About

Currently it will generate 2 lists:

[*]List of paths to all files

[*]List of paths to all files including the MD5 sum of each

 

These files are saved to /boot/filelist/

When I'm done developing this, it will save to a cache drive, to spare the flash. Currently I only own unRAID basic, which is why it doesn't support a cache drive yet.

 

Currently it supports the following switches/arguments:

• --flash : Will create lists of files on the flash drive
  
• --disk1 : Will create lists of files on disk 1
  
• --disk2 : Will create lists of files on disk 2
  

Installation

Note:The script have only been tested with python 2.7.3.

• Download and extract the zip file
  
• Install the python plugin
  
• Place the filelist.py script in /boot/
  

Execution

Execute the script with ./filelist.py --flash (or --disk1 or --disk2)

Tip: Run it in screen - It will run for quite some time...

madsrc_plugin.zip

[Automatic]

You do realize this is very easy to do just with the "find" command?

 

Disk one files only:-

find /mnt/disk1/ -type f

 

Disk one directories only:-

find /mnt/disk1/ -type d

 

Disk one all objects:-

find /mnt/disk1/

 

/etc

 

Then you can also start matching files between folders to check for duplicates over multiple disks:-

 

find /mnt/disk* -type f | sed "s/\/mnt\/disk[0-9]\///" | sort | uniq -d

 

Which will:-

 

find

 

all

-type f

files located at

 

/mnt/disk*

 

and then

sed "s/\/mnt\/disk[0-9]\///"

remove the prefix of "/mnt/disk{number/" to finally

sort

 

sort them and

uniq -d

disable non-unique (-duplicate) listings.

 

Then you can also generate the md5sum of all the files:-

find /mnt/disk* -type f | xargs md5sum

 

And also the sha1sum:-

find /mnt/disk* -type f | xargs sha1sum

 

For instance:-

find /boot/ -type f -name "*.bat" | xargs md5sum

 

Would output:-

bbfa5207af2aea433545aaa191dc5fd9  /boot/make_bootable.bat

 

Although it's nice of you to post this, it seems that the default commands in unix have you beat.

[MadsRC]

I do, but I like to do stuff in python

[Automatic]

I do, but I like to do stuff in python

 

But why? Not only does that limit the amount of things you can do (Find is a HELL of a lot more developed), but, also requires the dependency of Python.

 

Seems like a lose-lose to me.

[MadsRC]

I like developing things and I like Python. Yes I could do it in bash, but I've done more Python than bash - This makes it easier for me to use Python than bash.

 

But you got me on the dependency thing. Though that usually isn't a problem since most up to date linux distributions offer some kind of up to date Python.

Since unRAID doesn't, I created a easy to use plugin (Which will be refined later on).

[Automatic]

I like developing things and I like Python. Yes I could do it in bash, but I've done more Python than bash - This makes it easier for me to use Python than bash.

 

But you got me on the dependency thing. Though that usually isn't a problem since most up to date linux distributions offer some kind of up to date Python.

Since unRAID doesn't, I created a easy to use plugin (Which will be refined later on).

 

Find isn't made in bash, I would assume it's made from some kind of C derivative (Although honestly not sure). Bash is simply a scripting language to queue up commands to be executed. Assuming we follow your logic, that'd mean that what you coded was made in Bash and so what Python, simply because you can access them from the Bash engine.

 

Anyway, I honestly recommend you learn the basic unix I/O program arguments, it would mean that you can hop to a new machine and start doing the same commands without having to copy/recode your python scripts.

[MadsRC]

I know it isn't made in bash... I've been using Linux since 06 and work with it on a daily basis. Which also means I know bash, and I know how to bash script - I simply dislike scripting in bash.

[Automatic]

I know it isn't made in bash... I've been using Linux since 06 and work with it on a daily basis. Which also means I know bash, and I know how to bash script - I simply dislike scripting in bash.

 

Well my suggestion wasn't in bash.

 

Anyway, each to their own, although, I'd much prefer to use dependent free advanced software.

[Fireball3]

I'm looking for a, probably similar, solution to list the content of a share in a way that it delivers the information on which disk of the array each file of the share is located.

 

I don't want to list the drives 1 by 1 because there are shares where i want to have the files listed and there are shares where it's sufficient to list the directories only.