ProFTPD Plugin for unRAID v6.8.x


SlrG

Recommended Posts

@soder

Okay. Sorry. :) I was guessing wrong based on the ö in Söder. :)

 

Lets forget the plugin and binding directories to the ftp side on the cache for a moment. Let's just look on the unRAID side.

 

The question is,  what do you expect from your media share. Security or speed. If you create it normally, it will not use the cache drive and reside only on the array. That has the advantage, that every file you copy to your media share will be stored directly on your parity protected array. The drawback is, it is rather slow due to the parity calculations and write operations on the number of drives involved. (e.g 35 MB/sec using normal desktop grade HDDs)

 

If you set the share to use the cache drive any new file will be copied to the cache drive first. This doesn't involve parity calculation and is rather fast. (e.g. 115 MB/sec using desktop grade HDDs or faster using a single SSD or multiple Drive cache pool) The drawback is that the file will not be parity protected until it is moved to the array by the mover (which you should enable to run or new files will never get moved to the array and thus will never be protected).

 

The speeds depend on your drives, network, memory and cpu, so they might be slower or faster. Please don't take them as fixed values.

 

Now we come back to the proftpd plugin.

 

Mounting  directories  or shares to the FTP directory of the cache doesn't physically double the files. This is just some kind of gateway to make it possible to access the files on both locations. Files copied to /mnt/cache/FTP/media will be saved on the array, if media is a mount  of /mnt/user/media and the cache is not enabled for this share.

If the cache is enabled for the share the files will be  saved on the cache (but in /mnt/cache/media and not in /mnt/cache/FTP/media, though they will be accessible there, too)  and later on moved to the array by the mover.

 

I admit its a bit complex. I hope I was able to shed some light on it.

 

edit: The problems with your torrents could be, because you set the destination to be the cache (/mnt/cache/...) but the files are moved to the array by the mover vanishing on the cache. Setting the destination to /mnt/user/... should solve those problems.

 

Edited by SlrG
Link to comment

First, thanks for a great reply. I love it when I get replies like that.

 

Also, when we think about this, we should know that I do not use a parity disk for my array. Since it's mostly downloaded media, I decided that I could risk not having a parity drive for that array. Soo, there's no drawback with writing to the array, more than SSD>HDD in speed, and that a HDD needs to spin.

 

About using a cache drive for the FTP or not, I also think doesn't matter that much, since I only use the FTP to give some friends access to my media. I put it on /mnt/cache/FTP since I saw others did it that way, and of course that a SSD is faster and all that.

 

But just like you said, I right now doesn't keep my /mnt/user/media on the cache drive, since when I did many downloads, or large downloads, only one or two files could fit on my 80GB SSD cache. The first day I tried it, it worked well, since two files was using the cache, and the other downloads when directly to the array (since the cache had no space), but day two, when I tried to also download some files, the torrents that didn't fit on the cache, didn't just go directly to the array, but was instead stopped in Transmission since there was no space left on the storage. I then changed it back for my "media" to not use the cache.

 

And, when I later created the FTP folder (or share) in /mnt/cache/ and said "no" to if use cache, I got an "error" with the "fix common problems" plugin that I had the FTP folder on the cache, but had settings to "no".

 

And, about your last "edit", all my downloads in Transmission goes directly to "/downloads/videos/movies" in Transmission, where the "/downloads" part in the Transmission Docker is /mnt/user/media/, and there for my downloads all goes in to /mnt/user/media/videos/movies and not set to use the cache. 

 

/Söder

Link to comment

I'm glad you liked it. :) Thank you.

 

Conclusion from you description above is, you probably need no /mnt/cache/FTP directory. You could just point the home dir of the ftp user directly to /mnt/user/media and use the directory and limit rules to make it read only and prevent access to subdirs you don't want to be read at all. This should "solve" the fix common plugins error. Another way is to set the error to ignored in fix common problems.

 

As for the cache running full, this is probably, because the mover runs on a shedule and not if the disk space gets low.

 

If you have other questions, please let me know. :)

 

 

Link to comment
  • 3 weeks later...

Is there a way to make the logs working for proftpd? I've edited the config file, but it doesn't work.

 

# Control Logging - comment and uncomment as needed

# If logging Directory is world writeable the server won't start!

# If no SystemLog is defined proftpd will log to servers syslog.

#SystemLog               NONE

#SystemLog               /boot/config/plugins/ProFTPd/slog

#TransferLog             NONE

TransferLog             /boot/config/plugins/ProFTPd/xferlog

WtmpLog                 NONE

 

But the xferlog always empty for me...

 

/Söder

Link to comment

How do you web? The correct way is:

ftp://ip:21

The system should ask for username and password and allow you access to the folder you designated in the description of the user you created.

 

ftp://user@ip:21

is also possible. Then, the system should only ask for your password.

 

Using web based ftp will only allow browsing and downloads AFAIK. So you might want to take a look at dedicated FTP client programs like FileZilla, which will allow for uploads, too.

 

Link to comment
  • 1 month later...

Plugin update was continuously failing. Then managed to fully uninstall but after that installation now fails. 

Can someone help?

 

plugin: installing: https://github.com/SlrG/unRAID/raw/master/Plugins/ProFTPd.plg
plugin: downloading https://github.com/SlrG/unRAID/raw/master/Plugins/ProFTPd.plg
plugin: downloading: https://github.com/SlrG/unRAID/raw/master/Plugins/ProFTPd.plg ... done
Cleaning up previous install files...
Executing selfremoval commands if present...
Found correct version of plugin dependency files. Ignoring
...Cleanup complete!
Downloading any missing plugin files...
Latest plugin control files downloading
...Unable to download missing plugin control files. Aborting install
plugin: run failed: /bin/bash retval: 1

Link to comment

@quad

It seems to be the same problem, you reported one page back. The plugins check for a working internet connection seems to be very flaky on your system. Sadly I still had not the time to look into it. You fixed it by modifying the checks else condition so it always reports yes and then installing the plg manually. This will probably work again.

 

Link to comment

I've been dealing with a crazy cycle with my unraid server (latest version).  The server is on an ASUS P8-V77Z motherboard, 16gb memory, 11 disks using 2 LSI 9207-8i controllers.  I have a number of apps running, one of which is Proftpd which I rely on for backups (because of SMB issues).  I have one docker running (Plex) which works fine.

 

The problem I've had (and posted here: https://lime-technology.com/forums/topic/59269-persistent-syslog-or-how-to-save-syslog-in-a-crash/) didn't lead to a permanent answer.  The server can run for a couple of days, then just dies.  I opened a real-time log window (from the GUI) and just let it run hoping that the last entries might give me an answer.  The very last lines in the real-time log tail were ProFTPd's typical complaint about 'world-writable directory'.  I know this is a stretch, but I'm grasping at straws.  Is it possible that ProFTPd could be having a conflict with something in my system?  Is there a way to diagnose that?

 

Log tail attached.  Thanks in advance!

Hunternas Log Tail.txt

Link to comment

@jeffreywhunter

Well do you really need the transfer log? It doesn't seem to be correctly configured on your system anyway, so why not disable it completely by setting it to NONE?

That should stop the message from spamming your log.

 

I can't imagine the ProFTP plugin being responsible for your lockups, but to be absolutely sure, you will have to uninstall and run some time without the plugin. If the system still crashes, there is another underlying problem.

 

Looking at your log (but I am no expert in this and might be wrong) there seems to be a memory leak and you are running out of memory and thus the system crashes.

 

Maybe you could follow the tips from here and check your shfs usage and try to change the disk cache ratio?   

 

 

Link to comment

Sorry if this a redundant question, I skimmed through the thread and searched a bit but couldn't find a similar issue with a resolution. 

 

I have been having this plugin installed but I am having a slight issue. I currently have ProFTPd-2018.02.05.1 running and I am on OS 6.5.1. I have created a user and placed "ftpuser" in the description. (minus the quotes). I also opened up port 21 in my router for testing purposes. Will change ports once this is working. 

 

I can connect fine when I am on my LAN via 192.168.x.x, but have never been successful via WAN. I sometimes get the username and password dialog box but after hitting enter I get nothing. I have tried the Transmit iOS app and various other browsers ftp clients with no luck. The transmit app works fine on LAN. 

 

I feel there is one small piece of the puzzle I am missing. 

 

Thanks in advanced. 

Link to comment

For FTP to work, you need additional passive ports. In the local network they won't be blocked by your firewall and everything will work.

 

From the WAN some routers allow the automatic  opening of passive ports while others don't. So you might need to define a passive port range for proftp to use in the proftpd.conf and define the same range of open ports in your router.

 

Make sure to really change the port to something other as 21, as that will get constantly scanned with malicious intent.

Link to comment
On 5/7/2018 at 10:21 AM, SlrG said:

@jeffreywhunter

Well do you really need the transfer log? It doesn't seem to be correctly configured on your system anyway, so why not disable it completely by setting it to NONE?

That should stop the message from spamming your log.

 

I can't imagine the ProFTP plugin being responsible for your lockups, but to be absolutely sure, you will have to uninstall and run some time without the plugin. If the system still crashes, there is another underlying problem.

 

Looking at your log (but I am no expert in this and might be wrong) there seems to be a memory leak and you are running out of memory and thus the system crashes.

 

Maybe you could follow the tips from here and check your shfs usage and try to change the disk cache ratio?   

 

 

 

I took your advice and disabled the transferlog (TransferLog             NONE).  It ran for about 3 days, then crashed again.  I had turned on Fix Common problems troubleshooting mode.  I've attached the file.  Not sure what to look for.  The last couple of problems was:

 

May 10 04:12:50 HunterNAS root: Fix Common Problems Version 2018.04.25
May 10 04:12:51 HunterNAS root: Fix Common Problems: /var/log currently 2 % full
May 10 04:12:51 HunterNAS root: Fix Common Problems: rootfs (/) currently 8 % full
May 10 04:20:25 HunterNAS kernel: mdcmd (149): set md_write_method 0
May 10 04:20:25 HunterNAS kernel: 
May 10 04:22:51 HunterNAS root: Fix Common Problems: Capturing diagnostics.  When uploading diagnostics to the forum, also upload /logs/FCPsyslog_tail.txt on the flash drive

 

Attached is the Fix Common problems troubleshooting logs.  Appreciate any help!

hunternas-diagnostics-20180507-1131.zip

FCPsyslog_tail.txt

 

Last bit, I had found some posts about OOM issues here, I subsequently changed vm.dirty_background to 1% and vm.dirty_ratio to 2% based on the post, but then still had the problem.

 

Thanks in advance!

Edited by jeffreywhunter
Link to comment

@jeffreywhunter

The diagnostics file is "old" from 0507 while the FCPsyslog_tail.txt is from 0510 when your system crashed. The newer syslog shows no real problems going on before the crash. At least it seems not to be an classical out of memory problem now, as both values are below 10%. For now, I would reset the values you changed to their original settings.

 

Then I would monitor the shfs usage like described here and a syslog tail like you did before.

 

If this doesn't show anything unusual you will really have to uninstall the proftpd plugin and see if the system still crashes without it.

Link to comment
5 hours ago, SlrG said:

 

Then I would monitor the shfs usage like described here and a syslog tail like you did before.

 

 

I took a look at this, shfs on my system:

top - 08:03:26 up 15:15,  3 users,  load average: 0.69, 0.66, 0.72
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.0 us, 37.4 sy,  0.3 ni, 58.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
GiB Mem :   15.369 total,    2.964 free,    3.849 used,    8.557 buff/cache
GiB Swap:    0.000 total,    0.000 free,    0.000 used.   10.351 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
13276 root      20   0  747644 274744    644 S   0.3  1.7   5:23.63 shfs

 

I've started HTOP and added a memory meter just to see if the memory is growing.  So far the max I've seen is 4.44G/15.4G and its steady.

root@HunterNAS:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          15738        3684        3292         606        8761       10855
Swap:             0           0           0

Thoughts?

Edited by jeffreywhunter
Link to comment

@SlrG

I've run several backups and noticed no problems with proftpd.  I do see that the process is only alive when an actual ftp session is active.  i.e. it only shows up as a process when a backup is running.  Not seeing anything strange.  Is there something else I can monitor?

 

Is it possible that the Docker.img file has some runaway logging going on?  How would i investigate that?

Edited by jeffreywhunter
Link to comment

@jeffreywhunter

In this post the instructions to monitor shfs usage should be pretty clear. You create a file on your flash containing the given line and restart the array. Afterwards it will start logging shfs problems to your syslog. So if you tail that, problems should get visible, if there are any. The process to stop the logging is also described.

 

The syslog from the diagnostics file IMHO  doesn't show a reason for your crashes but again, I'm no expert in reading them. The root file system is not running full, thats good. The top info could be a bit suspicious:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
24715 ftpacco+  20   0  237632 204152   4456 R  81.2  1.3  17:11.28 proftpd
 8150 root      20   0  947424 354676    764 S   6.2  2.2  45:31.78 shfs

Sadly it doesn't show the exact time it was taken. If it was at the time of the crash, shfs is probably not responsible, as its memory usage is low. My plugin could be responsible as it is the top process at the time the info was saved (probably a file transfer in progress).

 

To be fully sure it is/is not my plugin, I can only recommend again to remove it fully and see if the system still crashes or not.

 

If it not crashes any longer, we could try to increase the logging of the plugin to get more clues what is going wrong.  

 

If it still does  with the plugin removed, it would be better to post your diagnostics file to the official bug reports forum, so more people and people with more knowledge than me can have a look at your problem.

 

If nothing is visible from the diagnostics file, the recommendation will be to start with a minimal system and expand from that. I know this isn't ideal, as it takes a lot of time while the system is not fully functional, but it is the only way to troubleshoot those problems. If a basic install without plugins, dockers and vms still crashes, it is probably some kind of hardware problem or a bug limetech needs to fix for your system. If that runs fine, you can start reinstalling things one by one. Install one plugin at a time and monitor if the system still crashes. If it is still stable, reinstall the next one. Enable docker. Monitor your system.  Install the first docker. Monitor your system. Install the next. Enable VMs. ...

 

That way you will find out, what exactly is responsible for your problems and then can contact the responsible developer to check it out.

Link to comment
On 5/10/2018 at 2:52 AM, SlrG said:

For FTP to work, you need additional passive ports. In the local network they won't be blocked by your firewall and everything will work.

 

From the WAN some routers allow the automatic  opening of passive ports while others don't. So you might need to define a passive port range for proftp to use in the proftpd.conf and define the same range of open ports in your router.

 

Make sure to really change the port to something other as 21, as that will get constantly scanned with malicious intent.

 

So after researching through the readme I went in my router and opened port range 60000-65534 and pointed that to my server. 

 

I then added the following to the proftpd.conf file:

PassivePorts 60000 65534

Still have successful LAN and no WAN access. Does that line of text go anywhere special in the conf file. I also can't add anything in the Edit .conf file section of the plugin settings page. With the plugin stopped and/or started. Should I be able to? 

ports.png

Link to comment

@jeffreywhunter

Normally the plugin disables the internal unRAID webserver automatically. It is reenabled, when the plugin is uninstalled. The reason is, that both use the default port 21 and proftp won't start, if the internal ftp already uses this port.

 

Looking at your logs, the internal ftp seems to be disabled by the plugin. The fix common error plugin says the internal ftp is disabled, but ftp users are defined. Which is normal, as the proftp plugin uses them.

 

So normally you should have no way to disable the internal unRAID webserver yourself, as the link to the settings page gets removed when my plugin is installed? If you still have both icons in your settings, its a bit odd.

 

@rmeaux

Your proftpd.conf looks fine and if you have opened the ports on the router everything should work in theory. Did you restart the proftpd server after making the change to the config? Otherwise changes will not get applied. If it still doesn't work, it might be you need to add a MasqueradeAddress, too. Either with your fixed IP or your dyndns name.

 

The plugins config editor needs a running webserver with access rights to the config dir. All the docker ones doesn't have that and the apache plugin webserver stopped running with the latest versions and will get no update. So the functionality to edit the config from the plugins settings page is broken and I have no idea how to make it work again at the moment. It makes sense from a security point of view, however that it has been made harder to access the config files of the server.

 

 

Link to comment

Thanks for the help.

I was reading about the masquerade feature after I played around with the ports. I will try that. I do have static IP’s which are assigned to sub domains so I will add it in.

I used the config file editor plugin to edit the files. Which seems to work great and the changes seem to hold. I went back and checked with TextWrangler to make sure.

I’ll post my findings or resolutions for anyone in the future.


Sent from my iPhone using Tapatalk

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.