ProFTPD module mod_xfer_mmap



Caveat: This is an experimental module and should be used with care.

The mod_xfer_mmap module was designed to improve the performance of downloads by using the mmap() function to map the file being downloaded into memory, and read directly from there, saving on system calls and buffer copies. The mmap() system call is available on most modern Unix derivates, but not on all. There are sometimes system-specific limits on the size and number of files that can be mmap()d; experimentation is probably the easiest way to find out.

The mod_xfer_mmap module can also create a list of statically configured files (via TransferMMapFile directives in the main server configuration) into memory. This mmap()ing is done only at server start or restart. So whenever one of the mapped files changes on the filesystem, you must restart the server (by sending it the HUP signal). To reiterate that point: if the files are modified in place without restarting the server you may end up serving stale data. You should update files by unlinking the old copy and putting a new copy in place. Most tools such as rdist and mv do this. The reason why this modules doesn't take care of changes to the files is that doing so would need an extra stat() every time, which is wasteful and against the intent of I/O reduction.

This module is contained in the mod_xfer_mmap.c file for ProFTPD 1.2.x, and is not compiled by default. Installation instructions are discussed here.

The most current version of mod_xfer_mmap can be found at:

  http://www.castaglia.org/proftpd/

Author

Please contact TJ Saunders <tj at castaglia.org> with any questions, concerns, or suggestions regarding this module.

Thanks

2004-10-31: Thanks to Michael Renner for testing out various versions of the module as it was developed.

Directives


TransferMMapEngine

Syntax: TransferMMapEngine on|off
Default: off
Context: server config, <VirtualHost>, <Global>
Module: mod_xfer_mmap
Compatibility: 1.2.10rc1

The TransferMMapEngine directive enables or disables the module's runtime mapping of files being downloaded. Note that if off, any files mapped via TransferMMapFile will be unmapped when the FTP session starts. Thus the daemon process will have mapped memory that is not used by session processes, which is not an efficient configuration.


TransferMMapFile

Syntax: TransferMMapFile path1 [path2 path3 ...]
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_xfer_mmap
Compatibility: 1.2.10rc1

The TransferMMapFile directive is used to "preload" files into memory by mmapping them at startup time, rather than just before the file is downloaded. They will be automatically unmapped when the server shuts down. When these mapped files have changed, you should restart the server to re-map them.


Installation

To install mod_xfer_mmap, copy the mod_xfer_mmap.c file into:
  proftpd-dir/contrib/
after unpacking the latest proftpd-1.2 source code. Then follow the usual steps for using third-party modules in proftpd:
  ./configure --with-modules=mod_xfer_mmap
  make
  make install


Usage

There are a couple of things to keep in mind when using mod_xfer_mmap. It only affects downloads from the server, not uploads to the server. If your proftpd uses sendfile(), that function will take precedence over mod_xfer_mmap. Files that are handled by mod_xfer_mmap can still be throttled by the TransferRate and modules like mod_shaper.

mod_xfer_mmap will not handle a file if the file is larger than mmap() can handle (which is around 2GB, depending on your platform's implementation), or if the file is to transfered as an ASCII file.

Example configuration:

  <IfModule mod_xfer_mmap.c>
    TransferMMapEngine on
  </IfModule>

Note that you may see the size of session processes changing strangely; this will happen as mod_xfer_mmap maps and unmaps files from the process' memory space.



Author: $Author: tj $
Last Updated: $Date: 2004/11/01 01:16:32 $


© Copyright 2004 TJ Saunders
All Rights Reserved