FTPmir v0.1u, 11/09/97 by CMcN,
additions feb 2000 onwards, by JLW.
==============================================================
SUMMARY
=======
A program to assist in using KA9Q to mirror a local directory to a
remote FTP server, by generating scripts to automate the upload
process.
---------------------------------------------------------------------
INTRODUCTION
============
FTPMIR builds a KA9Q ftp script to mirror a local directory to a
remote directory on an FTP server, such as a website.
FTPMIR allows you to use one ftp command to upload any new or changed
files, and (optionally) to delete redundant files from the remote
site. If you use a version of KA9Q that allows ftp scripts to be
invoked from the command-line, you can completely automate the
process, as follows:
1/ Ensure that FTPMIR runs before each KA9Q session
2/ Modify autoexec.net so that the FTP script is invoked (or remember
to invoke it manually if you use a version of KA9Q that doesn't
permit calling scripts from the command line)
Personally I (jlw) use Ian Smith's YAN variant of ka9q, and I
have a script on a function key defined in autoexec.net, e.g.
fkey 91 "[command]source wwwaut.scr\n"
In my \demon\wwwaut.scr file I have
cd /www
ftp @/www/upload.scr
and my mirror tree is rooted on my c:\www directory, in
other words c:\www\index.htm is the file that appears on my web
site home page.
This isn't the only way to do it, but it works for me.
Actually that's a slight simplification, because I have two
web sites, and both of them are mirrored.
---------------------------------------------------------------------
METHODOLOGY
===========
FTPMIR works as follows:
1/ A script generated by FTPMIR is used to retrieve a remote
directory listing.
2/ The directory listing is used to build a further script to upload
new or changed files from the local directory: files are selected
if they are newer than those on the remote site, or if their size
is different. The size calculation allows for ASCII conversion:
local files to be uploaded as ASCII have their ASCII size
calculated for a Unix machine (DOS servers are also catered for,
see the -a flag, but most servers are Unix based).
3/ When invoked, the upload script sends files (and optionally
deletes those on the remote site no longer held locally), and
finally generates a new directory listing reflecting the changes.
4/ Alter your local directory as desired.
5/ Back to step 2
To start the process off, use the -f flag to suppress uploads
and just get a directory listing script. If you generate a full
script initially, your entire directory will be uploaded (probably
not what you want, unless your site is bare).
If you have more than a single directory, use the -r flag (or,
better, add Recurse=on to your config file). You need to keep
running with the -f flag and connecting each time to run the
resulting script -- each time you'll get an extra level of your
web site's tree. When there's no further changes, step 1 above
is complete.
---------------------------------------------------------------------
CONFIGURATION FILE
==================
FTPMIR is controlled by a configuration file. A sample configuration
file (SAMPLE.CFG) is included: it should be self-explanatory. You'll
need a separate configuration file for each local directory you want
to mirror, unless you use the recursion feature.
Errors in the config file should be detected, but you would be
advised to use the -t command-line option to test the scripts created
until you're sure of what you're doing.
---------------------------------------------------------------------
COMMAND LINE
============
Usage: FTPMIR [options] configfile
Options:
-a Ascii file lengths as in directory, not Unix
-c suppress Comments in script file
-d delete Remote files not found in local directory
-f get Fresh remote directory listing, ignoring local copy (if any)
-g don't Generate script file, but show statistics
-h show Help screen
-ln Logging level n (0 = off, 3 = max, default = 1)
(not implented yet)
-q suppress quit command at end of script, to leave session open
-r recurse over local tree (even if not specified in .cfg)
-s suppress Statistics display
-t Test config file only; no script file generated
-u no Upload (or delete)
-vn on-screen Verbosity (0 = off, 3 = max, default = 1)
(not implented yet)
-? show this Help screen
---------------------------------------------------------------------
TIMEZONES
=========
FTPMIR assumes that the remote server runs on GMT (aka UTC), and that
the local system either runs on GMT or has a correctly set timezone
variable.
NB If the timezone is not correctly set, files may not be correctly
synchronised.
My C libraries do not support the enhanced time zone syntax used by
KA9Q (e.g. TZ=GMT0BST1,M3.5.0/01:00,M10.5.0/02:00), and use only
the initial part (e.g. TZ=GMT0BST).
Because the syntax doesn't allow a precise specification of when the
clocks are adjusted, there may be a week before or after the
changeover in which the local time is incorrectly reported by 1 hour.
This *can* lead to unnecessary uploads if local files are changed less
than one hour before a remote directory listing is generated.
---------------------------------------------------------------------
DRIVES & DIRECTORIES
====================
Note that because KA9Q can access only the current drives, all
files and directories referenced by FTPMIR must be on the same
drive as KA9Q. Attempts to specify other drives will be rejected.
---------------------------------------------------------------------
ABORTED UPLOADS
===============
Unfortunately, KA9Q does not allow logging of FTP sessions, which can
cause problems with aborted FTP sessions.
The last action of the script is to retrieve an updated listing of
the directory. If your upload is aborted, that will not be available,
so as a precaution the remote directory listing is also requested as
the first action in the script.
Consider the following situation:
a) remote listing retrieved
b) script generated, to upload 25 files.
c) KA9Q session aborted after 8 files uploaded.
A new directory listing will not be available, so the next locally
generated script will attempt to repeat the entire upload. If it is
also aborted, the cycle will continue. By listing the directory
first, an infinite loop can be avoided, as follows:
1/ Script generated to upload 25 files;
2/ Script invoked: initial directory list retrieved, 8 files
uploaded before abort;
3/ Script generated to upload 25 files;
4/ Script invoked: initial directory list retrieved, 12 files
uploaded;
5/ Script generated to upload 17 files; 11 files uploaded.
... and so on, until eventually all files are uploaded, though with
much duplication.
Ideally, to avoid this problem, a remote directory listing should be
retrieved in alternate KA9Q sessions:
Session 1: get directory listing (script generated using -u
command-line option)
Session 2: full script to upload and/or delete files
Session 3: get directory listing only
When recursing, each directory listing is fetched immediately after
directory upload is complete, instead of when all directories
are complete.
---------------------------------------------------------------------
LIMITATIONS
===========
This is a very early version, so it's not fully tested. I'm not
aware of any outstanding bugs, but would welcome reports of any you
find (to jlw). Email ftpmir 'at' johnwash 'dot' com. (Sorry, trying
to keep it spam free, but I'm sure you'll figure it out!)
There are limits to the number of Comment=, Exclude=, Ascii= and
Convert= entries in the configuration file: currently 9 Comments, 40
Excludes, 20 Ascii extensions, and 20 extension conversions. You'll
be told when you reach them; if they are too low, please let me know,
and they can be increased.
I (cmcn) haven't found a way of generating a recursive directory listing on
the FTP server, so each directory must be handled with a separate
configuration file, and a separate FTP script.
[jan/2000] -r switch implements recursion over local tree, or
put "Recurse=on" at the start of the configuration file. This
works for me (jlw).
Memory constraints impose a limit to the number of files that can be
handled: I don't think there should be problems with directories of
less than about 400 files.
---------------------------------------------------------------------
COMMENTS
========
I (cmcn) wrote this to keep an updated copy of a mailing list archive on the
web. I'm now using it for all my other web stuff -- it's the easiest
way to ensure I upload the right files. I'd be interested to hear of
other uses people find for it, as well (of course) as bug reports.
Bug reports to jlw please -- cmcn has requested not to receive
bug reports.
---------------------------------------------------------------------
TO-DO LIST
==========
* implement on-screen verbose output, and logging
* support enhanced timezone syntax
---------------------------------------------------------------------
WISH-LIST
=========
* inbound mirroring (using GET commands)
(July 2000: pretty much complete)
* ability to manually specify local and remote timezones (for servers
that don't use GMT)
* virtual memory (to remove limits on file numbers)
Additions to the wish-list may or may not be considered for future
versions.
------------------------------------------------------------------
