xferstats



XFERSTATS(8)                                                      XFERSTATS(8)




NAME

       xferstats - compiles information about file transfers from logfiles


SYNOPSIS

       xferstats  [-  |  +  ] [-c <config file>] [-T <number>] [[-+]C] [[-+]H]
       [[-+]n] [[-+]s] [[-+]r] [[-+]a] [[-+]g] [[-+]i] [[-+]u] [[-+]h] [[-+]m]
       [[-+]w]  [[-+]M]  [[-+]f  [number]]  [[-+]d  [number]] [[-+]t [number]]
       [[-+]O [number]] [[-+]o [number]] [+L <number>] [+A] [-D <number>]  [+D
       <number>]  [+P <prefix>] [+B <timestamp>] [+E <timestamp>] [-v] [--ver-
       sion] [--help] [--] [filename(s)]



DESCRIPTION

       xferstats is a utility  that  parses  wu-ftpd,  ncftpd,  and  (special)
       apache  xferlogs  (although it’s not difficult to add support for other
       log types) and gives various information  and  totals  about  the  data
       transferred.



OPTIONS

       +      The  single  plus tells xferstats to get the xferlog from stdin.
              For example:

              cat xferlog | xferstats +

       -      The single hyphen tells xferstats to  get  the  xferlog  from  a
              file.   While  this  is the default behaviour, the configuration
              file may change the default, and this option allows you to over-
              ride it.  For example:

              xferstats - /var/log/xferlog

       -c configuration file
              Specify  an  alternate  path  and filename for the configuration
              file.  The  packaged  default  is  /etc/xferstats.cfg  but  your
              administrator  may  have changed this.  A new default can be set
              by changing the appropriate variable in xferstats.h  and  recom-
              piling.

       -T log type
              Specify  the  type  of  log  file to be parsed.  Currently, only
              three types are defined, wu-ftpd for wu-ftpd-style logs,  ncftpd
              for  ncftpd-style  logs,  and  apache  for  special Apache logs.
              Note: you can also use "wu-ftp", "wuftp", "wuftpd", or  "ncftp".
              In  order for xferstats to parse Apache logfiles, you must use a
              special LogFile line: LogFormat "%{%a %b %d %X %Y}t %T %h %b  %f
              %u".   xferstats  may or may not support regular Apache logfiles
              in the future.  If this is something you  want,  please  let  me
              know.

       -v, --version
              Display what version of xferstats you’re running.

       --help Display a quick summary of the command line options.

       --     Use  a -- (double hyphen) to indicate that you are finished giv-
              ing parameters and will follow with a filename.  This is partic-
              ularly  useful  when  you want to specify a filename that begins
              with a + or -


   TOGGLE OPTIONS
       The options in this section can be either forced on with  +[option]  or
       forced  off  with  -[option].   Options in this section, like all other
       options, will always override any settings in the xferstats  configura-
       tion  file.   Any  arguments to toggle options should only be used when
       enabling an option.  Supplying an argument  when  disabling  an  option
       will generate an error.


       C      When  enabled,  xferstats will try to weed out corrupt log lines
              by strictly checking to make sure that the data  is  valid.   If
              you  --enable-debug_flow  at compile time, xferstats will notify
              you when corrupt lines are detected.

       H      When enabled, xferstats will generate  HTML  tables  and  graphs
              instead  of  text  tables.  In order to actually generate tables
              instead of broken graphic icons, you must put the graph JPGs  in
              the  directory  pointed  to by the GRAPH_PATH configuration com-
              mand.  For more information, see the CONFIGURATION FILE section.

       n      When enabled, xferstats will refrain from adding <HTML>, <BODY>,
              and <TITLE> tags, so that the output can be inserted into  pages
              containing custom headers.

       s      When enabled, xferstats will output HTML as a single page (file-
              name "xferstats.html") instead of a page for each section.  This
              option  also  disables  <HTML> and <BODY> tags, although it does
              produce a <TITLE> tag unless the NO_HTML_HEADERS option is  also
              specified.

       r      When enabled, xferstats will generate statistics about real user
              connections (but not necessarily only real user connections)

       a      When enabled, xferstats will generate statistics about anonymous
              connections (but not necessarily only anonymous connections)

       g      When  enabled,  xferstats  will  generate statistics about guest
              connections (but not necessarily only guest connections)

       i      When enabled, xferstats will generate statistics  about  inbound
              (the  product  of a PUT) files (but not necessarily only inbound
              files)

       u      When enabled, xferstats will generate statistics about  outbound
              (the  product of a GET) files (but not necessarily only outbound
              files)

       h      When enabled, xferstats will generate a report  listing  traffic
              by hour

       m      When  enabled,  xferstats will generate a report listing traffic
              by month (January, February, etc.)

       w      When enabled, xferstats will generate a report  listing  traffic
              by day of the week (Monday, Tuesday, etc.)

       M      When  enabled,  xferstats will generate a report listing traffic
              by day of the month (1st, 2nd, 3rd, etc.)

       f [number]
              When enabled, xferstats will generate a report  listing  traffic
              by  file.   You can supply an optional argument that will deter-
              mine how many file records xferstats displays (0 (zero) disables
              the limit)

       d [number]
              When  enabled,  xferstats will generate a report listing traffic
              by directory.  You can supply an  optional  argument  that  will
              determine  how  many  directory  records  xferstats  displays (0
              (zero) disables the limit).  See the -D and +D options, as  well
              as  the DIR_DEPTH and DIR_FILTER configuration commands for more
              information.

       t [number]
              When enabled, xferstats will generate a report  listing  traffic
              by  top-level  domain.  You can supply an optional argument that
              will determine how many TLD records xferstats displays (0 (zero)
              disables the limit)

       O [number]
              When  enabled,  xferstats will generate a report listing traffic
              by domain.  You can supply an optional argument that will deter-
              mine  how  many domain records xferstats displays (0 (zero) dis-
              ables the limit)

       o [number]
              When enabled, xferstats will generate a report  listing  traffic
              by  host.   You can supply an optional argument that will deter-
              mine how many host records xferstats displays (0 (zero) disables
              the limit)


   OTHER OPTIONS
       +A     Include all users, generate all reports

       +L <number>
              Limit the daily report to the last <number> days

       -D <number>
              Depth of path detail for directories (default 3)

       +D <directory>
              Generate  reports  on  files  located  only in <directory>.  For
              example:

              +D /pub will report only on paths under /pub.  Specifying multi-
              ple  +D options will instruct xferstats to include data from all
              specified directories.

       +P <pathprefix>
              Strip the <prefix> from any pathnames that begin with it.   This
              is the same as specifying a STRIP_PREFIX configuration file com-
              mand.  Keep in mind that  the  DIR_FILTER  (or  +D)  is  checked
              before the STRIP_PREFIX (or +P) when using these.

       +B <timestamp>
              Providing  a  +B  argument  instructs xferstats to only generate
              data on transfers occuring after that time.  The time stamp for-
              mat  is  "Jan  DD  HH:MM:ss YYYY".  One may combine +B and +E to
              generate a report about a specific window in time.

       +E <timestamp>
              Providing a +E argument instructs  xferstats  to  only  generate
              data  on  transfers  occuring  before that time.  The time stamp
              format is "Jan DD HH:MM:ss YYYY".  One may combine +B and +E  to
              generate a report about a specific window in time.


       The   default   compile-time   setting  for  the  xferlog  location  is
       /var/log/xferlog but your administrator  may  have  changed  this.   If
       xferstats  reports  an  error,  try specifying the filename as the last
       command line parameter.



CONFIGURATION FILE

       The default configuration file path is /etc/xferstats.cfg although your
       administrator may have changed this.

       Any text after a ’#’ is ignored, as long as the ’#’ begins a line or is
       prepended by whitespace.

       Commands (LOGFILE, ANON_TRAFFIC, etc) are not case sensitive,  although
       some arguments may be.


       LOGFILE <filename>
              sets  the  default log file to process.  It can be overridden by
              adding "<filename>" at the end of the command line options.  You
              may specify as many LOGFILE lines as you like; they will be pro-
              cessed in the order  specified  (although  the  order  processed
              should not change the output).  Example:

              LOGFILE   /var/log/xferlog  LOGFILE  /var/log/xferlog.1  LOGFILE
              /var/log/xferlog.2

       STRICT_CHECK
              takes no arguments.  When enabled, xferstats will  try  to  weed
              out corrupt log lines by strictly checking to make sure that the
              data is valid.  If  you  --enable-debug_flow  at  compile  time,
              xferstats  will  notify you when corrupt lines are detected.  It
              can be enabled with "+C" or disabled with "-C"  on  the  command
              line.

       ANON_TRAFFIC
              takes  no  arguments.   When  enabled,  xferstats  will generate
              statistics about anonymous connections (but not necessarily only
              anonymous connections).  It can be enabled with "+a" or disabled
              with "-a" on the command line.

       GUEST_TRAFFIC
              takes no  arguments.   When  enabled,  xferstats  will  generate
              statistics  about  guest  connections  (but not necessarily only
              guest connections).  This option has no effect when parsing non-
              wuftpd  logfiles.   It can be enabled with "+g" or disabled with
              "-g" on the command line.

       REAL_TRAFFIC
              takes no  arguments.   When  enabled,  xferstats  will  generate
              statistics about real user connections (but not necessarily only
              real user connections).  It can be enabled with "+r" or disabled
              with "-r" on the command line.

       INBOUND
              takes  no  arguments.   When  enabled,  xferstats  will generate
              statistics about inbound (the product of a PUT) files  (but  not
              necessarily  only inbound files). It can be enabled with "+i" or
              disabled with "-i" on the command line.

       OUTBOUND
              takes no  arguments.   When  enabled,  xferstats  will  generate
              statistics  about outbound (the product of a GET) files (but not
              necessarily only outbound files). It can be enabled with "+u" or
              disabled with "-u" on the command line.

       HOURLY_REPORT
              takes  no  arguments.   When  enabled, xferstats will generate a
              report listing traffic per hour.  It can be enabled with "+h" or
              disabled with "-h" on the command line.

       DOW_REPORT
              takes  no  arguments.   When  enabled, xferstats will generate a
              report listing traffic by day  of  the  week  (Monday,  Tuesday,
              etc).   It can be enabled with "+w" or disabled with "-w" on the
              command line.

       DOM_REPORT
              takes no arguments.  When enabled,  xferstats  will  generate  a
              report  listing traffic by day of the month (1st, 2nd, etc).  It
              can be enabled with "+M" or disabled with "-M"  on  the  command
              line.

       TLD_REPORT
              takes  no  arguments.   When  enabled, xferstats will generate a
              report listing traffic by top-level domain (com, edu,  net,  us,
              uk, etc).  It can be enabled with "+t <number>" or disabled with
              "-t" on the command line.

       DOMAIN_REPORT
              takes no arguments.  When enabled,  xferstats  will  generate  a
              report  listing  traffic by domain (foo.com, blah.edu, etc).  It
              can be enabled with "+O <number>" or disabled with "-O"  on  the
              command line.

       HOST_REPORT
              takes  no  arguments.   When  enabled, xferstats will generate a
              report listing traffic by host.  It can be enabled with "+o"  or
              disabled with "-o" on the command line.

       DIR_REPORT
              takes  no  arguments.   When  enabled, xferstats will generate a
              report listing traffic by directory.  The directory  depth  that
              xferstats  will keep track of can be controlled by the DIR_DEPTH
              configuration command or with  the  "-l  <depth>"  command  line
              option.   It  can  be enabled with "+d" or disabled with "-d" on
              the command line.

       FILE_REPORT
              takes no arguments.  When enabled,  xferstats  will  generate  a
              report  listing traffic by file.  It can be enabled with "+f" or
              disabled with "-f" on the command line.

       MONTHLY_REPORT
              takes no arguments.  When enabled,  xferstats  will  generate  a
              report  listing  traffic per month (January, February, etc).  It
              can be enabled with "+m" or disabled with "-m"  on  the  command
              line.

       HTML_OUTPUT
              takes  no  arguments.   When  enabled, xferstats will output all
              reports into individual HTML files.  Future versions  will  con-
              tain  a  configuration  option  to set the directory in which to
              place the HTML files.  For now they are placed  in  the  present
              working# directory. It can be enabled with "+H" or disabled with
              "-H" on the command line.

       USE_STDIN
              takes no arguments.  When enabled, xferstats will,  by  default,
              read  the logfile from the standard input instead of from a file
              on disk.  It can be enabled with "+" or disabled with "-" on the
              command line.

       LOG_TYPE
              takes  one  argument,  the type of log that will be processed by
              default.  The argument can be either a number or a case insensi-
              tive string and are as follows:

              wu-ftpd log: "1" or "wu-ftp" or "wu-ftpd" or "wuftp" or "wuftpd"

              ncftpd log : "2" or "ncftp" or "ncftpd"

              Apache log : "3" or "apache"

              Therefore:

              LOG_TYPE 1

              LOG_TYPE wu-ftp

              LOG_TYPE wu-ftpd

              LOG_TYPE wuftp

              LOG_TYPE wuftpd

              are all equivalent.

              Future versions will  contain  support  for  additional  logfile
              types. It can be set on the command line with "-T <number>"

       DIR_DEPTH
              takes one argument, the number of subdirectories deep that xfer-
              stats should go when generating statistics by  directory.   This
              configuration command has no effect when DIR_REPORT has not been
              enabled.  DIR_DEPTH has no upper  bound.   A  DIR_DEPTH  of  ’0’
              (zero)  tells  xferstats  to  process  all  subdirectories.  The
              default value is 3.  It can be set on the command line with  "-D
              <number>".  Example:

              DIR_DEPTH 3

       DIR_FILTER
              takes  one  argument, the directory that xferstats should report
              on.  xferstats will discard any data that does not have  a  path
              that  begins  with  the  value set by DIR_FILTER.  This value is
              case-sensitive, as it is part of a path name.  Specifying multi-
              ple  statements will instruct xferstats to include data from all
              specified directories.  It can be set on the command  line  with
              "+D <path>".  Example:

              DIR_FILTER /pub/foobar

       NUMBER_FILE_STATS
              takes one argument, the number of files to report on in the file
              statistics section.  It must be a number greater than  or  equal
              to zero, with zero denoting no limit.  It can be set on the com-
              mand line with "+f <number>".  The default is 50.

              NUMBER_FILE_STATS 50

       NUMBER_DIR_STATS
              takes one argument, the number of directories to  report  on  in
              the  directory  statistics section.  It must be a number greater
              than or equal to zero, with zero denoting no limit.  It  can  be
              set  on the command line with "+d <number>".  The default is 50.

       NUMBER_DAILY_STATS
              takes one argument, the number of days to report on in the daily
              statistics  section.   It must be a number greater than or equal
              to zero, with zero denoting no limit.  It can be set on the com-
              mand line with "+L <number>".  The default is 50.

              Note: the daily statistics section will always be displayed.

       NUMBER_TLD_STATS
              takes one argument, the number of top-level domains to report on
              in the TLD statistics section.  It must be a number greater than
              or equal to zero, with zero denoting no limit.  It can be set on
              the command line with "+t <number>".  The default is 50.

       NUMBER_DOMAIN_STATS
              takes one argument, the number of top-level domains to report on
              in  the  domain statistics section.  It must be a number greater
              than or equal to zero, with zero denoting no limit.  It  can  be
              set  on the command line with "+O <number>".  The default is 50.

       NUMBER_HOST_STATS
              takes one argument, the number of hosts to report on in the host
              statistics  section.   It must be a number greater than or equal
              to zero, with zero denoting no limit.  It can be set on the com-
              mand line with "+o <number>".  The default is 50.

       MAX_REPORT_SIZE
              takes  one  argument,  the maximum size of any one output table.
              This is particularly useful to keep  xferstats  from  generating
              gigantic  tables  that  make  graphical browsers choke.  It will
              split tables to be no larger  than#  this  value.   It  must  be
              greater  than  or equal to ten, or zero, zero denoting no limit.
              It can be set on the  command  line  with  "+R  <number>".   The
              default is 30.

       GRAPH_PATH
              takes one argument, the location of the JPGs used to draw graphs
              in  the  HTML  output.   It  can  be  an  absolute   path   (ie,
              http://foo.bar.com/graphs/) or a relative path (ie, ../graphs/).
              You must include a trailing ’/’  or  it  will  not  work.   This
              option  cannot be set on the command line.  The default is noth-
              ing (which will make the browser look  for  the  graphs  in  the
              directory with the HTML).  Example:

              GRAPH_PATH ../graphs/

       NO_HTML_HEADERS
              takes  no  arguments.  When enabled, xferstats will refrain from
              adding <HTML>, <BODY>, and <TITLE> tags, so that the output  can
              be  inserted  into  pages  containing  custom headers. It can be
              enabled with "+n" or disabled with "-n" on the command line.

       REFRESH
              takes one argument, the number of seconds after being  displayed
              that  the  browser should refresh it.  Unspecified or zero indi-
              cates no refresh.

       SINGLE_PAGE
              takes no arguments.  When enabled, xferstats will output HTML as
              a  single page (entitled "xferstats.html") instead of a page for
              each section.  This option also disables <HTML> and <BODY> tags,
              although  it does produce a <TITLE> tag unless the NO_HTML_HEAD-
              ERS option is also specified. It can be  enabled  with  "+s"  or
              disabled with "-s" on the command line.

       STRIP_PREFIX
              takes  one  argument.   When specified, xferstats will strip the
              given string from the beginning of any paths.  Keep in mind when
              using this option with DIR_FILTER that the DIR_FILTER is applied
              -before- stripping, so that you must take this into account when
              specifying  the DIR_FILTER string.  This option cannot be set on
              the command line.  Example:

              STRIP_PREFIX /home/ftp

       The next 11 entries are all basically the  same--they  define  how  the
       reports are to be sorted.

              0 -- Sort by the number of bytes downloaded

              1 -- Sort by the number of files downloaded

              2  --  Sort by the "name".  This varies by report; for the file,
              directory, TLD, etc reports, it sorts them  alphabetically;  for
              the  file  size  report  it lists the file sizes in alphabetical
              order; for the daily report, it lists the days  in  order.   You
              get the idea.  3 -- Sort by average throughput, in K/s

              FILE_SORT_PREF,  DIR_SORT_PREF, DOMAIN_SORT_PREF, TLD_SORT_PREF,
              HOST_SORT_PREF, DOM_SORT_PREF, DOW_SORT_PREF,  HOURLY_SORT_PREF,
              DAILY_SORT_PREF, SIZE_SORT_PREF, MONTHLY_SORT_PREF

       CHUNK_INPUT
              takes  one  argument.   When specified, xferstats will parse the
              logfiles in chunks of the  given  size  (in  number  of  lines),
              process  them,  and  continue.  This will keep it from trying to
              parse the entire log at once, using up a lot of memory for large
              logs.   Specifying  too  small  a  value will cause xferstats to
              waste a lot of time switching tasks, while too large a value may
              slow  it  down  as your system swaps.  Toy with this value a bit
              and see what works best for you.  If left unspecified, xferstats
              will continue to parse the logs in one pass.



FILES

       xferlog (generated by wu-ftpd or ncftpd)

       /etc/xferstats.cfg  contains  all of the default settings and many set-
       tings which cannot be set from the command line.

       If you installed the RPM version of xferstats, you’ll find the JPGs for
       the graphs in /usr/share/xferstats/graphs.



BUGS

       The  JPG  graph pieces included are not ideal; sometimes artifacts show
       up.  I will eventually get around to making new GIFs or something.

       Please send bug reports to phil@off.net  If  the  xferlog  that  you’re
       using  is  smaller  than  1 megabyte, please send a gzipped copy, along
       with the exact parameters you’re using.



ACKNOWLEDGEMENTS

       I greatly appreciate the suggestions and  assistance  of  Nat  Friedman
       <ndf@mit.edu>.

       Thanks   to   Mike   Shaver  <shaver@mozilla.org>  and  Jamie  Zawinski
       <jwz@netscape.com> for coughing up gigantic mozilla.org logs to aid  my
       testing.

       Thanks  to  Phil  Copeland  <bryce@zen.ics.uwe.ac.uk> for getting on my
       back constantly to add new stuff, fix nagging bugs, and  speed  it  up.
       Oh, and for using it, at least someone does :).

       Extreme gratitude to Mark Hagler <hagler@hagler.org>, someone whom I’ve
       left off of this list for too long.  Mark was kind enough to Amazon  me
       a  copy  of O’Reillys _Pthreads Programming_ which was the catalyst for
       the threaded version and really helped get me interested in the project
       again.   While  the threaded version doesn’t exist anymore, the book is
       still terribly useful for other projects.


SEE ALSO

       ftpd(8), ncftpd(8), xferlog(5)



LOCATION

       xferstats is available from the following locations.  The old addresses
       are  valid,  but are deprecated!  Please use the new addresses whenever
       possible.

              http://xferstats.off.net/

              ftp://xferstats.off.net/pub/xferstats/



AUTHOR

       xferstats is Copyright (c) 1997-2000 Phil Schwan <phil@off.net> and  is
       released under the terms and provisions of the GNU Public License.



                                  31 Oct 2000                     XFERSTATS(8)

Man(1) output converted with man2html