INTERNET DRAFT                           Phillip M. Hallam-Baker, W3C
   Expires in six months                          email: <hallam@w3.org>
                                                         Brian Behlendorf 
                                               email: <brian@organic.com>
                                                      21st February 1996
   

                          Extended Log File Format 

                      <draft-hallam-http-logfile-00.txt>

Status of this Memo

   This document is an Internet draft. Internet drafts are working
   documents of the Internet Engineering Task Force (IETF), its areas
   and its working groups. Note that other groups may also distribute
   working information as Internet drafts. 

   Internet Drafts are draft documents valid for a maximum of six
   months and can be updated, replaced or obsoleted by other documents
   at any time. It is inappropriate to use Internet drafts as reference
   material or to cite them as other than as "work in progress". 

   To learn the current status of any Internet draft please check the
   "lid-abstracts.txt" listing contained in the Internet drafts shadow
   directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
   munnari.oz.au (Pacific Rim), ds.internic.net (US East coast) or
   ftp.isi.edu (US West coast). Further information about the IETF can
   be found at URL: http://www.cnri.reston.va.us/ 

   Distribution of this document is unlimited. Please send comments to
   the HTTP working group (HTTP-WG) of the Internet Engineering Task
   Force (IETF) at < http://www.ics.uci.edu/pub/ietf/http/. This note
   is also avaliable as a World Wide Web Consortium Working Draft
   WD-logfile-960221, archived at 
   http://www.w3.org/pub/WWW/TR/WD-logfile-960221.html   

   Extended Log File Format 

                                                             WD-logfile-960221

                          Extended Log File Format 

                    W3C Working Draft _WD-logfile-960221_

   This version:
        http://www.w3.org/pub/WWW/TR/WD-logfile-960221.html 

   Latest version:
        http://www.w3.org/pub/WWW/TR/WD-logfile.html 

   Authors:
       Phillip M. Hallam-Baker <hallam@w3.org>

Phillip M. Hallam-Baker                                                 Page 1



Extended Log File Format                                                      

       Brian Behlendorf  <brian@organic.com>

------------------------------------------------------------------------------
Status of this document

   This is a W3C Working Draft for review by W3C members and other
   interested parties. It is a draft document and may be updated,
   replaced or obsoleted by other documents at any time. It is
   inappropriate to use W3C Working Drafts as reference material or to
   cite them as other than "work in progress". A list of current W3C
   working drafts can be found at: http://www.w3.org/pub/WWW/TR 

   Note: since working drafts are subject to frequent change, you are
   advised to reference the above URL, rather than the URLs for working
   drafts themselves. 







































Phillip M. Hallam-Baker                                                 Page 2



Extended Log File Format                                                      

Abstract 

   An improved format for Web server log files is presented. The format
   is extensible, permitting a wider range of data to be captured. This
   proposal is motivated by the need to capture a wider range of data
   for demographic analysis and also the needs of proxy caches. 

Introduction 

   Most Web servers offer the option to store logfiles in either the
   common log format or a proprietary format. The common log file
   format is supported by the majority of analysis tools but the
   information about each server transaction is fixed. In many cases it
   is desirable to record more information. Sites sensitive to personal
   data issues may wish to omit the recording of certain data. In
   addition ambiguities arise in analysing the common log file format
   since field separator characters may in some cases occur within
   fields. The extended log file format is designed to meet the
   following needs: 

   *   Permit control over the data recorded. 

   *   Support needs of proxies, clients and servers in a common format 

   *   Provide robust handling of character escaping issues 

   *   Allow exchange of demographic data. 

   *   Allow summary data to be expressed. 

   The log file format described permits customized logfiles to be
   recorded in a format readable by generic analysis tools. A header
   specifying the data types recorded is written out at the start of
   each log. 

   This work is in part motivated by the need to support collection of
   demographic data. This work is discussed at greater length in
   companion drafts describing  session identifier URIs [Hallam96a] and
   more consistent proxy behaviour [Hallam96b]. 

Format 

   A extended log file contains a sequence of _lines_ containing ASCII
   characters terminated by either the sequence CR or CRLF. Log file
   generators should follow the line termination convention for the
   platform on which they are executed. Analysers should accept either
   form. Each line may contain either a _directive_ or an _entry_. 

   Entries consist of a sequence of _fields_ relating to a single HTTP
   transaction. Fields are separated by whitespace, the use of tab
   characters for this purpose is encouraged. If a field is unused in a
   particular entry dash "-" marks the omitted field. Directives record
   information about the logging process itself. 

Phillip M. Hallam-Baker                                                 Page 3



Extended Log File Format                                                      

   The following directives are defined: 

   Version: _<integer>_._<integer>_ 
       The version of the extended log file format used. This draft
       defines version 1.0. 

   Fields: [_<specifier>_...] 
       Specifies the fields recorded in the log. 

   Software: _string_ 
       Identifies the software which generated the log. 

   Start-Date: _<date>_ _<time>_ 
       The date and time at which the log was started. 

   End-Date:_<date>_ _<time>_ 
       The date and time at which the log was finished. 

   Date:_<date>_ _<time>_ 
       The date and time at which the entry was added. 

   Remark: _<text>_ 
       Comment information. Data recorded in this field should be
       ignored by analysis tools. 

   The directives Version and Fields are required and should preceed
   all entries in the log. The Fields directive specifies the data
   recorded in the fields of each entry. 

Example 

   The following is an example file in the extended log format: 

   
   #Version: 1.0
   #Date: 12-Jan-1996
   #Fields: time cs-method cs-uri
   00:34:23 GET /foo/bar.html
   12:21:16 GET /foo/bar.html
   12:45:52 GET /foo/bar.html
   12:57:34 GET /foo/bar.html
   

Fields 

   The #Fields directive lists a sequence of _field identifiers_
   specifying the information recorded in each entry. Field identifiers
   may have one of the following forms: 

   _identifier_ 
       Identifier relates to the transaction as a whole. 

   _prefix_-_identifier_ 

Phillip M. Hallam-Baker                                                 Page 4



Extended Log File Format                                                      

       Identifier relates to information transfer between parties
       defined by the value _prefix_. 

   _prefix_(_header_) 
       Identifies the value of the HTTP header field _header_ for
       transfer between parties defined by the value _prefix_. Fields
       specified in this manner always have the value <string>. 

   The following prefixes are defined: 

   cs 
       Client to Server. 

   sc 
       Server to Client. 

   sr 
       Server to Remote Server, this prefix is used by proxies. 

   rs 
       Remote Server to Server, this prefix is used by proxies. 

   x 
       Application specific identifier. 

   The identifier cs-method thus refers to the method in the request
   sent by the client to the server while sc(Content-Type) refers to
   the content type field of the reply. 

Identifiers. 

   The following identifiers do not require a prefix 

   date 
       Date at which transaction completed, field has type <date> 

   time 
       Time at which transaction completed, field has type <time> 

   time-taken 
       Time taken for transaction to complete in seconds, field has
       type <fixed> 

   bytes 
       bytes transfered, field has type <integer> 

   cached 
       Records wether a cache hit occured, field has type <integer> 0
       indicates a cache miss. 

   The following identifiers require a prefix 

   ip 

Phillip M. Hallam-Baker                                                 Page 5



Extended Log File Format                                                      

       IP address and port, field has type <address> 

   dns 
       DNS name, field has type <name> 

   status 
       Status code, field has type <integer> 

   comment 
       Comment returned with status code, field has type <> 

   method 
       Method, field has type <name> 

   uri 
       URI, field has type <uri> 

   uri-stem 
       Stem protion alone of URI (omitting query), field has type <uri> 

   uri-query 
       Query portion alone of URI, field has type <uri> 

   host 
       DNS hostname used, field has type <name> 

Special fields for log summaries. 

   Analysis tools may generate log summaries. A log summary lists the
   number of operations performed on a URI but omits information
   specific to a particular transaction. 

   The following field is mandatory and must preceed all others: 

   count 
       The number of entries for which the listed data, field has type
       <> 

   The following fields may be used in place of time to allow
   aggregation of log file entries over intervals of time. 

   time-from 
       Time at which sampling began, field has type <time> 

   time-to 
       Time at which sampling ended, field has type <time> 

   interval 
       Time over which sampling occurred in seconds, field has type
       <integer> 

Entries 


Phillip M. Hallam-Baker                                                 Page 6



Extended Log File Format                                                      

   This section describes the data formats for log file field entries.
   These formats are chosen so as to avoid ambiguity, minimize the
   difficulty of generation and parsing and provide for human
   readability. 

   Each logfile entry consists of a sequence of fields separated by
   whitespace and terminated by a CR or CRLF sequence. The meanings of
   the fields are defined by a preceeding #Fields directive. If a field
   is ommitted for a particular entry a single dash "-" is substituted. 

   Log file parsers should be tolerant of errors. If an entry contains
   corrupt data or is terminated unexpectedly the parser should
   resynchronize using the end of line marker and continue to parse the
   following entries. Entries must not contain any ASCII control
   characters. 

   
   <entry> = <field>* <end-of-line>
   
   <field> = <integer> | <fixed> | <uri> | <date> | <time> | <string>
   
   <digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9"
   

Integer 

   
   <integer>	= <digit> +
   

   Integers are represented as a sequence of digits. 

Fixed Format Float 

   
   <float>	= <digit> [. <digit>*]
   

URI 

   A URI as specified by RFC1738, relative URIs are specified by
   RFC1808. URIs cannot by definition include whitespace or ASCII
   control characters. Consequently no ambiguity arises from their use. 

Date 

   
   <date>	= <digit> <digit> <digit> <digit>
   			"-" <digit> <digit> "-" <digit> <digit>
   

   Dates are recorded in the format YYYY-MM-DD where YYYY, MM and DD
   stand for the numeric year, month and day respectively. This format

Phillip M. Hallam-Baker                                                 Page 7



Extended Log File Format                                                      

   is chosen to assist collation using sort. 

Time 

   
   <time>	= <digit> <digit> ":" <digit> <digit>
   			[":" <digit> <digit> ["." <digit>*]
   
   

   Times are recorded in the form HH:MM, HH:MM:SS or HH:MM:SS.S where
   HH is the hour in 24 hour format, MM is minutes and SS is seconds. 

String 

   
   <string>	= '"' <schar>* '"'
   
   <schar>	= xchar | '"' '"'
   

   Strings are output in quoted form. If a string contains a quotation
   character the character is repeated. This format is unambiguous
   since fields are by definition separated by whitespace. 

   No mechanism for incorporating control characters is defined. 

Text 

   
   <text>	=  <char>* 
   

   The text field is used only by directives. 

Name 

   
   <name>	=  <alpha> [ "." <alpha> * ] 
   

   DNS name. 

Address 

   
   <name>	=  <integer> [ "." <integer> * ] [ ":" <integer> ]
   

   Numeric IP address and optional port specifier. 

Acknowledgements. 


Phillip M. Hallam-Baker                                                 Page 8



Extended Log File Format                                                      

   Robert Thau provided usefull advice and some code. John Mallery and
   Roger Hurwitz helped develop many of the ideas. 

   
   Phillip M. Hallam-Baker
   hallam@w3.org
   World Wid Web Consortium
   Cambridge MA
   
   Brian Behlendorf 
   brian@organic.com
   Organic Online
   

References. 

   
   [RFC1808] 
       R. Fielding _ Relative Uniform Resource Locators_, June 1995 

   [RFC1738] 
       T. Berners-Lee, L. Masinter, _ Uniform Resource Locators (URL)_,
       December 1994 

   [Luotonen95] 
       A. luotonen, _ The Common Logfile Format_, 1995,
       http://www.w3.org/pub/WWW/Daemon/User/Config/Logging.html 

   [Hallam96a] 
       P. M. Hallam-Baker _ Session Identification URI_ World Wide Web
       Consortium Working Draft, WD-session-id. 

   [Hallam96b] 
       P. M. Hallam-Baker _ Notification for Proxy Caches_ World Wide
       Web Consortium Working Draft, WD-proxy. 



















Phillip M. Hallam-Baker                                                 Page 9