HTTP Working Group Rohit Khare INTERNET-DRAFT World Wide Web Consortium Massachusetts Institute of Technology Expires: February 19, 1997 August 19, 1996 HTTP/1.2 EXTENSION PROTOCOL (PEP) Status of this memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress". To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Distribution of this document is unlimited. Please send comments to the HTTP working group at . Discussions of the working group are archived at http://www.ics.uci.edu/pub/ietf/http/. This Internet Draft is also available as W3 Consortium Working Draft WD-http-pep-960819. A current version is available at http://www.w3.org/pub/WWW/TR/WD-http-pep. Abstract PEP is a system for HTTP clients, servers, and proxies to reliably reason about custom extensions to HTTP. Traditionally, HTTP agents offer extended behavior by private agreement using additional message headers. PEP has features for expressing the scope, strength, and ordering of such extensions, as well as which extensions are available. PEP as described here is substantially simpler than previous drafts. The title has also been changed, since PEP addresses a chartered HTTP WG work item. 1. Introduction HTTP [3] messages, like most applications of RFC 822 [4], can be extended with additional header fields and content formats. Traditionally, the standards update process itself extends the protocol for major, global new features. However, this approach is problematic for deploying casual experiments which could scale up from pairwise to universal deployment, especially as HTTP agents become more modular, dynamic, and diverse. Casual extension of HTTP is hindered by several challlenges: 1. Naming. To guarantee interoperability, header names, encoding tokens, types, response codes, etc. must be centrally arbitrated by appealing to IETF standards-track documents or the Internet Assigned Numbers Authority (IANA). This requires more effort from implementors than an extension may warrant. 2. Importance. There is no way to indicate the consequences of ignoring an extension: can an agent silently ignore it, may it warn the user, or must it return an error message? 3. Scoping. There is no reliable way to indicate who is to process an extension: the next HTTP agent (proxy), or only the origin agent (client or server)? 1.1's Connection: is only a partial solution, since it cannot convey the criticality of stripped headers. 4. Ordering. There is no way to indicate in what order to process extensions. If several are applied, it can be critical to 'unpack' the message in the correct order. 5. Initiation. Short of unilaterally utilizing an extension, there is no way to request the other party to initiate it. 6. Advertising. Though it is possible to enumerate content variants and the headers they differ upon, it is not possible to describe available extensions in detail. 7. Inquiring. There is no way to inquire if an extension is supported before attempting to use it. This is critical for selecting among related extensions and upgrading to new versions. PEP introduces the concept of protocol extensions to systematically address these issues at a level of abstraction above header names and content-types (hence, `Protocol Extension Protocol'). With PEP, HTTP agents can interoperate correctly with known and unknown protocol extensions, select protocol extensions available to both sides, and query partners for specific capabilities. PEP adds to HTTP the extensibility lessons learned from ESMTP (extension naming) [10], IPv6 (unknown-option disposition) [5], and Telnet (option negotiation) [12]. In addition to reliably describing statically extended HTTP servers and clients, PEP will work with dynamically extended agents. Indeed, the authors expect that PEP will drive the deployment of a new generation of exensible agents (such as W3C's Jigsaw server and libWWW reference library). NOTE: PEP has been developed over the last year by many interested parties, including the W3C/CommerceNet Joint Electronic Payments Initiative (JEPI). Two strongly related drafts are also available from the IETF and W3C: Don Eastlake's Universal Payment Preamble [6] and ``Selecting Payment Mechanisms Over HTTP (or, seven examples of UPP over PEP)'' [9]. 1.1 EXAMPLE: PICS Consider the example presented by the first deployed PEP application, the Platform for Internet Content Selection (PICS) [11]. PICS describes a format for labeling almost any content. Since it's intended to be a fairly ubiquitous addition to various Internet protocols, PICS appropriates the PICS-Label: header for 822 applications, including mail, news, and web content. The spec could not use, say, Label:, since there's no way to effectively arbitrate between conflicting uses of header names (PICS was not developed under the IETF standards track). Even with an eventual dispensation from IANA to use this new header, new issues arise: If a server attaches this new header to label an HTTP response, will a caching server pass it along? Will it act upon the label at all? What kind of feedback will the end-user get when this ``unknown'' header appears? How can the user specify which rating systems to report? The direct fix to the first few problems is to add a machine-understandable statement about the extension and its new headers. Protocol: headers describe the name of the extension, the scope, importance, and any associated data headers. Moving further, though, additional information about the protocol can be added to that directive: which PICS services were employed, etc. Using PEP in this way also solves the next logical problem: how can a client request a server to initiate vending PICS labels? The same structure for a Protocol: can be reused in a Protocol-Request: asking the counterparty to start using the specified extension. In addition, PEP affords additional capabilities, like using Protocol-Query: to test if PICS is available and Protocol-Info: to advertise it. In all cases, the PICS protocol extension can specify additional parameters, such as which rating systems, services, which kinds of labels, and which other URLs are labelled. 2. The PEP Model Protocol extensions can modify HTTP messages, most commonly by adding headers, but also by modifying the contents, much like a proxy server might do today. There are four points at which PEP processing occurs in a basic HTTP transaction (see Section 3 for usage descriptions at each step): 1. Sending an HTTP Request 2. Receiving an HTTP Request 3. Sending an HTTP Reply 4. Receiving an HTTP Reply During a conversation, either side may invoke a Protocol:, issue a Protocol-Request:, advertise some Protocol-Info:, or test a Protocol-Query:. PEP-compliant recipients decode extended messages and reply to each Protocol-Request: and Protocol-Query:. Unlike some previous extension proposals [9], and unlike some existing HTTP/1.1 features (Upgrade: and Connection: tokens), extensions are attributes of a resource rather than a server. This means all PEP statements are applied to the single resource mentioned in any HTTP message (though there is a facility to hint at which other resources the same statement applies to). An HTTP agent can begin using a protocol extension (or query for one) because: * It is configured to do so by a user or webmaster * It has been requested to do so by the other party * The other party has hinted that an extension is available * Someone else has hinted that an extension is available Another aspect of the PEP model is that protocol extensions can have> negotiable parameters. Unlike previous proposals which model extensions as binary have/don't have features, PEP expects protocol extensions will be able to handle different versions of a particular protocol, invocation of ``compatible protocols'', and the selection of compatible modes from several that might be advertised. To summarize, PEP as proposed here will allow HTTP/1.2 implementors to make the following kinds of statements: * ``This agent has and is using (or does not have and isn't using) the http://w3.org/DES protocol extension with mode parameter CFB and header DES-Info:'' * ``This agent requires/requests/forbids the recipient to begin using the http://w3.org/DES protocol extension with mode parameter ECB'' * ``This agent has/requires/refuses the http://pics.org/PICS-1.1 protocol extension available for any URIs beginning with http://www.my.site/homepages/'' * ``Does the recipient have the http://pics.org/PICS protocol extension available for service parameter http://rsac.org/rsaci ?'' * ``This agent has applied two protocol extensions to this message in this order.'' * ``This agent is using the `generic' http://w3.org/UPP payment negotiation extension via the payment protocol http://cybercash.com/CCCP.'' 3. PEP Usage Before presenting specifics about each of the four PEP directives and the four points where they are processed, here are a few common points: * Attribute-value bags are used to represent all the data carried in PEP headers. * Each directive specifies which agent must process it and the consequences of not processing it. * Each directive is associated with a set of resources: either the resource represented by the HTTP request or reply, or a list of resource and resource prefixes it applies to. A bag is an attribute-value list, where bags are allowed recursively as values as well. In PEP, each directive begins with values describing the protocol (making the extension name the top attribute). Some of those values are common to all directives: the strength (str), scope (scope), for list (for) , extension-specific parameters (params), and other headers associated with this extension (headers). The formal BNF is included in Section 3.5.1. Strength and scope are important common concepts. Any directive can be ignored, but strength specifies whether an error should be generated as a result. Similarly, scope defines which agents should handle the directive at all; either the immediate (conn) or final (origin) recipient. The combination of the two concepts makes PEP powerful enough to deploy features that in the past required a minor-version number bump. With PEP, for example, Content-Transfer-Encoding: could be described as {str req} {scope conn} for rapid experimentation with compressed headers, sticky headers, etc. A word about the for list: Since each PEP directive applies only to the resource at hand (rather than, say, the entire server or all resources of a given type), a hint as to which other resources the directive applies to will prevent negotiation round-trips each time. Thus, many PEP applications could include {for /*} to apply to an entire server. Remember, there already are examples in HTTP/1.1 of features which only apply to particular resources, such as Range:. When fetching a URL, a client will typically search through its database of hints to find all applicable directives before sending an HTTP request. Of course, an agent which does not want to use hints can ignore them: if the for list includes the URI of the current message, just throw away the list; if it does not, just throw away the entire directive. Finally, note that the Protocol-Query: / Protocol-Info: pair describes hypothetical capabilities of agents and resources but that the Protocol-Request: / Protocol: pair has immediate consequences for the current HTTP transaction. Each pair hangs together and must be implemented together, though only the latter pair is mandatory. 3.1 PROTOCOL-QUERY The first statement in a conversation might be to ask if the counterparty supports an extension at all. The Protocol-Query: header lets one agent ask the other if a particular extension is supported; the response comes back in a subsequent Protocol-Info:. A query directive specifies its scope, but not a strength, which defaults to opt; that's because a query always asks optionally for more information, but can neither demand an answer (since query support is optional), nor refuse one. An agent can query if an extension is supported for an entire list of URIs. This means that an answer is expected when the replying agent sends a message about any resource on the list; and that an answer should reveal as much as can be said about extension support for that URI set. This way, a server can ask for a client to be prepared to use an extension in a part of his site (e.g. UPP in /Catalog/*); or a client can ask a server if an extension applies to all its documents of the same type (e.g. indexed range retreival for *.pdf). 3.1.1 Sending an HTTP Request If a client believes an extension may be supported, it can attach a Protocol-Query: to an outgoing request to test if an extension is supported for that URI (empty for list), an entire server ({for /*}) or something in between. A client can wait to piggyback its queries on other HTTP transactions or it can explictly issue an OPTIONS request with its queries. Note that a protocol extension might be defined to have side-effects when it is queried. For example, the UPP payment selection protocol defines that the answer to any queries for UPP should include additional information about particular payment mechanisms. Thus, in a dynamically extended agent an agent should call back the extension to handle a query. 3.1.2 Receiving an HTTP Request When an agent receives a query, it should respond whether the stated configuration is acceptable or refused for any of the URI(s) specified. Upon receipt of a query, an agent should test if the extension is available and, if so, its response to the proposition. The server should be prepared to send the resulting information in its reply. 3.1.3 Sending an HTTP Reply In preparing an HTTP reply, a server must respond to queries in the HTTP request, as well as issue its own queries on the reply. For the responses, see Section 3.2 below. 3.1.4 Receiving an HTTP Reply The answers returned to the client are in the form of Protocol-Info:; see Section 3.2. The server may have attached queries of its own; the client should prepare an answer and attach it to the next HTTP request it makes for a URI which matches the realm of the query. Here, too, a dynamically extended client should immediately call back an extension, since an extension's response to Protocol-Query: may result in immediate side effects (e.g. a payment selection interface appearing with highlighted possible selections). 3.2 PROTOCOL-INFO The second statement in a conversation might be to declare that one party requires/allows/forbids a given protocol extension, either unilaterally or in response to an earlier query. The Protocol-Info: header lets one agent tell the other if a particular extension is supported; there is no required response. Remember, though, that Protocol-Info: itself is the required response to Protocol-Query: (see Section 3.1). An informational response states if the protocol is required, allowed, or forbidden for all matching URIs. 3.2.1 Sending an HTTP Request A client will add Protocol-Info: directives if required by a previous server query (Section 3.1.4) or if it has any unilateral declarations to advertise. 3.2.2 Receiving an HTTP Request A server should take note of any available information; knowledge of a client's capabilities is important for step #3 (Section 3.3.3 and 3.4.3). 3.2.3 Sending an HTTP Reply A server will add Protocol-Info: directives if required by a previous client query (Section 3.1.1) or if it has any unilateral declarations to make. 3.2.4 Receiving an HTTP Reply A server should take note of any available information; knowledge of a client's capabilities is important for step #1 (Section 3.3.1 and 3.4.1). The response can be to either ignore this information, request initiation, or directly initiate the extension on the next request. 3.3 PROTOCOL-REQUEST The third statement in a conversation might be to directly demand or forbid that the counterparty begin using an extension. Protocol-Request: lets one agent tell the other to initiate a particular extension; the response requires a subsequent Protocol: directive for each request. 3.3.1 Sending an HTTP Request When a client wants to tell the server to begin using some extension, it should check if it knows that the extension will work because of stored Protocol-Info:. The Protocol-Request: describes the desired protocol; it may include a range of parameters for the respondent to pick and choose from. 3.3.2 Receiving an HTTP Request A server must test if a request will succeed. If so, the server must begin using the specified extension. If it cannot, the preferred mechanism is to echo back the same Protocol-Request: but with {str ref}. In addition, if the original request was {str req}, then indicate that the request failed with a 42x client or 52x server error status code (Section 3.5.2). 3.3.3 Sending an HTTP Reply A server must reply to the client's request to require, accept or forbid an extension by either initiating or preventing use of the specified protocol or a compatible equivalent (Section 3.4.3). At this point, a server may also attach its own requests for a client to initiate extensions when accessing this URI or others. Note that if a server Protocol-Request: later fails to elicit a Protocol: response from the client, nothing can be assumed about the client's ability to do PEP, since it may have ``forgotten'' the request in the meantime. 3.3.4 Receiving an HTTP Reply The client must evaluate each server Protocol-Request: and, if valid, be prepared to use the specified protocol extension when accessing any URI matching the request's for list (Section 3.4.1). If it is not willing or able to use the specified protocol, it must echo back an error on the next access as in Section 3.3.2. Note that if a client Protocol-Request: fails to elicit a Protocol: or PEP error response (assuming the response URI is in the request's realm) at, the client must conclude the origin server is not PEP-compliant. 3.4 PROTOCOL The final statement in a conversation might be to use an extension directly. Protocol: lets one agent declare that the particular extension is in use for the current message; the response may be to continue using the extension. An agent can apply an extension at any time; the preceeding three conversational steps of testing and requesting are only an example. This directive packages together any added header lines and any reprocessing of the message body. Since HTTP/1.1 eliminates many earlier ambiguities about the ordering of headers, we can state categorically that protocol directives must be dispatched in the order they are listed in the Protocol: header. [The Content-Encoding: pipelining mechanism of earlier proposals has been scrapped]. It is worth noting that the application of a protocol extension can affect the cacheablity of the response. This document does not define a cache control directive, but expects it to be feasible within the 1.1 model. Connection-scope extensions can remain backward-compatible by listing every associated header and the name of the extension in the Connection: header. 3.4.1 Sending an HTTP Request The client must decide which extensions to apply to the current request: it must respond to each applicable Protocol-Request:, it can continue using extensions the server has already used for the request-URI, and it may initiate others. As each extension is applied, added headers are referenced from the Protocol: line in order. If some Protocol-Request: is satisfied by some other protocol extension compatible with the one requested, the client will issue two protocol directives, one for the extension used, and another indicating the extension requested is being used via the other. 3.4.2 Receiving an HTTP Request The server evaluates each protocol extension which is in scope according to the order it is listed in the Protocol: header. As the server evaluates each protocol, it can interrupt and return HTTP error status codes: if a client encoding error is detected (42x), the server can't or won't decode properly (52x), or if a protocol is used inappropriately (it's refused, not allowed in conjunction with certain other protocols, missing, etc.). Proxy servers operate similarly: they can simply remove unknown optional connection-scope directives but must report errors for required directives. 3.4.3 Sending an HTTP Reply The server must also choose protocols it will use from the sets of those requested by the client, used by the client, and those it will initiate independently. If it cannot construct a valid reply due to contradictions among those requirements, it must return HTTP error status 520 and enumerate the server's constraints in Protocol-Request: headers of the error message. 3.4.4 Receiving an HTTP Reply The client evaluates each protocol extension which is in scope according to the order it is listed in the Protocol: header. As the client evaluates each protocol, it could fail for many reasons. One recovery technique is to reissue its request with the offending protocol refused in a Protocol-Request: header. Errors from executing required extensions should be reported to a user or log. 3.5 SYNTAX PEP-related syntax is specified here relative to the definitions and rules of the HTTP/1.0, HTTP/1.1, and the relative URL specification. 3.5.1 PEP Headers PEP defines two new required general header fields, Protocol: and Protocol-Request:, as well as two new optional general header fields, Protocol-Query: and Protocol-Info:. /* Added to General Header rule, Sec 4.3 of HTTP/1.1 */ Protocol = "Protocol" ":" 1#bag Protocol-Request = "Protocol-Request" ":" 1#bag Protocol-Query = "Protocol-Query" ":" 1#bag Protocol-Info = "Protocol-Info" ":" 1#bag /* Following rules are copied from Section 2.2 of HTTP/1.1 */ bag = "{" bagname 1*LWS *bagitem "}" bagname = token | URI bagitem = bag | token | quoted-string word = token | quoted-string token = 1* tspecials = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <"> | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT quoted-string = ( <"> *(qdtext) <"> ) qdtext = and CTLs but including LWS> Each bag has several top-level reserved attributes defined: { {scope (origin | conn)} {str (opt | req | ref)} {headers *} {params ...} {for *} {via }} protocol-identifier The well-known protocol extension URI or an IANA-registered token. scope scope is either of "conn" or "origin" ; the default is "origin". It defines which agent must recognize this directive: the immediate recipient or the origin client/server. str strength is one of "req", "ref" or "opt" ; the default is "opt". In particular, a protocol directive only allows req or opt; request and informational directives allow all three; and queries only allow opt. headers A list of associated message headers; the default is the empty set. If a header token ends in "*", it must be used to refer to all message headers matching that prefix. Each directive enumerates its own associated headers to disambiguate `ownership'. This information is critical when 1) reporting an error, 2) stripping an extension off of a message (if possible), and 3) allowing multiple applications of an extension. params A list of parameters configuring the extension, according to the protocol specification; the default is empty. for a list of relative or absolute URIs [1,2,7] which a protocol request is binding upon; the default when the for list is empty, is the request-URI or Location: response header. If a reference ends in "*", it should be understood as a wild-card character (particular implementations may choose to only recognize restricted applications of "*", e.g. `only in the last component of http: URLs'). via When responding to a request addressed to one protocol extension (a query or request directive), an agent may answer via another (more specific, upgraded, compatible) extension (token or URI). The latter extension is expected to be used elsewhere in the same message. This is a hint and only allowed in the Protocol: and Protocol-Info: directives. The default is empty. Note: Multiple occurrences of PEP-defined attribute names in a list ("{str req}...{str ref}") will yield undefined behavior. 3.5.2 PEP Status Codes PEP defines several new status codes for HTTP replies. Note that the HTTP/1.1 specification states in Section 6.1.1: The first digit of the Status-Code defines the class of response. The last two digits do not have any categorization role. Informally, PEP distinguishes x2z response codes: 200 Class 220 Uses Protocol Extensions 400 Class 420 Bad Protocol Extension Request 421 Protocol Extension Unknown 422 Protocol Extension Refused 423 Bad Protocol Extension Parameters 500 Class 520 Protocol Extension Error 521 Protocol Extension Not Implemented 522 Protocol Extension Parameters Not Acceptable Each of 400 and 500 class responses may include entity bodies with an explanation of the error, and an indication of whether the problem is temporary or permanent. A 220 response should be used only when some PEP feature is required for correct handling -- it affects the cacheability of the response. 4. Conclusions This is a discussion draft, not a complete specification. In the meantime, this will be the basis for all further JEPI work. For additional details and explorations of various issues (e.g. security concerns), readers are encouraged to study the earlier drafts, available at http://www.w3.org/pub/WWW/TR/WD-http-pep. 5. References [1] Berners-Lee, T., "Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web", RFC 1630, CERN, June 1994. [2] Berners-Lee, T., Masinter, L., and M. McCahill, Editors, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox Corporation, University of Minnesota, December 1994. [3] Jim Gettys, et al, "Hypertext Transfer Protocol -- HTTP/1.1". Internet Draft W3 Consortium/MIT, August 1996 (Work in Progress). [4] D. H. Crocker. "Standard for the Format of ARPA Internet Text Messages." STD 11, RFC 822, UDEL, August 1982. [5] S. Deering, R. Hinden, Editors, "Internet Protocol, Version 6 (IPv6) Specification", RFC 1883, December 1995. [6] D. Eastlake, "Universal Payment Preamble", Internet Draft CyberCash, March 1996 (Work in Progress). [7] R. Fielding. "Relative Uniform Resource Locators." RFC 1808 , UC Irvine, June 1995. [8] JEPI, "Selecting Payment Mechanisms Over HTTP", Internet Draft, August 1996 (Work in Progress). [Also available as http://www.w3.org/pub/WWW/Payments/JEPI/draft-jepi-uppflow-00 .html] [9] D. Kristol, "A Proposed Extension Mechanism for HTTP", Internet Draft, January 1995 (Work in Progress, Expired). [10] J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker. "SMTP Service Extensions." RFC 1869. MCI, Innosoft, Dover Beach Consulting, Network Management Associates, Brandenburg Consulting, November 1995. [11] J. Miller. "Label Syntax and Communication Protocols (Version 1.1)." W3C Proposed Recommendation PR-PICS-services , W3 Consortium/MIT, August 1996. [12] J. Postel, J. Reynolds, "Telnet Protocol specification." STD 8, RFC 854, USC/ISI, May 1983 6. Author's Address [Note: a great many people have participated heavily in the research, design, development, and description of the ideas presented here. Future revisions will sort out the contributions of various folks, likely adding co-authors &c. In the meantime, I will be the single contact point for this draft. Many, many thanks to these folks -- you know who you are!] Rohit Khare Technical Staff, W3 Consortium MIT Laboratory for Computer Science 545 Technology Square Cambridge, MA 02139, U.S.A. Tel: +1 (617) 253 5884 Fax: +1 (617) 258 5999 Email: khare@w3.org Web: http://www.w3.org/People/Khare