Caching subgroup of HTTP working group Jeffrey Mogul/DECWRL Internet-Draft 15 February 1996 Expires: 20 August 1996 A design for caching in HTTP 1.1 Preliminary Draft draft-mogul-http-caching-00.txt STATUS OF THIS MEMO THIS IS A PRELIMINARY DRAFT. IT DOES NOT AT THE MOMENT REPRESENT THE CONSENSUS OF THE CACHING SUBGROUP OF THE HTTP WORKING GROUP, OR OF THE HTTP WORKING GROUP. This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress". To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). WARNING: The specification in this document is subject to change, and will certainly change. It is inappropriate AND STUPID to implement to the proposed specification in this document. In particular, anyone who implements to this specification and then complains when it changes will be properly viewed as an idiot, and any such complaints shall be ignored. YOU HAVE BEEN WARNED. Distribution of this document is unlimited. Please send comments to the HTTP working group at . Discussions of the working group are archived at . General discussions about HTTP and the applications which use HTTP should take place on the mailing list. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 1] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 ABSTRACT I propose a caching design for HTTP 1.1, including the definition of terms, the description of a model, the specification of related HTTP headers, and the algorithms to be used. The proposed design uses opaque cache validators and explicit expiration values to allow the server to control the tradeoff between cache performance and staleness of the data presented to users. The server may choose to ensure that a user never unwittingly sees stale data, or to minimize network traffic, or to compromise between these two extremes. The proposed design also allows the server to control whether a client sees stale data after another client performs an update. This draft is not intended to become a standards-track document. The ultimate output of the HTTP subgroup will be a set of changes to the HTTP/1.1 draft specification [1]. In the interests of making rapid forward progress and in capturing some of the motivation and rationale of our design(s), it seems useful to produce an Internet-draft document in the interim. Note that this draft in its entirety does not currently represent the consensus of the subgroup, although several sections do represent an apparent consensus. TABLE OF CONTENTS 1 Introduction 3 2 Philosophy: make everything work, optimize for common cases 4 3 Caching model 5 3.1 Overview 5 3.2 Definitions 5 3.3 Goals 8 3.4 Cache keys 9 3.5 Cache validation and the ``immutable cache'' concept 9 3.6 Opaque validators 11 3.7 Cache operation when receiving validator from client 13 3.8 Cache operation when receiving errors or incomplete 14 responses 3.9 Bounding freshness 14 3.9.1 Disambiguating expiration values 16 3.9.2 Disambiguating multiple responses 17 3.10 Explicit Cache-control 18 3.11 Warnings 20 3.12 Compatibility with earlier versions of HTTP 21 DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 2] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 3.12.1 Compatibility between HTTP/1.1 clients and older 21 servers 3.12.2 Compatibility between HTTP/1.1 servers and older 21 clients 3.12.3 Compatibility between HTTP/1.1 and older proxies 21 3.13 Update consistency 21 3.14 Volume validation 23 3.15 Side effects of GET and HEAD 25 3.16 Write-through mandatory 26 4 HTTP headers related to caching 26 4.1 Cache-validation 26 4.2 Expiration values 27 4.3 Age values 28 4.4 Cache-control 28 4.5 Warnings 29 4.6 Volume validation 29 5 Rules summarized by method 30 6 Cache replacement algorithms 30 7 Specific issues 30 7.1 Content negotiation 30 7.2 Authentication 30 7.3 State 30 7.4 Caching and methods with side effects 31 7.5 Network partitions 31 7.6 Caching of negative responses 31 7.7 History Lists 31 7.8 Hit Metering 31 7.9 Bypassing in caching hierarchies 32 7.10 Explicit warnings regarding user-specified overrides 33 8 Security Considerations 33 8.1 Caching of responses to authorized requests 33 8.2 Location: headers and spoofing 34 9 Acknowledgements 36 10 References 36 11 Author's address 36 1 Introduction The World Wide Web is a distributed system, and so its performance can be improved by the use of caches. These caches are typically placed at proxies and in the clients themselves. The use of caches introduces two problems: how do we ensure that the caches return the ``correct'' values, and how do we obtain the best possible performance. In general, there may be a tradeoff between how ``correct'' the returned values are, and how effectively the caches perform. At issue in the specification of HTTP, therefore, is the design of a caching mechanism that produces the most appropriate possible DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 3] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 tradeoff between correctness and performance. In this document, I propose a design that uses opaque cache validators and explicit expiration values to allow the server to control the tradeoff between cache performance and staleness of the data presented to users. The server may choose to ensure that a user never unwittingly sees stale data, or to minimize network traffic, or to compromise between these two extremes. One can get into lengthy debates about what ``correct'' means with respect to HTTP caching. To sidestep such pointless debates, I will avoid using the word ``correctness'' and use the specific term ``semantic transparency.'' A cache is completely semantically transparent if clients cannot tell that it exists, except for the speed at which responses are delivered. Part of the semantic transparency problem is that when multiple copies of a value exist, and one copy is updated, a client using another copy may not see the update. This can lead to inconsistencies. The proposed design also allows the server to control whether a client sees stale data after another client performs an update. Server-based control is also important because HTTP may be used for a wide variety of ``applications.'' The design of a Web application (for example, a stock-trading system) may be peculiar to the server, while Web browsers are generic to all Web applications. Because the precise behavior of an application cannot be known to the implementor of a browser, but can be controlled by the implementor of a server, servers need to have the option of direct control over the caching mechanism. Because the world is not perfect, we also need to give users and browsers some control over caching, but this is at best a contingency plan. 2 Philosophy: make everything work, optimize for common cases Applications in which HTTP is used span a wide space of interaction styles. For some of those applications, the origin server needs to impose strict controls on when and where values are cached, or else the application simply fails to work properly. We can refer to these as the ``corner cases.'' In (perhaps) most other cases, on the other hand, caching does not interfere with the application semantics. We can call this the ``common case.'' Caching in HTTP should provide the best possible performance in the common case, but the HTTP protocol must entirely support the semantics of the corner cases, and in particular an origin server must be able to defeat caching in such a way that any attempt to override this decision cannot be made without an explicit understanding that in doing so the proxy or client is going to suffer from incorrect behavior. In other words, if the origin server says DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 4] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 ``do not cache'' and you decide to cache anyway, you have to do the equivalent of signing a waiver form. We explicitly reject an approach in which the protocol is designed to maximize performance for the common case by making the corner cases fail to work correctly. 3 Caching model In this section, I present a model for HTTP caching but do not provide detailed specifications for headers and algorithms. 3.1 Overview The caching model has these major design elements: 1. The basic protocol provides a way to enforce strict semantic transparency; if servers, caches, and clients follow these rules, then users will never see something other than what the server intends. 2. The protocol provides ways for the servers, caches, and clients to loosen the strictness rules. In effect, the server grants permission for a cache to behave loosely. The amount of looseness is explicitly bounded by the participants, and user agents are always aware if a response may be the result of a loosening of the rules. In this proposal, loosening is done by bounding the amount of time that an invalid cached response can be used, but other techniques might be developed. 3. A cache can check if a cached value is valid by checking with a server, and only reloading the value if the cached version may be invalid. This is done using ``conditional'' HTTP methods, whose semantics depends on the validity check. 4. The protocol includes an explicit Cache-control mechanism that allows servers or clients to directly control aspects of caching. 5. The protocol includes a warning mechanism to transmit indications of possibly non-transparent caching. 3.2 Definitions Here are some useful terms that need specific definition. First, we have found it useful to distinguish between various places where responses, or pointers to responses, might be stored, and between idealized and practical instances of each: DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 5] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 ideal cache exists to model the origin server's responses to requests. Semantically transparent to the user for all practical purposes, but should improve performance according to one or more metrics. ideal history buffer records what the user has seen during past interactions, and provides a means for the user to revisit those views exactly as they were seen. ideal location log records what locations the user has visited during past interactions; provides a means for the user to revisit those locations to discover their current contents. practical cache exists to model the origin server's responses to requests. Semantically transparent to the user, except that the origin server, user, or cache administrator may choose to compromise transparency in the service of performance or availability. Should improve performance according to one or more metrics. practical history buffer records what the user has seen during some subset of past interactions, and provides a means for the user to revisit those views exactly as they were seen, or as best as can be approximated given finite buffer space. practical location log records a subset of the locations the user has visited during past interactions; provides a means for the user to revisit those locations to discover their current contents. Note that we are not aware of any existing implementations that make such clear distinctions, but they seem useful for the purposes of specifying protocol behavior. The rest of this document specifies the behavior of caches, not history buffers or location logs. We leave the behavior of a history buffer up to the browser implementor. If a browser confuses the distinctions made between caches and history buffers, then any resulting erroneous behavior is not the fault of the servers or proxies. A browser implementor may choose to use the browser's cache to approximate the history buffer, in which case it is possible that a user will see something other than the ``historically DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 6] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 accurate view'' of a page, but that is between the user and the browser vendor; it is not something that the HTTP protocol specification can define. This includes issues such as ``does `subset' mean terminal subsequence?'' and does `` `approximated' mean one can retrieve a new copy of the page if the old view didn't fit into the buffer?'' Similarly, we do not specify the implementation of a location log, although we recommend that it include a terminal subsequence of the locations visited. XXX Shel Kaphan may add something more specific here. conditional method An HTTP method whose semantics depends on the value of one or more request headers. cached object The unit of caching, which consists of an entity body, entity headers, perhaps other response headers, perhaps some related request headers, and perhaps the request entity body. cache key The portion of a cached object that the caching algorithms use to determine if the cached object matches a request. document expiration time The time, if any, at which an entity should no longer be seen by a user. valid A cached entity is valid, with respect to a given request at a given time, if it is exactly what the origin server would return in response to that request at that time. invalid A cached entity is invalid, with respect to a given request at a given time, if it is not exactly what the origin server would return in response to that request at that time. unknown validity A cached entity is of unknown validity if the cache does not know whether it is valid or invalid. firsthand A response is firsthand if it comes directly and without unnecessary delay from the origin server, perhaps via one or more proxies. A response is also firsthand if its validity has just been checked directly with the origin server. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 7] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 expiration time The time at which the origin server intends that an entity should no longer be returned by a cache unless the cache first checks with the origin server to see if the cached object is valid. All cached objects have an expiration time, but this might be ``Forever'' (infinitely in the future), ``already'' (in the past), or ``undefined.'' fresh A cached entity is fresh if it has not reached its expiration time. stale A cached entity is stale if it has reached its expiration time. Stale entities are not necessarily useless. inbound towards the origin server; the request chain flows inbound. outbound towards the ultimate client system; the response chain flows outbound. 3.3 Goals This caching design values semantic transparency over performance. That is, performance considerations are not allowed to overrule transparency considerations in the design itself. In actual operation of HTTP, it is possible for servers, proxies, or clients to explicitly choose performance over transparency, but the protocol does not force this choice on them. Some people have argued that a ``correct'' caching design must optimize performance as well as any semantic correctness. For the purposes of this document, I will explicitly OMIT performance from the definition of ``correctness.'' It is understood that a GOOD caching design includes good performance as well as some level of semantic transparency. This does NOT make performance part of the definition of semantic transparency. This depends on a clear definition of what it is to be ``semantically transparent.'' I will start with this definition: A response returned by a cache in response to a request is a semantically transparent result if and only if it includes exactly the same entity headers and entity body that would be returned by the origin server to the same client making the same request at the same time. A cached response that includes slightly different general headers (see section 4.3 of [1]) may still be transparent. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 8] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Another way to look at transparency is that: A user should never have to take explicit action to see exactly the version of an entity that the server wants the user to see. Semantic transparency means that the only user-visible effect of a cache is a probable improvement in performance. We do not insist on semantic transparency in all cases, but we insist that if the user and server both want semantic transparency, then the caches MUST provide it. Users should not have to use ``Reload'' buttons to simply assure themselves that they are seeing valid copies. One will still have to use ``Reload'' to refresh the page currently on your screen, but only to indicate to the user agent that you have finished reading it and are willing to see a new copy, if any. I now state three goals: - The protocol must allow semantic transparency, in which the user is guaranteed to see a valid copy of an object unless some network failure prevents it. - The protocol must allow users, servers, and proxies to choose performance over transparency if they want to, but must not force them to do so. - When the protocol delivers a response that may be non-transparent, the client must made be aware of this. Because the world is already full of of HTTP implementations, we also must require that a HTTP/1.1 implementation must interoperate with pre-1.1 implementations, without being any less transparent than a pre-1.1 implementation would be, and without being significantly slower. 3.4 Cache keys The first step in using a cache is to decide if the requested entity is present in the cache. This is done by searching the set of cached objects for one with a cache key that matches the request headers. For now, we ignore the problem of how the cache matches requests against its cached objects. This is covered in section 7.1. In order to understand the other aspects of the model, one can pretend that the cache key is simply the URL of a cached object. 3.5 Cache validation and the ``immutable cache'' concept The primary mechanism for ensuring semantic transparency is cache validation. When a downstream client makes a request via a cache, the cache must not return a cached entity unless it is fresh, or the cache has checked with the inbound server that its cached object is DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 9] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 valid. A cache may always return a firsthand (uncached) response, of course. When multiple caches are involved, the validation check may be answered by a outbound cache, but in most cases will be answered by the origin server. To restate this: a cache may not return a stale entry without first checking its validity, unless the user or cache administrator has explicitly allowed the use of stale responses. This differs from the language in section 10.19 of the draft HTTP/1.1 specification [1], which says ``Applications must not cache this entity beyond the [Expires:] date given.'' That section should be reworded to say ``Applications must not return this entity from their cache after the date given, without first validating it with the origin server.'' This changes the spec to make it clear when a cache must use a conditional method before responding to a request. In HTTP, a client may say to a server ``tell me if my cached copy is valid, and if it is not, please return a valid copy.'' If a cached object is found to be invalid, the cache normally will receive a new copy of the object from the server, return that firsthand (and presumptively valid) copy to the client, and may enter the new copy into the cache. Such requests are normally done using the GET method (although this mechanism might apply to other methods), and if so, this is referred to as a ``conditional GET.'' Note: the current HTTP/1.1 draft says that there is no such thing as a ``conditional HEAD'' request. Do we want/need to retain this restriction? Or is there some possible value to a conditional HEAD? This mechanism, all by itself, yields semantically transparent caching in all cases, although it may not yield optimal performance. Note that it says nothing at all about removing objects from the cache. One could in theory implement a cache as an immutable time-ordered log of cache objects, where once an object is entered into the cache, it stays there forever. Because this ``immutable cache'' is a time-ordered log, only the most recent entry with a given cache key will match a request. While the log will sometimes contain stale, invalid entries that are not hidden behind newer entries with the same key, this is not a problem for transparency because the cache will never return such entries to a client. The performance of an infinite immutable cache would not actually be that bad, if we ignore the cost of doing a key lookup in an infinite database. Clearly, the cache-hit ratio would be optimal, since all misses would be compulsory or coherency misses, not capacity misses. Consider a cache that discarded invalid cache entries. In this case, DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 10] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 the server would return the same number of object copies, but it would do fewer validity checks and so might see slightly less load. However, in most cases an invalid entry would immediately be hidden behind a newer copy of the entity, and so these would seldom be found on a cache lookup. The only case that seems to matter is when a subsequent request arrives at the cache before the retrieval is complete; performance considerations suggest that the cache should not make parallel requests to the server for the same URI. Note that this design does not involve any explicit flushing of cache entries; in fact, the ``immutable cache'' model is incompatible with flushing (that is, removing one or more entries). Cache-flushing has been proposed to deal with updates, removals, and other changes, but it is in fact both unnecessary and insufficient. This will be discussed in section 3.13. Some people have suggested the adoption of a mechanism by which a server can explicitly revoke a permission it had previously given to cache something loosely. This might be done, for example, using a callback mechanism. This is a promising approach but requires more consideration than is feasible for HTTP/1.1. However, section 3.14 presents a technique that should make revocation less necessary. 3.6 Opaque validators Section 3.5 states ``the cache must not return a cached entity unless it is either fresh or the cache has checked with the inbound server that its cached object is valid.'' How does the cache perform this check? In the existing protocol (HTTP/1.0 and earlier), the cache can use the If-modified-since: header to instruct the server to send a full response only if the entity has not changed since the cache entry was made. We must retain this mechanism in HTTP/1.1 to provide compatibility with existing implementations, but If-modified-since has at least three major faults: 1. This header uses the HTTP-date format to express a time value. HTTP-date has a resolution of 1 second, and so is not sufficient to detect if a document is changing more frequently than that. For example, if client retrieves a document with a modification time of 9:08:07.01, and then the document is modified again at 9:08:07.04, the If-modified-since: mechanism will fail to detect the change. 2. This forces the server to maintain modification times for all cachable entities. While a traditional file-system based HTTP server can do this fairly easily, if the server stores entities in a relational database, or constructs them in some other manner, it may be inconvenient for the server to store modification-date values with every entity. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 11] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 3. On some popular operating systems, including UNIX, renaming a file does not change its modification date. This can confuse the if-modified-since logic. For example, if a URL initially points to file A with a modification time of 9:07, and then (through server-side renaming operations) the same URL later points to an older file B with a modification time of 9:06, a client doing a ``GET if-modified-since'' with a comparison date of 9:07 will not get the new value of the resource. A proposal was made in [1] to allow the client to do the cache-validity check using a general predicate on all header fields (the ``logic bag'') approach. While this avoids some of the problems with If-modified-since, it still does not give the server full flexibility, since the choice of how to determine validity is essentially left up to the client. This also makes the implementation of logic bags mandatory, and such an implementation could be complex or slow. While logic bags may be useful for other purposes, they seem inappropriate for checking cache validity. Instead, we can use an ``opaque validator'' mechanism to give the server full flexibility, while making the implementation as simple as possible. In this approach, the server returns (using a new response header) a ``cache validator'' value with each entity. A client MUST NOT do anything with this value except to store it with a cached object, and to present it to the server as part of a cache-validation check. A cache may compare two validators in certain circumstances, but MUST NOT do anything else except for storing or presenting it. When the cache wants to check the validity of a cached object, it presents the cache-validator value to the server (using a new request header), along with the object's URI. The server then decides whether the cached object associated with that cache-validator value is still valid, and informs the cache (for example, by returning "304 Not Modified", or by returning the new value). When a cache receives a request, checks the validity of the cached object, and the server indicates that the object is still valid, the cache may then treat its cached response as a firsthand response, but only for the purpose of responding to the current request. An origin server can use almost any algorithm it chooses to decide if the validator indicates that the cached object is still valid. It need not use simple equality comparison; for example, the server may decide that although the cached entity is not exactly the same as the current copy, it is close enough for any practical purpose. However, validators are also used by intermediate caches to decide whether to forward a conditional request to the origin server, and caches can only perform equality comparisons (because validators are opaque). This puts a strict requirement on the validator: if two values for the entity associated with a given resource are semantically DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 12] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 different, then the associated validators MUST be different. In most cases, therefore, simple equality is the best comparison algorithm. The server could use any number of methods to construct cache-validator values. For example, it could simply use the modification date. However, since the value is opaque, it need not use the HTTP-date format; it could encode a 64-bit (seconds, nanoseconds) representation as 16 hex digits. Or it could use a fingerprinting algorithm, such as MD5. Or it could use the address of a LISP cell, concatenated with a garbage-collection generation number. And it could use different algorithms for different kinds of object. A cache or client MUST NOT assume that two cached entities with different URIs but identical cache validators are identical. For example, the server may be using modification times in its validator, and two different documents may have identical modification dates. An HTTP/1.1 server SHOULD return a cache-validator value with every response. If a server does not return a cache-validator with a response, but does return a Last-Modified: date, a client or cache SHOULD validate a cached object using the If-Modified-Since: header, as is done in HTTP/1.0. However, HTTP/1.1 clients MUST NOT use ``If-Modified-Since:'' if a cache-validator value was provided by the server. The provision of a cache-validator value is the server's only way of stating that the Last-Modified date is not a useful cache validator (but is being supplied for other reasons). XXX what if neither a validator or Last-Modified date? 3.7 Cache operation when receiving validator from client When a client makes a request to a cache, and includes a cache-validator value (that is, the request method is conditional on the cache-validator value), if the cache does not have a fresh cached object that matches the request, it simply forwards the request to the origin server (or the next inbound cache). If the request does match a fresh cached object, then the cache MUST check the offered cache-validator value against the one stored with the cached object. If the cache-validator value matches, then the cache can return ``304 Not Modified''. Otherwise, it MUST return a full response (either from its cache or from the server). The cache checks the cache-validator values by comparing them for equality, with the proviso that a null value can never compare equal to any value. If a request does not include a cache-validator, then it is not a conditional request and the cache MUST return a full response. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 13] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 3.8 Cache operation when receiving errors or incomplete responses A cache that receives an incomplete response (for example, with few bytes of data than specified in a Content-length: header) MUST not cache the response or any part of it. XXX We need to come up with rules for whether to cache responses with statuses other than - 200 OK - 206 Partial content - 300 Multiple Choices - 301 Moved Permanently - 304 Not Modified - 410 Gone which seem safe to cache, and - 302 Moved Temporarily which are cachable if the server explicitly allows it. 3.9 Bounding freshness The server bounds the looseness of caching for an entity by specifying how long it remains fresh. A server that wants to enforce complete strictness can set this duration to zero; in other cases, a server may allow an entry to be fresh for a definite period, or forever. Section 3.5 states ``the cache must not return a cached entity unless it is either fresh or the cache has checked with the inbound server that its cached object is valid.'' How does the cache decide what is fresh and what is stale? Recall that a cached entity is fresh until it reaches its expiration time, and is stale thereafter. The expiration time is provided by the server in the response headers of the initial response that loads the entity into the cache. It may also be updated as the result of a validation check that does not otherwise change the entity (e.g., a conditional GET that returns "304 Not Modified"). In order to avoid problems caused by incorrectly synchronized clocks, the protocol provides a means for caches to convey the amount of time (``age'') they have held a response. We compute the total allowed duration of a response's freshness by subtracting the origin server's Date: value from its Expires: value, and then use the response's age to decide if it is still fresh. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 14] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Each cache keeps track of how long it has stored a response without its being refreshed (which is done by obtaining a new expiration time from the origin server). We call this the ``age'' of the cached response. When a cached response is sent to a client, the response headers transmit the response's current age using a (new) ``Age:'' header. If a response has been stored by multiple caches along the path, each one adds its own Age: value. We allow a cache to coalesce multiple Age: headers into one such header, by performing the obvious addition. It is possible that a response may be held for some time by an HTTP/1.0 cache, which would not include the Age: header. This could lead to improper comparisons between ages and expiration values. Therefore, we adopt the rule that the initial value for a cached response's age is not zero, but rather max(0, ((now - date_value) - age_sum)) where ``now'' is the cache's current clock, ``date_value'' is the value of the origin server's Date: header, and ``age_sum'' is the sum of all of the Age: header values already carried by the response. This means that each HTTP/1.1 cache is responsible for correcting for any ``missing'' time that the response spent in an HTTP/1.0 cache, since the last time that response left an HTTP/1.1 system. - This correction mechanism adds extra age if the cache's clock is running ahead of the origin server's clock, and fails to correct for missing age if the cache's clock is running behind. However, it is the best that can be done in the presence of HTTP/1.0 caches. A similar algorithm allows the Age: values to be used to partially correct Expiration: dates in the presence of clock skew. Because of network-imposed delays, some significant interval may pass from the time that an origin server generates an expiration value, and the time it is received at the next outbound cache or client. If uncorrected, this delay could result in improperly low ages. We can correct for this by noting that the request that resulted in the returned Age: value must have taken place prior to the value's generation. Thus, when an Age: value is received, it MUST be interpreted relative to the time the request was sent, not the time that the response was received. This algorithm results in conservative behavior no matter how much delay is experienced. To decide whether a response is fresh, a cache performs this algorithm: DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 15] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 /* * age_sum * is the sum of all of the Age: header values received * by the cache with this response. * date_value * is the value of the origin server's Date: header * request_time * is the (local) time when the cache made the request * that resulted in this cached response * response_time * is the (local) time when the cache received the * cached response * now * is the current (local) time * expires_value * is the expiration value sent by the origin server * request_max_age * is the Cache-control: max-age value, if any, in the * request */ apparent_age = response_time - date_value; age_compensation = max(0, ((apparent_age) - age_sum)); local_age = (now - request_time) + age_compensation; total_age = local_age + age_sum; freshness_duration = max(0, expires_value - date_value); if ((total_age < freshness_duration) && (total_age < request_max_age)) { /* response is still fresh */ } else { /* response is stale */ } In section 4.2, we discuss how to transmit the expiration value, including an encoding for ``Forever'' (infinitely in the future) and one for ``undefined'' (the server explicitly grants the cache permission to use any reasonable heuristic). In section 4.3, we discuss how to transmit the Age: value. An HTTP/1.1 server SHOULD provide an Expires: value with every cachable entity, but if it does not, a cache uses a heuristic to decide how long to treat the response as fresh. See section 4.2 for constraints on this heuristic. 3.9.1 Disambiguating expiration values Because expiration values are assigned optimistically, it is possible that two caches may contain different fresh values for the same resource. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 16] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 If a client doing a retrieval receives a non-firsthand response for a resource that was already fresh in its own cache, and the Date: header in its existing cache entry is newer than the Date: on the new response, then the client MAY ignore the response. If so, it MAY retry the request with a ``Cache-control: revalidate'' directive (see section 3.10), to force a check with the origin server. XXX Why would a client make a request if it still has a fresh copy in its own cache, and not make that request demand revalidation? Perhaps because it is honoring a fresh-min directive? If a cache that is pooling cached responses from other caches sees two fresh responses for the same resource with different validators, it SHOULD use the one with the newer Date: header. 3.9.2 Disambiguating multiple responses Because a client may be receiving responses via multiple paths, so that some responses flow through one set of caches and other respones flow through a different set of caches, a client may receive responses in an order different from that in which the origin server generated them. We would like the client to use the most recently generated response, even if older responses are still apparently fresh. Neither the opaque validator nor the expiration value can impose an ordering on responses, since it is possible that a later response intentionally carries an earlier expiration time. However, the HTTP/1.1 specification requires the transmission of Date: headers on every response, and the Date: values are ordered to a granularity of one second. If a client does a [XXX conditional?] request for a resource that it already has in its cache, and the response it receives contains a Date: value that appears to be older than the one it already has in its cache, then the client SHOULD repeat the request unconditionally, and include Cache-control: revalidate to force any intermediate caches to validate their copies directly with the origin server, or Cache-control: reload to force any intermediate caches to obtain a new copy from the origin server. This prevents certain paradoxes arising from the use of multiple caches. If the Date: values are equal, then the client may use either response (or may, if it is being extremely prudent, request a new DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 17] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 response). Servers MUST not depend on clients being able to choose deterministically between responses generated during the same second, if their expiration times overlap. 3.10 Explicit Cache-control It is useful to provide an explicit mechanism to control certain aspects of the caching mechanism. The HTTP/1.1 draft includes a new Cache-Control: header, which can carry four kinds of directive: ``cachable'', ``max-age'', ``private'', and ``no-cache''. This can be sent in either direction (inbound or outbound), although the ``cachable'' and ``private'' directives are not allowed in requests. We replace the proposed ``cachable'' directive with the more explicit ``public'' directive, used on responses to indicate that a response to a request which included an Authorization header field may be returned from a cache entry. This overrides restrictions stated elsewhere in the specification on cache behavior for such requests, allowing a server that uses authentication for purposes other than limiting access to encourage caching. See section 8.1 for more details. XXX Action item: is ``public'' allowed on responses to any method, or do we need to restrict it? The ``no-cache'' directive on responses is useful, since it is a more polite way to defeat caching than simply returning a bogus cache-validator value and a past expiration time. With ``Cache-Control: no-cache'', the cache knows not to waste storage space on the entity. The ``no-cache'' directive on requests is also useful, because it allows a user agent (or other client) to override a cache's belief that a cached object is fresh. It is likely that some servers will assign expiration times that turn out to be optimistic, or that by some accident an incorrect cache-validator may be stored with the cached object, and clients need a way to force reloading. However, it seems unnecessarily economical to use the same directive for two purposes, and might be less confusing to use the name ``reload'' here. It also seems useful to provide a ``revalidate'' directive for requests, to force a cache to do a conditional GET on a cached object, but not necessarily to reload it. This would be useful if the expiration time is wrong, but the cache-validator is right. An over-optimistic expiration seems far more likely to happen than an incorrect cache-validator. The ``private'' directive, which is like a ``no-cache'' response except that it only applies to shared caches, is clearly useful. See section 8.1 for more details. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 18] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 The original proposal for ``no-cache'' and ``private'' allows them to specify particular fields of the response, but it is not clear that this is necessary or useful. It means that a subsequent request to the cache cannot use the entire cached entity. Does this mean that the cache should return only a partial entity (which seems relatively useless)? Does it mean that the cache should retrieve the entire entity from the origin server (in which case, why cache any of it)? Or does it suggest that the cache ought to be able to request a partial entity from the origin server to fill in the missing pieces (but the protocol does not currently include a means to do so)? XXX re-think this in the context of extensibility The ``max-age'' directive on responses effectively duplicates the expiration time, and (as currently specified) does not provide as much protection against timing delay, so it is unnecessary. The original purpose of the ``max-age'' directive on requests is to tell a cache to revalidate the value if it has been in the cache more than a specified number of seconds. This is basically a way for the client, rather than the server, to change the looseness bounds on a cached entity. We can use the same age-computation algorithm here as is used with expiration values (see section 3.9). In addition to ``max-age'', we can use two new directives for requests to allow the client to set looseness bounds. The ``fresh-min'' directive specifies the minimum number of seconds of remaining freshness necessary. For example, ``fresh-min 10'' means that the client only wants to see a cached response if it has at least 10 seconds of freshness left (measured by the cache). This allows the client to increase (make more strict) the looseness bounds. Similarly, the ``stale-max'' directive specifies a maximum number of seconds past the expiration time that the client is willing to accept. For example, ``stale-max 5'' means that the client is willing to accept a cached response that has already been stale for 5 seconds. This allows the client to decrease (make less strict) the looseness bounds. If both fresh-min and stale-max are given in a request, the stale-max is ignored. It may be appropriate, in some circumstances, for a user to specify that the response to a request should not be cached. (This might be because the user believes that an intervening cache may be insecure, but has no choice but to use the cache.) We include the ``dont-cache-response'' directive for this purpose; caches SHOULD obey it but the user MUST never rely on this. We do not specify what should happen if the cache already contains the response. XXX anything else? DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 19] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 3.11 Warnings Because one of our goals is that possibly non-transparent responses should not be represented as transparent ones, we introduce the notion of a ``warning''. A warning is a header that may be added to a response, either by the origin server or by a cache, indicating that the response might in some way be suspect. This is not done using the Status-line, because such responses are not really failures; they are simply somewhat suspicious. Warnings are advisory; they may be used by a user agent to control the presentation of a response, but do not otherwise affect the handling of responses. Warnings are always cachable, both because they are advisory and because they never weaken the transparency of a response. This means that warnings can be passed to HTTP/1.0 caches without danger; such caches will simply pass the warning along as a response header. Warnings are assigned numbers between 0 and 99. This specification defines the code numbers and meanings of each warning, but a server may send the warning message text in any appropriate natural language (perhaps based on the client's Accept: headers). Multiple warning messages may be attached to a response (either by the origin server or by a cache), including multiple warnings with the same code number. This allows a server to, for example, provide the same warning with texts in both English and Basque. The currently defined warnings are 90 Not firsthand MUST be sent by a cache with any response that is taken from a cache entry, except if the cache has validated the cached object with the origin server as a result of the current request. Note that if the cache attempts to validate and receives a ``90 Not firsthand'', then the validation did not come from the origin server. 91 Response is stale MUST be included whenever the returned response is stale. A cache may add this warning to any response, but may never remove it until the response is known to be fresh. 92 Revalidation failed MUST be included if a cache returns a stale response because an attempt to revalidate the response failed, due to an inability to reach the server. A cache may add this warning to any response, but may never remove it until the response is known to be fresh. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 20] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 98 Caching may violate law SHOULD be sent with any response if the server is aware of any law that specifically prohibits the caching (including storage) of the response. This may include copyright laws, confidential-records laws, etc. Such responses SHOULD also be marked with an expiration value in the past, to prevent an unwitting cache from returning the value in a cached response. 99 Miscellaneous warning The warning text may include arbitrary information to be presented to a human user, or logged. A system receiving this warning MUST NOT take any automated action. XXX anything else? 3.12 Compatibility with earlier versions of HTTP 3.12.1 Compatibility between HTTP/1.1 clients and older servers XXX when must an HTTP/1.1 client validate (using Get I-M-S) a response that comes from an HTTP/1.0 origin server? 3.12.2 Compatibility between HTTP/1.1 servers and older clients An HTTP/1.1 server SHOULD send a Last-Modified: date whenever possible. This allows older clients and proxies to use the If-Modified-Since: header, and may provide useful information even for HTTP/1.1 clients. An HTTP/1.1 server SHOULD send a cache-validator value (perhaps null) in every potentially cachable response. Older clients will simply ignore this, but the response may be cached by a proxy that speaks HTTP/1.1, and the cached value may later be used by an HTTP/1.1 client. 3.12.3 Compatibility between HTTP/1.1 and older proxies Since a proxy is always acting as either a client or a server, the rules for compatibility between clients and servers should suffice to provide compatibility between caches and servers (cache acting as client), and between caches and clients (cache acting as server). XXX is this true? 3.13 Update consistency Various operations, either by an HTTP client or local to the origin server, can change a resource so as to make cached copies invalid. This leads to inconsistencies, in that different clients may then see different versions of the resource. We would like to provide as much ``update consistency'' as possible. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 21] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 The draft HTTP/1.1 specification [1] includes several statements of the form: If the request passes through a cache and the Request-URI identifies a currently cached entity, that entity must be removed from the cache. There is no harm in taking such action, and it can eliminate certain inconsistencies that cannot be prevented by other means, but it is not sufficient to prevent inconsistency. For example, suppose that an entity is cached by two different caching proxies A and B, and client X of proxy A performs a DELETE method via that proxy. Proxy A will remove the entry from its cache, but proxy B will know nothing about the interaction, and so will continue to cache the deleted entity. This means that a different client Y, using proxy B, will have a view that is inconsistent with client X. Even worse, if the two proxies are pooling their entries, proxy A might end up re-acquiring the (bogus) cached entry from proxy B. For this reason, we should change the wording quoted above from ``must be removed from the cache'' to ``must be marked invalid in the cache'', at least for those caches that participate in pooling. Regardless of the wording here, any requirements that caches must delete or invalidate certain entries when doing certain operations are at best ``prudent,'' and at worst ``ineffective.'' There is no way for an origin server to force other caches to remove or invalidate entries; in fact, there is no way for an origin server even to discover the set of caches that might be holding an entry. We must recognize this, and look for other solutions to the update consistency problem. One simple solution would be for origin servers to mark potentially updatable resources as uncachable (using a past expiration value, or Cache-control: no-cache). If this is done conservatively, it solves the update consistency problem, but it also eliminates the performance benefits of caching for client requests between updates. Another solution would be for the server to refuse to perform update methods (such as PUT or DELETE) as long as any existing cache entries might be fresh. This requires the server to record the latest expiration value it has issued for a resource, and is similar to the Leases mechanism of Gray and Cheriton [2]. This would solve the update consistency problem without preventing caching, but it could lead to confused users, whose update operations would fail at unpredictable times. (This includes local users or administrators of the server, as well as HTTP clients.) One might take the approach that updates that conflict with existing expiration leases are accepted but queued for later application; this avoids unpredictable failures, but leads to mysterious effects. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 22] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Either of these solutions can be implemented unilaterally by the origin server, with the caching mechanisms described elsewhere in this document. Another approach, inspired by some work done at CMU XXX find appropriate reference XXX, uses the concept of a ``volume'' containing a number of individual resources, and allows the server to invalidate an entire volume at once. A cache finds out about a volume invalidation whenever it contacts the origin server regarding any member of the volume; this improves the chances that a cache will notice that a resource has been updated before it provides an inconsistent cache response to a client. However, it cannot eliminate inconsistency. We give a specific proposal for volume validation in section 3.14. Some combination of these approaches (uncachable responses, leases, and volume validation) may allow servers to reduce the update consistency problem to manageable levels, while still allowing reasonably effective caching and predictable responses to users doing updates. However, it is easy to construct scenarios in which it is simply impossible to optimize on all three metrics at once. In order to solve this problem, we would have to introduce some sort of ``callback'' mechanism, which is not feasible for HTTP/1.1. 3.14 Volume validation We propose a simple and optional ``volume validation'' mechanism for the HTTP protocol, to reduce the chances that caches are inconsistent with origin servers or with each other. If a server wants to use volume validation, it assigns each resource to one of a number of sets, called ``volumes.'' Volumes do not necessarily map onto storage-hierarchy concepts such as disks; they might be based on file type, for example. Members of a volume ought to have similar lifetimes. The server might assign all of its resources to the same volume, or it might use a number of volumes to distinguish between (for example), probably immutable resources, things that change slowly (say, once a week), things that change often (say, once an hour), and things that are very dynamic (changing at intervals of seconds or minutes). If a group of resources is typically changed together, then that group also forms a natural volume. When the server returns a response to a cache, it includes the usual cache control info (as specified elsewhere in this draft), and it also returns these three new header values: Volume-ID: Volume-version: Volume-expires: HTTP-date (in RFC-822 form) DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 23] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 XXX we probably have to correct the volume-expiration value using the same Age: headers that we deduct from expiration values. The cache, in addition to keeping the individual resources, also keeps a cache of this per-volume information. Each of the individual resource cache entries includes a pointer to the associated volume info (which is managed as a cache, and therefore might not always be present). Also, the Volume-version: value is stored with each individual resource entry. Note that the per-volume information must also include some sort of unique ID for the server, such as its IP address or host name. Each time the server receives a response from a server, it can update the per-volume information from that response. This allows the server to keep increasing the expiration value for a volume. However, if any resource in the volume is modified, then the server must change the Volume-version: value to one it has never used before. (This could be done using a timestamp or sequence number; note that the clients and caches MUST NOT attempt to impose an ordering on these values.) When the cache receives a client request, it would normally check the expiration information stored with the relevant individual-resource entry to decide if it has to reload the response from the server. However, if the resource is associated with a volume that the cache knows about, then it can do this: if (volume-version stored with resource matches current volume version stored in per-volume entry) then if (volume-expiration time has not yet been reached) or if (resource expiration time not yet reached) then it's OK to return the cached response else must do conditional GET from server else if (resource expiration time not yet reached) then it's OK to return the cached response else must do conditional GET from server In other words, the per-volume information is used to extend the expiration time for what might be a large set of cached resources (if the version value remains unchanged) or to invalidate a large set of cached resources (if the version value changes). Of course, this would only pay off when a proxy caches a sufficiently large number of resources from the same server (and from the same volume). But it lets the server assign relatively short expiration times to the individual resources (which makes update inconsistency DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 24] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 less likely), and still prevent a busy proxy from bombarding it with conditional requests for resources that haven't been modified. It is entirely optional for the server or the cache to implement this, and the implementation at the cache side of things seems to pretty simple. On the server side, implementation complexity will depend on how the server detects whether a member of a volume is modified. A small amount of support from the underlying file system or database might be quite useful, but in many cases a fairly simple scheme would work. For example, if all of the items in a product catalog were assigned to the same volume, the server administrator could simply change the volume-version value whenever the catalog is updated. If a cache receives a response that changes the Volume-ID of a resource currently in the cache (whether fresh or not), the cache MUST set the volume-expiration value for the old volume to be zero, forcing the cache to check with the origin server on any subsequent request for a member of the old volume. A change in the Volume-ID for a cached resource implies that the volume itself has changed in a significant way. XXX should we insist that all of the cached responses from the old volume be revalidated? In case they are no longer in this volume? In general, does this design adequately deal with the problem of moving resources between volumes? XXX should we provide some mechanism for a cache to advise an origin server that it is willing to obey the volume validation protocol (using Connection: to avoid confusion from intervening caches)? This would allow the origin server to allow looser caching without as much risk of inconsistency. 3.15 Side effects of GET and HEAD Section 14.2 (``Safe Methods'') of the draft HTTP/1.1 specification [1] implies that GET and HEAD methods should not have any side effects that would prevent caching the results of these methods, unless the origin server explicitly prohibits caching. We make this explicit: unless the server explicitly disables caching, GET and HEAD methods SHOULD NOT have side effects. XXX alternate wording: unless the server explicitly disables caching, GET and HEAD methods SHOULD NOT have side effects that would lead to erroneous behavior if their responses are taken from a cache. They may still have side effects, but a cache is not required to consider such side effects in its caching decisions. Caches are always expected to observe an origin server's explicit restrictions on caching. Apparently, some applications have used GETs and HEADs with query URLs (those containing a ``?'' in the rel_path part) to perform DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 25] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 operations with significant side effects. Therefore, caches MUST NOT treat responses to such URLs as fresh unless the server provides an explicit expiration time. This specifically means that responses from HTTP/1.0 servers for such URIs should not be taken from a cache, since an HTTP/1.0 server provides no way to indicate that two query responses issued during the same second are in fact different. 3.16 Write-through mandatory All methods that may be expected to cause modifications to the origin server's resources MUST be written through to the origin server. This currently includes all methods except for GET and HEAD. A cache MUST NOT reply to such a request from a client before having transmitted the request to the inbound server, and having received a corresponding response from the inbound server. The alternative (known as ``write-back'' or ``copy-back'' caching) is not allowed in HTTP/1.1, due to the difficulty of providing consistent updates and the problems arising from server, cache, or network failure prior to write-back. 4 HTTP headers related to caching XXX this section is still quite sketchy 4.1 Cache-validation Validators are quoted strings of printing ASCII characters: opaque-value = quoted-string The empty string (``""'') is explicitly allowed, and is defined as the null value, for which all equality comparisons MUST fail. In particular, two null strings do not compare as equal. XXX do we need to specify a maximum length for these strings? Servers return cache-validator values using Cache-validator: opaque-value Clients use If-invalid: opaque-value for conditional retrievals. Clients use DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 26] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Modify-if-valid: opaque-value for conditional updates. 4.2 Expiration values XXX this section includes certain provisional parameters in [brackets]. Expiration values are absolute dates in HTTP-date format. Although previous HTTP specifications have allowed several formats, including one with only two digits to represent the date, we need to be able to unambiguously represent expiration times in the future. Therefore, origin servers MUST NOT use the RFC-850 form, and SHOULD use the RFC-822 form (as already stated in the HTTP/1.1 specification). An HTTP/1.1 server MUST always send a Date: header if it sends an Expires: header. The two headers should use the same date form. Origin servers MUST NOT use the RFC-850 form, and SHOULD use the RFC-822 form. Although servers MUST NOT send dates in RFC-850 form, clients and caches MUST be accept these if they are at all sensible. HTTP/1.1 clients and caches SHOULD assume that an RFC-850 date which appears to be more than [50]XXX years in the future is in fact in the past (this helps solve the "year 2000" problem). To mark a response as ``already expired,'' an origin server should use an Expires: date that is equal to the Date: header value. Clients and caches MUST assume that a response whose Expires: and Date: fields are representations of the same value is immediately stale. Also, any response whose Expires: value is older than its Date: value is immediately stale. To mark a response as ``never expires,'' an origin server should use Expires: date approximately [one]XXX year from the time the response is generated. HTTP/1.1 servers should not send Expires: dates more than [one]XXX year in the future. XXX Proposed rules for requirements on parsing and representation of dates: HTTP/1.1 clients (and caches) SHOULD be able to parse HTTP-date values at least one year in the future (from the time that a response is received). HTTP/1.1 clients and caches should assume that an RFC-850 date which appears to be more than [50]XXX years in the future is in fact in the past (this helps solve the "year 2000" problem). All HTTP/1.1 implementations MUST be capable of correctly parsing any HTTP-date value that is no less than [25]XXX years in the future from the date that the implementation is sold or otherwise distributed. An HTTP/1.1 implementation may internally represent a parsed DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 27] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Expires: date as earlier than the proper value, but MUST NOT internally represent a parsed Expires: date as later than the proper value. If the origin server includes an Expires: header but not a Date: header, a cache or client should compare the expiration value to its current local clock to decide if a cached copy is fresh or stale If the origin server does not include an Expires: value, and does not otherwise prohibit caching, then a cache may use a heuristic to determine how long the response may be treated as fresh. Caches should not treat such ``undefined'' expiration values as ``forever.'' A service administrator who fails to assign an Expires: value should be aware that some caches may treat this as practically ``forever''. This MUST never be longer than [7]XXX days. XXX Action item: We need a heuristic and a maximum value. The heuristic should deduct any Age: value already associated with the response. Also, we probably need to say that the heuristic must return zero for methods other than GET or HEAD. 4.3 Age values Caches transmit age values using: Age = "Age" ":" delta-seconds Age values are non-negative decimal integers, representing time in seconds. If a cache receives a value larger than the largest positive integer it can represent, it should replace with the largest representable positive integer. Caches SHOULD use a representation with at least 31 bits of range. Caches MUST NOT delete Age: headers, although they may coalesce them into a single value. They may also combine multiple Age: headers into one, with the values separated by commas. 4.4 Cache-control The Cache-control: header can include one or more of the following directives on requests: DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 28] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 reload revalidate "fresh-min" "=" delta-seconds "stale-max" "=" delta-seconds dont-cache-response The Cache-control: header can include one or more of the following directives on responses: no-cache private public 4.5 Warnings Warning headers are sent with responses using: "Warning" ":" 2DIGIT SP *TEXT The currently defined Status-code/Reason-Phrase pairs allowed in warnings include: 90 Not firsthand 91 Response is stale 92 Revalidation failed 98 Caching may violate law 99 Miscellaneous warning 4.6 Volume validation Servers send volume validation information in responses using these headers: Volume-ID: Volume-version: Volume-expires: HTTP-date (in RFC-822 format) Volume-ID strings and Volume-version strings are opaque to clients and caches. The Volume-expiration values follow the same rules as do expiration values. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 29] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Servers are not required to send these headers; clients and caches are not required to implement them. However, if a cache does implement any of them, it MUST follow the specification for all of them. 5 Rules summarized by method This section includes caching-related rules for each HTTP/1.1 method. TBS 6 Cache replacement algorithms TBS 7 Specific issues XXX this section is likely to be reorganized 7.1 Content negotiation TBS 7.2 Authentication TBS 7.3 State XXX From Shel Kaphan, may need editing and references to other documents: Briefly, cookies can't be cached by public (shared) caches, but since public documents may make up part of a "stateful dialog", and in particular the first document in a stateful dialog may be (for example) a public and cachable home page, servers that wish to receive the client's cookie on each request, or to issue a new cookie on requests for a document must set the document up to require validation on each request. (e.g., by having it be "pre-expired"). Otherwise, the cache control headers for responses control what a proxy has to do. If a document is fresh in a cache, a request containing a cookie does not have to be forwarded to the origin server, since (by definition) if the document is servable from a cache, there aren't important side effects at the origin relating to requests for that document, and so, no changes to the cookie. One important issue bearing on caching is that for conditional requests that go through to the origin server, for which the origin server responds with 304 and also with a set-cookie header, caches must splice the set-cookie sent by the origin server into their own DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 30] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 response. This is, for example, how it can work to have a home page that is in a cache, but stale, so that the only traffic to the origin server is to validate the home page, receiving a 304 and potentially a new cookie. 7.4 Caching and methods with side effects TBS 7.5 Network partitions TBS 7.6 Caching of negative responses TBS 7.7 History Lists TBS XXX should be treated as separate from caches XXX browser could alert the user to stale page, without automatically refreshing. 7.8 Hit Metering Some customers insist that the hit counts reported at servers include the hits that happen only at caches. In the absence of a mechanism to do this, some servers disable caching simply to obtain hit counts, which is not good for overall performance. It would not be possible to make hit-metering mandatory for caches, simply because there is no obvious way to enforce an anti-cheating policy. Also, hit metering may run afoul of privacy concerns, which in some jurisdictions have the force of law. But it should be possible to provide a mechanism for cooperative caches and servers to use. Customers have asked for several kinds of hit-metering, with increasing levels of complexity and cost: 1. simple hit counts: the cache tells the server how many hits have been made on a particular cached value. 2. access-log transfers: the cache sends to the server complete access logs (in some standardized format) so that the server can tell which clients have made what requests and when. 3. client navigation traces: the cache sends to the server enough information to determine how the user got from one page to another. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 31] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 It is not entirely clear whether (3) is really different from (2), or whether the access logs contain enough information to satisfy the people who want (3). XXX Netscape has internal specifications for (1) and (2?), and will write them up and circulate them to the subgroup. XXX Jim Gettys adds: Rohit Khare did point out that people want to know very much not only that a document has been hit, but that it was cancelled before delivery was complete. This is how webmasters track down GIFS or boring pages. Adding this much to the simple hit count reporting seems like the point in the design space we should shoot for. 7.9 Bypassing in caching hierarchies Many HTTP caching configurations involve hierarchies of caches, often designed to reduce bandwidth requirements rather than improving latency. However, if a cache at a low level in the hierarchy is sure that the cache(s) above it do not contain a cache entry to match a given request, that low-level cache can transmit the request directly to the origin server. This improves retrieval latency without increasing total bandwidth requirements (it even eliminates some packet transmissions) and is entirely appropriate for resources whose values are explicitly not cached. We call this technique ``request bypassing.'' Note that although the bypassing decision might be done by the ultimate client, in many cases the use of firewalls or unsophisticated clients means that the decision must be made by an intermediate-level cache. In order to do request bypassing in the most efficient possible way, the caches must be able to determine from the request whether the response is likely to be cachable. (It is important to err on the side of assuming cachability, since the assuming converse could seriously reduce the effectiveness of the higher-level caches.) The current HTTP/1.1 draft specification does not include a foolproof mechanism to mark requests in this way. While we generally do not allow caching of responses to GET requests for URLs with a ``?'' in the rel_path part (see section 3.15), we also allow the origin server to mark responses to such queries as cachable. Therefore, any bypassing done using this heuristic runs the risk of giving up perfectly good opportunities to cache some resources. XXX we have discussed various approaches for marking requests, all of which apparently require some kind of change to HTML to allow the origin server to pass the marks to the ultimate client. Some people suggest using special methods that are explicitly always cachable (``POST_WITH_NO_SIDE_EFFECTS'', or more concisely ``POSTC'') or never cachable (``GET_QUERY'', or more concisely ``GETQ''). Others have suggested adding tags to DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 32] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 HTML that would cause subsequent requests to carry some special sort of header. Neither solution has resulted in a consensus. An origin server would be able to use POSTC only with HTTP/1.1 clients and proxies, and so would have to return different HTML forms depending on the protocol version in the request header. This would also imply using the proposed Vary: header with some token that indicates ``varies based on request HTTP version,'' since we don't want a cache returning one of these HTML responses to an HTTP/1.0 client. 7.10 Explicit warnings regarding user-specified overrides Many browsers make it possible for users to override the basic caching mechanisms. For example, the browser may allow the user to specify that cached entities (even explictly stale ones) are never validated. Or the browser might habitually add ``Cache-control: stale-max=3600'' or ``Cache-control: reload'' to every request. We recognize that there may be situations which require such overrides, although browsers SHOULD NOT default to any behavior contrary to the HTTP/1.1 specification. That is, the user should have to explicitly request either non-transparent behavior, or behavior that results in abnormally ineffective caching. If the user has overridden the basic caching mechanisms, the browser should explicitly warn the user whenever this results in the display of information that might not meet the server's transparency requirements (in particular, if the displayed resource is known to be stale). Since the protocol normally allows the browser to determine if responses are stale or not, this indication need only be displayed when this actually happens. The indication need not be a dialog box; it could be an icon (for example, a picture of a rotting fish) or some other visual indicator. If the user has overridden the caching mechanisms in a way that would abnormally reduce the effectiveness of caches, the browser should continually display a warning indication (for example, a picture of currency in flames) so that the user does not inadvertently consume excess resources or suffer from excessive latency. 8 Security Considerations 8.1 Caching of responses to authorized requests The draft HTTP/1.1 specification says (section 10.6): Responses to requests containing an Authorization field are not cachable. We suggest restating this as DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 33] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Responses to requests containing an Authorization field MUST not be returned from a shared cache to any user other than the user that made the initial request. If the cache cannot securely determine the identity of the user, then it must act as if the identity does not match. A ``shared cache'' is one that accepts requests from more than one user. This definition normally applies to all caches except those integrated with single-user browsers (and not accessible except from their containing browsers). Such non-shared caches may return cached responses to authorized requests. Protection against replay attacks seems to be a function of how the cache ``securely'' determines user identies. That is, if the mechanism does not protect against replay attacks, then it is by definition not secure. 8.2 Location: headers and spoofing TBS XXX Contribution from Shel Kaphan, may need editing and shortening: The location header in 2xx responses identifies a URI at which the client could request another copy of the resource being included as the body of the response. We discussed two issues in connection with this: "invalidation" (i.e. "making stale") of other objects in a cache due to the receipt of a response containing a Location header, and spoofing. I'm going to use the word "invalidation" to mean "making stale", since it sounds better. If a cache were to use the Location URI to cause the response entity to be cached in such a way that a subsequent GET could retrieve the entity from the cache using that URI, then a response containing a Location URI not under the same authority as the request-URI could accidentally or intentionally confuse a cache, and cause subsequent requests to receive erroneous data in response. If such replacement is allowed, there seems to be no obvious way to keep this spoofing from happening. (Any ideas?) Paul Leach suggested that it might be "necessary but not sufficient" to at least require the Location URI to share the same hostname as the request URI. It is not sufficient because there may be multiple authorities operating on the same host. It is also not necessary because a given authority may span multiple hosts, and a given host may have multiple names. A legitimate Location URI might not match the request URI's hostname, whereas an illegimate one might sometimes match the request hostname. Paul seemed to have some other objections related to worrying about this spoofing problem that I didn't understand. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 34] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 Jeff Mogul suggested that invalidation of objects in the cache whose request-URI matches the location URI is not sufficient to solve the cache coherency problem. This is of course true -- if there are multiple caches containing copies of some object, invalidating objects in one cache has no effect on other caches. Therefore, Jeff suggested that if it is critical for an up-to-date object to be served for every request, that its expiration should be set so that it is validated on each request. I agree with that. However, I believe there is a class of resources for which it is desirable but not critical for an up-to-date version to be seen, and for these objects it seems reasonable that a single cache should try to do its best (forgive the anthropomorphizing) to provide the up-to-date version. Perhaps this can be viewed as an optional optimization not required by the protocol, since it is always possible that a series of requests may be served by different caches anyway, and so the protocol doesn't guarantee that a cachable ("fresh") object would ever be invalidated. A context in which this invalidation could be very useful is content negotiation. If a Location header in a negotiated response ever matches an explicit request URI for some object already in a cache, but the negotiated response contains a different version that has come into existence before the expiration date of the previously cached version, a subsequent request on that URI in this cache will retrieve the previous version. This seems rather annoying, and can't be fixed by replacement using the Location URI because of the spoofing problem mentioned above. The best that can be done is to invalidate other objects in the cache that were keyed by a request-URI matching this location-URI, so that future requests will require validation. Since this is useful for the content negotiation case, the same mechanism may also be used to help with other uses of the Location header -- for instance a response to a POST may be a new version of a GETtable resource that may already have been cached. Again, as Jeff points out, if this is critical, that GETtable resource should be set to require validation on each request, but it might be just desirable and not critical. Another minor point: in the content negotiation example above, if the explicit GET occurred after the negotiated GET, then invalidation would still be desired, but this time the request-URI of the GET (assuming no Location header was returned) would be used to invalidate not only other occurrences of the request-URI in the cache, but also entries for which that URI was returned as a Location URI. So, for this purpose the Location URI would serve as a secondary key to the cache. DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 35] Internet-Draft HTTP 1.1 caching (DRAFT) 15 February 1996 18:02 9 Acknowledgements A lot of the content and presentation of this document is due to suggestions and comments from individuals including: Shel Kaphan, Paul Leach, Koen Holtman, David Morris, Larry Masinter, and Roy Fielding. 10 References 1. Roy T. Fielding, Henrik Frystyk Nielsen, and Tim Berners-Lee. Hypertext Transfer Protocol -- HTTP/1.1. Internet-Draft draft-ietf-http-v11-spec-00.txt, HTTP Working Group, November, 1995. 2. Cary G. Gray and David R. Cheriton. Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency. Proc. 12th Symp. on Operating Systems Principles, Litchfield Park, AZ, December, 1989, pp. 202-210. 11 Author's address Jeffrey C. Mogul Western Research Laboratory Digital Equipment Corporation 250 University Avenue Palo Alto, California, 94305, U.S.A. Email: mogul@wrl.dec.com Phone: 1 415 617 3304 (email preferred) DO NOT IMPLEMENT TO THIS DOCUMENT! [Page 36]