Network Working Group J. Mogul, DECWRL Internet-Draft 12 September 1997 Expires: 26 March 1998 Generation of the Age header field in HTTP/1.1 draft-mogul-http-age-00.txt STATUS OF THIS MEMO This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Distribution of this document is unlimited. Please send comments to the HTTP working group at . Discussions of the working group are archived at . General discussions about HTTP and the applications which use HTTP should take place on the mailing list. ABSTRACT The 'Age' response-header field in HTTP/1.1 [RFC 2068] is intended to provide a lower bound of an estimate of a response message's age (time since generation), by explicitly indicating the amount of time that is known to have passed since the response message was retrieved or revalidated. There has been considerable controversy over when the Age header field should be added to a response. This document explains the issues, rebuts a previous proposal, and provides a set of proposed changes for the revision of RFC 2068. Mogul [Page 1] Internet-Draft HTTP Age generation 12 September 1997 13:48 TABLE OF CONTENTS 1 Introduction 2 2 Proposed change to RFC2068 3 3 Discussion 3 3.1 Quantifying the error of the Age estimate: Interpretation 4 C 3.2 Quantifying the error of the Age estimate: Interpretation 5 B 3.3 Prevalence of clock skew in the Web 5 3.4 Implications of overestimating the Age value: 6 Interpretation C 3.5 Implications of underestimating the Age value: 7 Interpretation B 3.6 So what's all the fuss about, anyway? 8 4 Acknowledgements 9 5 References 9 6 Author's address 9 1 Introduction RFC2068 [3] defines, in section 14.6, the Age header field for HTTP/1.1: The Age response-header field conveys the sender's estimate of the amount of time since the response (or its revalidation) was generated at the origin server. A cached response is "fresh" if its age does not exceed its freshness lifetime. Age values are calculated as specified in section 13.2.3. Age = "Age" ":" age-value age-value = delta-seconds Age values are non-negative decimal integers, representing time in seconds. If a cache receives a value larger than the largest positive integer it can represent, or if any of its age calculations overflows, it MUST transmit an Age header with a value of 2147483648 (2^31). HTTP/1.1 caches MUST send an Age header in every response. Caches SHOULD use an arithmetic type of at least 31 bits of range. Roy Fielding has taken issue with the statement ``HTTP/1.1 caches MUST send an Age header in every response.'' [2]. Fielding correctly points out that the use of the word ``caches'' in this sentence is incorrect. He lists two possible rewordings for this sentence: Mogul [Page 2] Internet-Draft HTTP Age generation 12 September 1997 13:48 a) An HTTP/1.1 server that includes a cache MUST send an Age header field in every response. b) An HTTP/1.1 server that includes a cache MUST include an Age header field in every response generated from its own cache. There is, however, a third possible rewording, if one realizes that the original drafting error was to sloppily substitute the word ``cache'' where ``proxy'' was meant (mea culpa): c) An HTTP/1.1 proxy MUST send an Age header field in every response. Note that an HTTP proxy does not necessarily include a cache. The other sentences referring to ``caches'' in that paragraph reflect the same drafting error. 2 Proposed change to RFC2068 For this paragraph in section 14.6 (Age), If a cache receives a value larger than the largest positive integer it can represent, or if any of its age calculations overflows, it MUST transmit an Age header with a value of 2147483648 (2^31). HTTP/1.1 caches MUST send an Age header in every response. Caches SHOULD use an arithmetic type of at least 31 bits of range. substitute If a proxy server receives a value larger than the largest positive integer it can represent, or if any of its age calculations overflows, it MUST transmit an Age header with a value of 2147483648 (2^31). An HTTP/1.1 proxy MUST send an Age header in every response. Proxies SHOULD use an arithmetic type of at least 31 bits of range to represent Age values. 3 Discussion Because of facts of life such as resolution granularity, clock skews, and the theory of relativity, the Age value can never be exactly accurate; it is always an estimate. We would like to bound the inaccuracy of this estimate. Most important, we would like to ensure that any inaccuracy remaining in the Age estimate not cause avoidable and significant harm. Mogul [Page 3] Internet-Draft HTTP Age generation 12 September 1997 13:48 3.1 Quantifying the error of the Age estimate: Interpretation C If interpretation (c) is used (i.e., the change proposed above in section 2), then the value of the Age header field might overestimate the actual amount of time since the response was generated at the origin server. One can calculate the size of the estimation error for a path containing N HTTP/1.1 proxies: Error_C = Mean_RTT * N * (N + 1)/2 where Mean_RTT is the mean round-trip time (RTT) between neighboring pairs of proxies. Note that this measures the error in the Age value at the final recipient client, not the error at the last HTTP/1.1 proxy on the path. Actually, this analysis assumes that the RTT on each hop is the Mean_RTT; in fact, the contribution from the hops closest to the origin server contribute more to the true error value than this simplified formula, and so it is not necessarily an upper bound on the error. If one uses Error_C_bound = Max_RTT * N * (N + 1)/2 where Max_RTT is the largest of the RTTs over all of the hops, then the Error_C_bound estimate is an upper bound. In order to quantify this bound, we need parameter values for N and Max_RTT. It's not clear exactly how many proxy caches are used on an arbitrary path through the Internet, especially since many of these paths start within large intranets, but I think it would be realistic to assume that most such paths have somewhere between N=1 and N=6. Clearly, the Max_RTT value is harder to quantify, since over extremely lossy paths, the RTT can be nearly infinite. As a surrogate for this value, I looked at the total retrieval time for each of 504736 retrievals in a two-day proxy trace made in December, 1996. (See [5] for more discussion of this trace.) Among these retrievals, the mean total retrieval time was 1.25 seconds, with a median of 0.25 seconds and a standard deviation of 13.7 seconds. As far as I can tell, a large part of the variation is due to servers that drop SYN packets due to buggy TCP stacks. Given this, and given that the median is closer to 1/4 second, and given that a full HTTP/1.0 retrieval takes at least two round-trips (and probably more), I think it's fair to assume a value of 2 seconds as an approximate upper bound for the Max_RTT for most paths. Therefore, doing the math Error_C_bound[N=1] = 2 seconds Error_C_bound[N=6] = 42 seconds and, just for comparison, if one uses the observed median retrieval Mogul [Page 4] Internet-Draft HTTP Age generation 12 September 1997 13:48 time (0.25 seconds) and assumes the use of no more than 3 HTTP/1.1 proxies, then Error_C_bound = 1.5 seconds. 3.2 Quantifying the error of the Age estimate: Interpretation B If interpretation (b) is used, then the value of the Age header field might underestimate the actual amount of time since the response was generated at the origin server. The scenario that generates the greatest underestimate is when the path between the origin server and the HTTP/1.1 client's cache includes an HTTP/1.0 proxy cache, and the client has a skewed clock. In particular, the client's clock is set into the past by some amount. In this scenario, since an HTTP/1.0 proxy cache will not add the Age header, there is no indication that the response was delayed by a cache between the origin server and the first HTTP/1.1 cache. The only way for that HTTP/1.1 cache to construct its initial Age estimate is to compare its own clock against the Date value. If the client's clock is wrong, this error is, in general, impossible to correct. For example, suppose the client's clock is one hour slow. If the origin server generates the response at 10:00AM, and then the response sits in the HTTP/1.0 cache for 65 minutes, when the response arrives at the HTTP/1.1 client, it will appear to be only 5 minutes old. The benefit of using interpretation (c) is that if there is a non-caching HTTP/1.1 proxy between the HTTP/1.0 cache and the client, unless its clock is also skewed, it will add a nearly-correct Age header field to the response. I.e., the client will realize that the response is at least 65 minutes old, regardless of its clock skew. So, while the error is impossible to correct "in general", it is possible to correct in certain specific cases, and this is the value of interpretation (c). 3.3 Prevalence of clock skew in the Web Is clock skew a real problem? Unfortunately, I know of no systematic study of HTTP client clock skews. This is difficult, in part, because HTTP requests generally do not include a Date header. However, since I do have access to a trace of the headers flowing through a proxy whose clock, at the time of the trace, was carefully synchronized using NTP, I was able to look at the clock-skew distribution of a large set of HTTP servers. (The trace covers 22034 distinct server IP addresses.) While this is not the same as a population of HTTP clients, one might actually expect a set of HTTP servers to have better clock synchronization characteristics than a set of HTTP clients. After all, many HTTP clients run on personal computers or workstations, and are managed by non-experts; most Web servers on the Internet have at least some semblance of administration (e.g., someone at least had to obtain a DNS name). In other words, whatever the situation with Web server clocks, one would expect the situation among clients to be worse. Mogul [Page 5] Internet-Draft HTTP Age generation 12 September 1997 13:48 For each response in the trace, I compared the Date header field value (if any) to the proxy's NTP-synchronized timestamps for the start of the connection and the end of the connection. If the server's clock is accurate, the Date value ought to be between those two timestamps. If the server's clock is slow, the Date value would be lower than the start-timestamp; if the server's clock is fast, the Date value would be higher than the end-timestamp. Because of the 1-second granularity of Date, I treated as "valid" any values less than 1 second in error. I also treated as "obviously bogus" any Date where the server's clock appeared to be more than 1 day wrong, since one could assume that such a badly skewed server clock would be abnormal. The trace contained 503969 responses with parsable response headers. Of these, only 286779 actually had Date headers (most of the rest appear to be PointCast responses). 1087 of these had Date values that were clearly bogus (by the "1-day-wrong" test). Of the others, 116966 (41%) showed a server with a "slow" clock (by at least one second), and 83782 (29%) showed a "fast" clock. Only 84944 (30%) had apparently-synchronized clocks. What if we set the threshold for an OK clock at +/- 60 seconds (which, by the earlier analysis, is somewhat larger than the Error_C_bound for N = 6 and Max_RTT = 2)? In this case, we still find 79443 (27%) responses indicating "slow" clocks, and 56429 responses (20%) indicating "fast" clocks. In other words, a lot of the clocks are off by a lot of time. Using the 1-second threshold, the mean error in the slow clocks is 1287 seconds, with a median error of 113 seconds. For the fast clocks, the mean error is 1383 seconds, with a median of 97 seconds. Using the 60-second threshold, the mean error in the slow clocks is 1884 seconds, with a median error of 198 seconds. For the fast clocks, the mean error is 2039 seconds, with a median of 152 seconds. (We're removing the small-error samples from these sets, so we're left with sets biased towards high-error samples.) In summary, clock skew seems to be prevalent among HTTP servers, and the skews seem to be fairly large. One might be justified in guessing that the situation is worse among HTTP clients. NOTE: I should reanalyze this data, breaking it down by server address, rather than by response, but that will have to wait for another draft of this document. 3.4 Implications of overestimating the Age value: Interpretation C What happens if the Age value is overestimated? If this happens, some "fresh" responses appear to be "stale", and so unnecessary cache misses may be generated. Except in the case where the network is Mogul [Page 6] Internet-Draft HTTP Age generation 12 September 1997 13:48 partitioned, this is a performance problem, but does not lead to the delivery of responses with the wrong body or headers. (When the network is partitioned, caches are allowed to return stale values with an appropriate Warning, so this is irrelevant to the current discussion). Further, only a very small subset of the fresh, cachable responses will be affected. In particular, this can only affect a response whose actual remaining freshness lifetime is smaller than the error in the Age estimate. Presumably, this is only a significant issue for responses whose initial max-age value is small (since otherwise it would be a low-probability coincidence that a request arrives at the cache just before the freshness lifetime expires). (Remember that the Error_C_bound = 42 for N = 6 and Max_RTT = 2, and is a lot lower for shorter proxy paths.) We have no good information on what kinds of lifetimes HTTP/1.1 origin servers will be assigning to their responses (very few current servers send Expires), but we can make a guess that of the responses that are likely to be cachable, most will be assigned fairly long lifetimes. This is because existing caches already assign, by heuristic, fairly long lifetimes to responses without Expires or max-age values, and (for the most part) this kind of caching seems to be tolerated. Certainly, one would hope that forthcoming HTTP/1.1 server implementations will make it easier for service authors to specify small freshness lifetimes. However, it is unlikely that this could lead to a great improvement in the number of cache hits. Several trace-based studies [1, 4] have shown that there is an upper limit to the performance of straightforward HTTP caches, because for many resources, no two requests result in the same response body. Therefore, one should not expect the currently-observed cache hit rates to get a lot better. 3.5 Implications of underestimating the Age value: Interpretation B What happens if the Age value is underestimated? If this happens, some "stale" responses appear to be "fresh", and are returned to the client without any Warning. The client, in this case, naively obtains the wrong response value. This is a far more serious error than causing an extra cache miss. Delivering the wrong response is not always a significant error, but in some cases it can lead to serious external consequences. Moreover, the set of cachable responses vulnerable to this error is much larger than for an overestimation error. The reason is, as we saw above, clock-skew errors appear to be quite common, especially slow-clock errors, and the mean error is measured in tens of minutes. This is likely to be much closer to the magnitude of freshness lifetimes than the approximately 1-minute overestimation error. Mogul [Page 7] Internet-Draft HTTP Age generation 12 September 1997 13:48 The adoption of interpretation (b), in an attempt to avoid unnecessary cache misses, could have a perverse effect: by significantly increasing the likelihood of undetected delivery of stale responses when the freshness lifetime is short, this might discourage service authors from allowing such responses to be cached at all. In other words, service authors generating responses with short cachable lifetimes might be driven to declare these responses as totally uncachable, to avoid the possibility of unwarranted caching. One should also note that a small (but certainly non-empty) fraction of the bogus Date values in our trace are wrong by days, not hours. If such a problem afflicts many clients, this could affect the willingness of service authors to allow caching even of responses with relatively long lifetimes. 3.6 So what's all the fuss about, anyway? It might seem that arguing over the issue of caching responses with relatively short freshness lifetimes is a tempest in a teapot. At some level, it is: straightforward HTTP caching will probably never yield the kind of hit rates seen for CPU caches, and this is especially true for short-lifetime values. However, there are a number of ways in which the effectiveness of an HTTP cache could be improved; for example, prefetching seems to be useful for decreasing latency (although it doesn't help if the path is already bandwidth-limited). Prefetching might take advantage of the cachability of responses with relatively short lifetimes, but it will not be feasible if service authors cannot trust the prefetching caches to be punctilious about freshness lifetimes. In other fields of computer science (for example, compilers or multiprocessor caches), aggressive optimization always depends on a reliable understanding of the situation at hand. That is, if one cannot be sure that the transformation done by the optimization preserves the semantics of the system, one cannot safely do that optimization (and, consequently, one may be prevented from doing a lot of other optimizations that are enabled by the first one). Even when optimizations are done "speculatively", this always involves being able to check the results for semantic correctness before committing them. Attempts to make things faster at the cost of semantic transparency might have some short-term attractions, especially in a setting where we have historically been sloppy about semantic correctness. But this is ultimately a dead-end path; after once giving up on semantic correctness, it's almost impossible to go back. Mogul [Page 8] Internet-Draft HTTP Age generation 12 September 1997 13:48 4 Acknowledgements I would like to thank Jim Gettys for comments on this draft. 5 References 1. Fred Douglis, Anja Feldmann, Balachander Krishnamurthy, and Jeffrey Mogul. Rate of Change and Other Metrics: a Live Study of the World Wide Web. Proc. Symposium on Internet Technologies and Systems, USENIX, Monterey, CA, December, 1997. To appear. 2. Roy T. Fielding. Age Header Field in HTTP/1.1. Internet-Draft draft-fielding-http-age-00.txt, Network Working Group, March, 1997. 3. Roy T. Fielding, Jim Gettys, Jeffrey C. Mogul, Henrik Frystyk Nielsen, and Tim Berners-Lee. Hypertext Transfer Protocol -- HTTP/1.1. RFC 2068, HTTP Working Group, January, 1997. 4. Thomas M. Kroeger, Darrell D. E. Long, and Jeffrey C. Mogul. Exploring the Bounds of Web Latency Reduction from Caching and Prefetching. Proc. Symposium on Internet Technologies and Systems, USENIX, Monterey, CA, December, 1997. To appear. 5. Jeffrey C. Mogul, Fred Douglis, Anja Feldmann, and Balachander Krishnamurthy. Potential benefits of delta encoding and data compression for HTTP. Research Report 97/4, DECWRL, July, 1997. URL http://www.research.digital.com/wrl/techreports/abstracts/97.4.html. 6 Author's address Jeffrey C. Mogul Western Research Laboratory Digital Equipment Corporation 250 University Avenue Palo Alto, California, 94305, USA Email: mogul@wrl.dec.com Mogul [Page 9]