Network Working Group F. Yergeau Internet Draft G. Nicol G. Adams Expires 2 December 1996 M. Duerst 27 May 1996 Internationalization of the Hypertext Markup Language Status of this Memo This document is an Internet-Draft. Internet-Drafts are working doc- uments of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute work- ing documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months. Internet-Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet- Drafts as reference material or to cite them other than as a "working draft" or "work in progress". To learn the current status of any Internet-Draft, please check the 1id-abstracts.txt listing contained in the Internet-Drafts Shadow Directories on ds.internic.net (US East Coast), nic.nordu.net (Europe), ftp.isi.edu (US West Coast), or munnari.oz.au (Pacific Rim). Distribution of this document is unlimited. Please send comments to the HTML working group (HTML-WG) of the Internet Engineering Task Force (IETF) at . Subscription address is . Discussions of the group are archived at . Abstract The Hypertext Markup Language (HTML) is a simple markup language used to create hypertext documents that are platform independent. Ini- tially, the application of HTML on the World Wide Web was seriously restricted by its reliance on the ISO-8859-1 coded character set, which is appropriate only for Western European languages. Despite this restriction, HTML has been widely used with other languages, using other coded character sets or character encodings, at the expense of interoperability. This document is meant to address the issue of the Expires 2 December 1996 [Page 1] Internet Draft HTML internationalization 27 May 1996 internationalization (i18n, i followed by 18 letters followed by n) of HTML by extending the specification of HTML and giving additional recommendations for proper internationalization support. A foremost consideration is to make sure that HTML remains a valid application of SGML, while enabling its use in all languages of the world. Table of contents 1. Introduction .................................................. 2 1.1. Scope ...................................................... 3 1.2. Conformance ................................................ 3 2. The document character set ..................................... 4 2.1. Reference processing model ................................. 4 2.2. The document character set ................................. 6 2.3. Undisplayable characters ................................... 8 3. The LANG attribute.............................................. 8 4. Additional entities, attributes and elements ................... 9 4.1. Full Latin-1 entity set .................................... 9 4.2. Markup for language-dependent presentation ................. 9 5. Forms ..........................................................15 5.1. DTD additions ..............................................15 5.2. Form submission ............................................15 6. Miscellaneous ..................................................17 7. HTML public text ...............................................18 7.1. HTML DTD ...................................................18 7.2. SGML declaration for HTML ..................................34 7.3. ISO Latin 1 character entity set ...........................35 Bibliography ......................................................38 Authors' Addresses ................................................40 1. Introduction The Hypertext Markup Language (HTML) is a simple markup language used to create hypertext documents that are platform independent. Ini- tially, the application of HTML on the World Wide Web was seriously restricted by its reliance on the ISO-8859-1 coded character set, which is appropriate only for Western European languages. Despite this restriction, HTML has been widely used with other languages, using other coded character sets or character encodings, through var- ious ad hoc extensions to the language [TAKADA]. This document is meant to address the issue of the internationaliza- tion of HTML by extending the specification of HTML and giving addi- tional recommendations for proper internationalization support. It is in good part based on a paper by one of the authors on multilin- gualism on the WWW [NICOL]. A foremost consideration is to make sure Expires 2 December 1996 [Page 2] Internet Draft HTML internationalization 27 May 1996 that HTML remains a valid application of SGML, while enabling its use in all languages of the world. The specific issues addressed are the SGML document character set to be used for HTML, the proper treatment of the charset parameter asso- ciated with the "text/html" content type and the specification of some additional elements and entities. 1.1 Scope HTML has been in use by the World-Wide Web (WWW) global information initiative since 1990. This specification extends the capabilities of HTML 2.0 (RFC 1866), primarily by removing the restriction to the ISO-8859-1 coded character set [ISO-8859-1]. HTML is an application of ISO Standard 8879:1986, Information Pro- cessing Text and Office Systems -- Standard Generalized Markup Lan- guage (SGML) [ISO-8879]. The HTML Document Type Definition (DTD) is a formal definition of the HTML syntax in terms of SGML. This specifi- cation amends the DTD of HTML in order to make it applicable to docu- ments encompassing a character repertoire much larger than that of ISO-8859-1, while still remaining SGML conformant. Both formal and actual development of HTML are advancing very fast. The features described in this document are designed so that they can (and should) be added to other forms of HTML besides that described in RFC 1866. Where indicated, attributes introduced here should be extended to the appropriate elements. 1.2 Conformance This specification changes slightly the conformance requirements of HTML documents and HTML user agents. 1.2.1 Documents All HTML 2.0 conforming documents remain conforming with this speci- fication. However, the extensions introduced here make valid cer- tains documents that would not be HTML 2.0 conforming, in particular those containing characters or character references outside of the repertoire of ISO 8859-1, and those containing markup introduced herein. Expires 2 December 1996 [Page 3] Internet Draft HTML internationalization 27 May 1996 1.2.2. User agents In addition to the requirements of RFC 1866, the following require- ments are placed on HTML user agents. To ensure interoperability and proper support for at least ISO-8859-1 in an environment where character encoding schemes other than ISO-8859-1 are present, user agents must correctly interpret the charset parameter accompanying an HTML document received from the network. Furthermore, conforming user-agents are required to at least parse correctly all numeric character references within the range of ISO 10646-1 [ISO-10646]. Conforming user-agents are required to apply the BIDI presentation algorithm if they display right-to-left characters. If there is no displayable right-to-left character in a document, there is no need to apply BIDI processing. 2. The document character set 2.1. Reference processing model This overview explains a reference processing model used for HTML, and in particular the SGML concept of a document character set. An actual implementation may widely differ in its internal workings from the model given below, but should behave as described to an outside observer. Because there are various widely differing encodings of text, SGML does not directly address the question of how characters are encoded e.g. in a file. SGML views the characters as a single set (called a "character repertoire"), and a "code set" that assigns an integer number (known as "character number") to each character in the reper- toire. The document character set declaration defines what each of the character numbers represents [GOLD90, p. 451]. In most cases, an SGML DTD and all documents that refer to it have a single document character set, and all markup and data characters are part of this set. HTML, as an application of SGML, does not directly address the ques- tion of how characters are encoded as octets in external representa- tions such as files. This is deferred to mechanisms external to HTML, such as MIME as used by the HTTP protocol or by electronic mail. For the HTTP protocol [RFC1945], the way characters are encoded is Expires 2 December 1996 [Page 4] Internet Draft HTML internationalization 27 May 1996 defined by the "charset" parameter[1] of the "Content-Type" field of the header of an HTTP response. For example, to indicate that the transmitted document is encoded in the "JIS" encoding of Japanese [RFC1468], the header will contain the following line: Content-Type: text/html; charset=ISO-2022-JP The HTTP protocol also defines a mechanism for the client to specify the character encodings it can accept. Clients and servers are strongly requested to use these mechanisms to assure correct trans- mission and interpretation of any document. Provisions that can be taken to help correct interpretation, even in cases where a server or client do not yet use these mechanisms, are described in section 6. Similarly, if HTML documents are transferred by electronic mail, the character encoding is defined by the "charset" parameter of the "Con- tent-Type" MIME header line [RFC1521], and defaults to US-ASCII in its absence. In the case any other way of transferring and storing HTML documents are defined or become popular, it is advised that similar provisions be made to clearly identify the character encoding used and/or to use a single/default encoding capable of representing the widest range of characters used in an international context. Whatever the external character encoding may be, the reference pro- cessing model translates it to a representation of the document char- acter set specified in Section 2.2 before processing specific to SGML/HTML. The reference processing model can be depicted as fol- lows: [resource]->[decoder]->[entity ]->[ SGML ]->[application]->[display] [manager] [parser] ^ | | | +----------+ The decoder is responsible for decoding the external representation of the resource to a representation using the document character set. The entity manager, the parser, and the application deal only with characters of the document character set. A display-oriented part of the application or the display machinery itself may again convert ----------- 1 The term "charset" in MIME is used to designate a char- acter encoding, rather than a coded character set as the term may suggest. A character encoding is a mapping (possi- bly many-to-one) of a sequence of octets to a sequence of characters taken from one or more character repertoires. Expires 2 December 1996 [Page 5] Internet Draft HTML internationalization 27 May 1996 characters represented in the document character set to some other representation more suitable for their purpose. In any case, the entity manager, the parser, and the application, as far as character semantics are concerned, are using the HTML document character set only. An actual implementation may choose, or not, to translate the docu- ment into some encoding of the document character set as described above; the behaviour described by this reference processing model can be achieved otherwise. This subject is well out of the scope of this specification, however, and the reader is invited to consult the SGML standard [ISO-8879] or an SGML handbook [BRYAN88] [GOLD90] [VANH90] [SQ91] for further information. The most important consequence of this reference processing model is that numeric character references are always resolved with respect to the fixed document character set, and thus to the same characters, whatever the external encoding actually used. For an example, see Section 2.2. 2.2. The document character set The document character set, in the SGML sense, is the Universal Char- acter Set (UCS) of ISO 10646:1993 [ISO-10646], as amended. Cur- rently, this is code-by-code identical with the Unicode standard, version 1.1 [UNICODE]. NOTE -- implementers should be aware that ISO 10646 is amended from time to time; 4 amendments have been adopted since the initial 1993 publication, none of which signifi- cantly affects this specification. A fifth amendment, now under consideration, will introduce incompatible changes to the standard: 6556 Korean Hangul syllables allocated between code positions 3400 and 4DFF (hexadecimal) will be moved to new positions (and 4516 new syllables added), thus making references to the old positions invalid. Since the Unicode consortium has already adopted a corresponding amendment for inclusion in the forthcoming Unicode 2.0, adoption of DAM 5 is considered likely and implementers should probably consider the old code positions as already invalid. Despite this one-time change, the relevant stan- dard bodies appear to remain committed not to change any allocated code position in the future. To encode Korean Hangul irrespective of these changes, the combining Hangul Jamo in the range 1110-11F9 can be used. The adoption of this document character set implies a change in the SGML declaration specified in the HTML 2.0 specification (section 9.5 Expires 2 December 1996 [Page 6] Internet Draft HTML internationalization 27 May 1996 of [RFC1866]). The change amounts to removing the first BASESET specification and its accompanying DESCSET declaration, replacing them with the following declaration: BASESET "ISO Registration Number 177//CHARSET ISO/IEC 10646-1:1993 UCS-4 with implementation level 3 //ESC 2/5 2/15 4/6" DESCSET 0 9 UNUSED 9 2 9 11 2 UNUSED 13 1 13 14 18 UNUSED 32 95 32 127 1 UNUSED 128 32 UNUSED 160 2147483486 160 Making the UCS the document character set does not create non- conformance of any expression, construct or document that is conform- ing to HTML 2.0. It does make conforming certain constructs that are not admissible in HTML 2.0. One consequence is that data characters outside the repertoire of ISO-8859-1, but within that of UCS-4 become valid SGML characters. Another is that the upper limit of the range of numeric character references is extended from 255 to 2147483645; thus, И is a valid reference to a "CYRILLIC CAPITAL LETTER I". [ERCS] is a good source of information on Unicode and SGML, although its scope and technical content differ greatly from this specifica- tion. NOTE -- the above SGML declaration, like that of HTML 2.0, specifies the character numbers 128 to 159 (80 to 9F hex) as UNUSED. This means that numeric character references within that range (e.g. ’) are illegal in HTML. Nei- ther ISO 8859-1 nor ISO 10646 contain characters in that range, which is reserved for control characters. ISO 10646-1:1993 is the most encompassing character set currently existing, and there is no other character set that could take its place as the document character set for HTML. If nevertheless for a specific application there is a need to use characters outside this standard, this should be done by avoiding any conflicts with present or future versions of ISO 10646, i.e. by assigning these characters to a private zone. Also, it should be borne in mind that such a use will be highly unportable; in many cases, it may be better to use inline bitmaps. Expires 2 December 1996 [Page 7] Internet Draft HTML internationalization 27 May 1996 2.3. Undisplayable characters With the document character set being the full ISO 10646, the possi- bility that a character cannot be displayed due to lack of appropri- ate resources (fonts) cannot be avoided. Because there are many dif- ferent things that can be done in such a case, this document does not prescribe any specific behaviour. Depending on the implementation, this may also be handled by the underlaying display system and not the application itself. The following considerations, however, may be of help: - A clearly visible, but unobtrusive behaviour should be preferred. Some documents may contain many characters that cannot be renden- dered, and so showing an alert for each of them is not the right thing to do. - In case a numeric representation of the missing character is given, its hexadecimal (not decimal) form is to be preferred, because this form is used in character set standards [ERCS]. 3. The LANG attribute Language tags can be used to control rendering of a marked up docu- ment in various ways: glyph disambiguation, in cases where the char- acter encoding is not sufficient to resolve to a specific glyph; quo- tation marks; hyphenation; ligatures; spacing; voice synthesis; etc. Independently of rendering issues, language markup is useful as con- tent markup for purposes such as classification and searching. Since any text can logically be assigned a language, almost all HTML elements admit the LANG attribute. The DTD reflects this. It is also intended that any new element introduced in later versions of HTML will admit the LANG attribute, unless there is a good reason not to do so. The language attribute, LANG, takes as its value a language tag that identifies a natural language spoken, written, or otherwise conveyed by human beings for communication of information to other human beings. Computer languages are explicitly excluded. The syntax and registry of HTML language tags is the same as that defined by RFC 1766 [RFC1766]. In summary, a language tag is composed of one or more parts: A primary language tag and a possibly empty series of subtags: language-tag = primary-tag *( "-" subtag ) primary-tag = 1*8ALPHA subtag = 1*8ALPHA Expires 2 December 1996 [Page 8] Internet Draft HTML internationalization 27 May 1996 Whitespace is not allowed within the tag and all tags are case- insensitive. The namespace of language tags is administered by the IANA. Example tags include: en, en-US, en-cockney, i-cherokee, x-pig-latin In the context of HTML, a language tag is not to be interpreted as a single token, as per RFC 1766, but as a hierarchy. For example, a user agent that adjusts rendering according to language should con- sider that it has a match when a language tag in a style sheet entry matches the initial portion of the language tag of an element. An exact match should be preferred. This interpretation allows an ele- ment marked up as, for instance, "en-US" to trigger styles corre- sponding to, in order of preference, US-English ("en-US") or 'plain' or 'international' English ("en"). NOTE -- using the language tag as a hierarchy does not imply that all languages with a common prefix will be understood by those fluent in one or more of those lan- guages; it simply allows the user to request this commonal- ity when it is true for that user. The rendering of elements may be affected by the LANG attribute. For any element, the value of the LANG attribute overrides the value specified by the LANG attribute of any enclosing element and the value (if any) of the HTTP Content-Language header. If none of these are set, a suitable default, perhaps controlled by user preferences, by automatic context analysis or by the user's locale, should be used to control rendering. 4. Additional entities, attributes and elements 4.1. Full Latin-1 entity set According to the suggestion of section 14 of [RFC1866], the set of Latin-1 entities is extended to cover the whole right part of ISO-8859-1 (all code positions with the high-order bit set), includ- ing the already commonly used  , © and ®. The names of the entities are taken from the appendices of SGML [ISO-8879]. A list is provided in section 7.3 of this specification. 4.2. Markup for language-dependent presentation 4.2.1. Overview For the correct presentation of text in certain languages (irrespec- tive of formatting issues), some support in the form of additional Expires 2 December 1996 [Page 9] Internet Draft HTML internationalization 27 May 1996 entities and elements is needed. In particular, the following features are dealt with: - Markup of bidirectional text, i.e. text where left-to-right and right-to-left scripts are mixed. - Control of cursive joining behaviour in contexts where the default behaviour is not appropriate. - Language-dependent rendering of short (in-line) quotations. - Better justification control for languages where this is impor- tant. - Superscripts and subscripts for languages where they appear as part of general text. Some of the above features need very little additional support; oth- ers need more. The additional features are introduced below with brief comments only. Explanations on cursive joining behaviour and bidirectional text follow later. For cursive joining behaviour and bidirectional text, this document follows [UNICODE] in that: i) char- acter semantics, where applicable, are identical to [UNICODE], and ii) where functionality is moved to HTML as a higher level protocol, this is done in a way that allows straightforward conversion to the lower-level mechanisms defined in [UNICODE]. 4.2.2. List of entities, elements, and attributes First, a generic container is needed to carry the LANG and DIR (see below) attributes in cases where no other element is appropriate; the SPAN element is introduced for that purpose. A set of named character entities is added for use with bidirectional rendering and cursive joining control: These entities can be used in place of the corresponding formatting characters whenever convenient, for example to ease keyboard entry or when a formatting character is not available in the character encod- ing of the document. Expires 2 December 1996 [Page 10] Internet Draft HTML internationalization 27 May 1996 Next, an attribute called DIR is introduced, restricted to the values LTR (left-to-right) and RTL (right-to-left) and admitted by most ele- ments, for the indication of directionality in the context of bidi- rectional text (see 4.2.4 below for details). Since any text and many other elements (e.g. tables) can logically be assigned a direc- tionality, almost all HTML elements admit the DIR attribute. The DTD reflects this. It is also intended that any new element introduced in later versions of HTML will admit the DIR attribute, unless there is a good reason not to do so. A new element called BDO (BIDI Override) is introduced, which requires the DIR attribute to specify whether the override is left- to-right or right-to-left. This element is required for bidirec- tional text control; for detailed explanations, see section 4.2.4. The element is introduced to allow language-dependent rendering of short quotations depending on language and platform capability. As the following examples show, in particular the quotation marks surrounding the quotation are affected: "a quotation in English", `another, slightly better one', ,,a quotation in German'', << a quo- tation in French >>. The contents of the element does not include quotation marks, they have to be added by the rendering pro- cess. NOTE -- elements can be nested. Many languages use dif- ferent quotation styles for outer and inner quotations, and this should be respected by user-agents implementing this element. Many languages require superscripts for proper rendering: as an exam- ple, the French "Mlle Dupont" should have "lle" in superscript. The element, and its sibling , are introduced to allow proper markup of such text. and contents are restricted to PCDATA to avoid nesting problems. Finally, in many languages text justification is much more important than it is in Western languages, and justifies markup. The ALIGN attribute, admitting values of LEFT, RIGHT, CENTER and JUSTIFY, is added to a selection of elements where it makes sense (block-like). If a user-agent chooses to have LEFT as a default for blocks of left- to-right directionality, it should use RIGHT for blocks of right-to- left directionality. In the DTD, the LANG and DIR attributes are grouped together in a parameter entity called attrs. In addition, the ID and CLASS attributes from RFC 1942 [RFC1942] were added to attrs, as was done in the latter. The ID, and CLASS attributes are required for use with style sheets, and RFC 1942 defines them as follows: Expires 2 December 1996 [Page 11] Internet Draft HTML internationalization 27 May 1996 ID Used to define a document-wide identifier. This can be used for naming positions within documents as the destination of a hypertext link. It may also be used by style sheets for ren- dering an element in a unique style. An ID attribute value is an SGML NAME token. NAME tokens are formed by an initial let- ter followed by letters, digits, "-" and "." characters. The letters are restricted to A-Z and a-z. CLASS A space separated list of SGML NAME tokens. CLASS names spec- ify that the element belongs to the corresponding named classes. It allows authors to distinguish different roles played by the same tag. The classes may be used by style sheets to provide different renderings as appropriate to these roles. 4.2.3. Cursive joining behaviour Markup is needed in some cases to force cursive joining behavior in contexts in which it would not normally occur, or to block it when it would normally occur. The zero-width joiner and non-joiner (‍ and ‌) are used to control cursive joining behaviour. For example, ARABIC LETTER HEH is used in isolation to abbreviate "Hijri" (the Islamic calendrical sys- tem); however, the initial form of the letter is desired, because the isolated form of HEH looks like the digit five as employed in Arabic script. This is obtained by following the HEH with a zero-width joiner whose only effect is to provide context. In Persian texts, there are cases where a letter that normally would join a subsequent letter in a cursive connection does not. Here a zero-width non- joiner is used. 4.2.4. Bidirectional text Many languages are written in horizontal lines from left to right, while others are written from right to left. When both writing directions are present, one talks of bidirectional text (BIDI for short). BIDI text requires markup in special circumstances where ambiguities as to the directionality of some characters have to be resolved. This markup affects the ability to render BIDI text in a semantically legible fashion. That is, without this special BIDI markup, cases arise which would prevent *any* rendering whatsoever that reflected the basic meaning of the text. Plain text may contain this markup (joining or BIDI) in the form of special-purpose charac- ters; in HTML, these are supplemented by SGML markup. BIDI is a complex issue, and implementers are advised to consult appropriate documentation such as [UNICODE]. Here, explanations are Expires 2 December 1996 [Page 12] Internet Draft HTML internationalization 27 May 1996 given only as far as they are needed to understand the necessity of the features introduced and to define their exact semantics. The Unicode BIDI algorithm is based on a logical sequence of text characters and works mainly by reference to the implicit directional- ity of characters (e.g. Hebrew and Arabic characters are specified to be rendered from right to left, etc.). The left-to-right and right-to-left marks (‎ and ‏) are used to disambiguate directionality of neutral characters. For example, when a double quote sits between an Arabic and a Latin letter, its direction is ambiguous; if a directional mark is added on one side such that the quotation mark is surrounded by characters of only one directionality, the ambiguity is removed. These characters are like zero width spaces which have a directional property (but no word/line break property). Nested embeddings of contra-directional text runs, due to nested quo- tations or to the pasting of text from one BIDI context to another, is also a case where the implicit directionality of characters is not sufficient, requiring markup. Also, it is frequently desirable to specify the basic directionality of a block of text. For these pur- poses, the DIR attribute is used. On block-type elements, the DIR attribute indicates the base direc- tionality of the text in the block; if omitted it is inherited from the parent element. The default directionality of the overall HTML document is left-to-right. On inline elements, it makes the element start a new embedding level (to be explained below); if omitted the inline element does not start a new embedding level. NOTE -- the PRE, XMP and LISTING elements admit the DIR attribute, indicating that the contents should not be con- sidered as preformatted with respect to bidirectional lay- out. The BIDI algorithm still needs to be applied to each line of text. Following is an example of a case where embedding is needed, showing its effect: Given the following latin (upper case) and arabic (lower case) letters in backing store with the specified embed- dings: AB xy CD zw EF Expires 2 December 1996 [Page 13] Internet Draft HTML internationalization 27 May 1996 One gets the following rendering (with [] showing the directional transitions): [ AB [ wz [ CD ] yx ] EF ] On the other hand, without this markup and with a base direction of LTR one gets the following rendering: [ AB [ yx ] CD [ wz ] EF ] Notice that yx is on the left and wz on the right unlike the above case where the embedding levels are used. With- out the embedding markup one has at most two levels: a base directional level and a single counterflow directional level. The DIR attribute on inline elements is equivalent to the formatting characters LEFT-TO-RIGHT EMBEDDING (202A) and RIGHT-TO-LEFT EMBED- DING (202B) of ISO 10646. The end tag of the element is equivalent to the POP DIRECTIONAL FORMATTING (202C) character. Directional override, as provided by the element, is needed to deal with unusual short pieces of text in which directionality cannot be resolved from context in an unambiguous fashion. For example, it can be used to force left-to-right (or right-to-left) display of part numbers composed of Latin letters, digits and Hebrew letters. The effect of is to force the directionality of all characters within it to the value of DIR, irrespective of their intrinsic direc- tional properties. It is equivalent to using the LEFT-TO-RIGHT OVER- RIDE (202D) or RIGHT-TO-LEFT OVERRIDE (202E) characters of ISO 10646, the end tag again being equivalent to the POP DIRECTIONAL FORMATTING (202C) character. NOTE -- authors and authoring software writers should be aware that conflicts can arise if the DIR attribute is used on inline elements (including ) concurrently with the use of the corresponding ISO 10646 formatting characters. Preferably one or the other should be used exclusively; the markup method is better able to guarantee document struc- tural integrity, and alleviates some problems when editing bidirectional HTML text with a simple text editor, but some software may be more apt at using the 10646 characters. If both methods are used, great care should be exercised to insure proper nesting of markup and directional embedding or override; otherwise, rendering results are undefined. Expires 2 December 1996 [Page 14] Internet Draft HTML internationalization 27 May 1996 5. Forms 5.1. DTD additions It is natural to expect input in any language in forms, as they pro- vide one of the only ways of obtaining user input. While this is pri- marily a UI issue, there are some things that should be specified at the HTML level to guide behavior and promote interoperability. To ensure full interoperability, it is necessary for the user agent (and the user) to have an indication of the character encoding(s) that the server providing a form will be able to handle upon submis- sion of the filled-in form. Such an indication is provided by the ACCEPT-CHARSET attribute of the INPUT and TEXTAREA elements, modeled on the HTTP Accept-Charset header (see [HTTP-1.1]), which contains a space and/or comma delimited list of character sets acceptable to the server. A user agent may want to somehow advise the user of the con- tents of this attribute, or to restrict his possibility to enter characters outside the repertoires of the listed character sets. NOTE -- The list of character sets is to be interpreted as an EXCLUSIVE-OR list; the server announces that it is ready to accept any ONE of these character encoding schemes for each part of a multipart entity. The client may perform character encoding translation to satisfy the server if necessary. NOTE -- The default value for the ACCEPT-CHARSET attribute of an INPUT or TEXTAREA element is the reserved value "UNKNOWN". A user agent may interpret that value as the character encoding scheme that was used to transmit the document containing that element. 5.2. Form submission The HTML 2.0 form submission mechanism, based on the "application/x- www-form-urlencoded" media type, is ill-equipped with regard to internationalization. In fact, since URLs are restricted to ASCII characters, the mechanism is akward even for ISO-8859-1 text. Sec- tion 2.2 of [RFC1738] specifies that octets may be encoded using the "%HH" notation, but text submitted from a form is composed of charac- ters, not octets. Lacking a specification of a character encoding scheme, the "%HH" notation has no well-defined meaning. The best solution is to use the "multipart/form-data" media type described in [RFC1867] with the POST method of form submission. This Expires 2 December 1996 [Page 15] Internet Draft HTML internationalization 27 May 1996 mechanism encapsulates the value part of each name-value pair in a body-part of a multipart MIME body that is sent as the HTTP entity; each body part can be labeled with an appropriate Content-Type, including if necessary a charset parameter that specifies the charac- ter encoding scheme. The changes to the DTD necessary to support this method of form submission have been incorporated in the DTD included in this specification. A less satisfactory solution is to add a MIME charset parameter to the "application/x-www-form-urlencoded" media type specifier sent along with a POST method form submission, with the understanding that the URL encoding of [RFC1738] is applied on top of the specified character encoding, as a kind of implicit Content-Transfer-Encoding. One problem with both solutions above is that current browsers do not generally allow for bookmarks to specify the POST method; this should be improved. Conversely, the GET method could be used with the form data transmitted in the body instead of in the URL. Nothing in the protocol seems to prevent it, but no implementations appear to exist at present. How the user agent determines the encoding of the text entered by the user is outside the scope of this specification. NOTE -- Designers of forms and their handling scripts should be aware of an important caveat: when the default value of a field (the VALUE attribute) is returned upon form submission (i.e. the user did not modify this value), it cannot be guaranteed to be transmitted as a sequence of octets identical to that in the source document -- only as a possibly different but valid encoding of the same sequence of text elements. This may be true even if the encoding of the document containing the form and that used for submission are the same. Differences can occur when a sequence of characters can be represented by various sequences of octets, and also when a composite sequence (a base character plus one or more com- bining diacritics) can be represented by either a different but equivalent composite sequence or by a fully precomposed character. For instance, the UCS-2 sequence 00EA+0232 (LATIN SMALL LETTER E WITH CIRCUMFLEX ACCENT + COMBINING DOT BELOW) may be transformed into 1EC7 (LATIN SMALL LETTER E WITH CIRCUMFLEX ACCENT AND DOT BELOW), into 0065+0302+0323 (LATIN SMALL LETTER E + COMBINING CIRCUMFLEX ACCENT + COMBINING DOT BELOW), as well as into other equiv- alent composite sequences. Expires 2 December 1996 [Page 16] Internet Draft HTML internationalization 27 May 1996 6. Miscellaneous Proper interpretation of a text document requires that the character encoding scheme be known. Current HTTP servers, however, do not gen- erally include an appropriate charset parameter with the Content-Type header. This is bad behaviour[2], and as such strongly discouraged, but some preventive measures can be taken to minimize the detrimental effects. In the case where a document is accessed from a hyperlink in an ori- gin HTML document, a CHARSET attribute is added to the attribute list of elements with link semantics (A and LINK), specifically by adding it to the linkExtraAttributes entity. The value of that attribute is to be considered a hint to the User Agent as to the character encod- ing scheme used by the ressource pointed to by the hyperlink; it should be the appropriate value of the MIME charset parameter for that ressource. In any document, it is possible to include an indication of the encoding scheme like the following, as early as possible within the HEAD of the document: This is not foolproof, but will work if the encoding scheme is such that ASCII characters stand for themselves at least until the META element is parsed. Note that there are better ways for a server to obtain character encoding information, instead of the unreliable above; see [NICOL2] for some details and a proposal. For definiteness, the "charset" parameter received from the source of the document should be considered the most authoritative, followed in order of preference by the contents of a META element such as the above, and finally the CHARSET parameter of the anchor that was fol- lowed (if any). When HTML text is transmitted directly in UCS-2 or UCS-4 form, the question of byte order arises: does the high-order byte of each multi-byte character come first or last? For definiteness, this specification recommends that UCS-2 and UCS-4 be transmitted in big- ----------- 2 This bad behaviour is even encouraged by the continued existence of browsers that declare an unrecognized media type when they receive a charset parameter. User agent implementators are strongly encouraged to make their soft- ware tolerant of this parameter, even if they cannot take advantage of it. Expires 2 December 1996 [Page 17] Internet Draft HTML internationalization 27 May 1996 endian byte order (high order byte first), which corresponds to the established network byte order for two- and four-byte quantities, to the Unicode recommendation for serialized text data and to RFC 1641. Furthermore, to maximize chances of proper interpretation, it is rec- ommended that documents transmitted as UCS-2 or UCS-4 always begin with a ZERO-WIDTH NON-BREAKING SPACE character (hexadecimal FEFF or 0000FEFF) which, when byte-reversed becomes number FFFE or FFFE0000, a character guaranteed to be never assigned. Thus, a user-agent receiving an FFFE as the first octets of a text would know that bytes have to be reversed for the remainder of the text. There exist so-called UCS Transformation Formats than can be used to transmit UCS data, in addition to UCS-2 and UCS-4. UTF-7 [RFC1642] and UTF-8 [UTF-8] have favorable properties (no byte-ordering prob- lem, different flavours of ASCII compatibility) that make them worthy of consideration, especially for transmission of multilingual text. Another encoding scheme, MNEM [RFC1345], also has interesting proper- ties and the capability to transmit the full UCS. The UTF-1 trans- formation format of ISO 10646:1993 (registered by IANA as ISO-10646-UTF-1), has been removed from ISO 10646 by amendment 4, and should not be used. The SOFT HYPHEN character (U+00AD) needs a little attention from user-agent implementers. It is present in many character sets (including the whole ISO 8859 series and, of course, ISO 10646), and has semantics different from the plain HYPHEN. If not used for hyphenation, the soft hyphen must be completely ignored. For exam- ple, "rec­ord" should display as "record", should match a search for "record", and should sort as "record". Non-observance of these semantics effectively discourages its use on the World Wide Web, even with software that does support it. 7. HTML Public Text 7.1. HTML DTD This section contains a DTD for HTML based on the HTML 2.0 DTD of RFC 1866, incorporating the changes for file upload as specified in RFC 1867, and the changes deriving from this document. ... -- > ]]> Expires 2 December 1996 [Page 19] Internet Draft HTML internationalization 27 May 1996 %ISOlat1; Expires 2 December 1996 [Page 20] Internet Draft HTML internationalization 27 May 1996 Expires 2 December 1996 [Page 21] Internet Draft HTML internationalization 27 May 1996 ]]> Heading is preferred to Expires 2 December 1996 [Page 23] Internet Draft HTML internationalization 27 May 1996

Heading

--> ]]> " > #AttVal(Alt)" > Expires 2 December 1996 [Page 24] Internet Draft HTML internationalization 27 May 1996 Expires 2 December 1996 [Page 25] Internet Draft HTML internationalization 27 May 1996 ]]> ]]> ]]> Expires 2 December 1996 [Page 27] Internet Draft HTML internationalization 27 May 1996 Directory" > Menu" > Expires 2 December 1996 [Page 28] Internet Draft HTML internationalization 27 May 1996 Heading

Text ... is preferred to

Heading

Text ... --> ]]> Form:" %SDASUFF; "Form End." > Expires 2 December 1996 [Page 30] Internet Draft HTML internationalization 27 May 1996 Select #AttVal(Multiple)" > ]]> ]]> " > Expires 2 December 1996 [Page 32] Internet Draft HTML internationalization 27 May 1996 [Document is indexed/searchable.]"> ]]> 7.2. SGML Declaration for HTML 7.3. ISO Latin 1 entity set The following public text lists each of the characters specified in the Added Latin 1 entity set, along with its name, syntax for use, and description. This list is derived from ISO Standard 8879:1986//ENTITIES Added Latin 1//EN. HTML includes the entire entity set, and adds entities for all missing characters in the right part of ISO-8859-1. Expires 2 December 1996 [Page 35] Internet Draft HTML internationalization 27 May 1996 Expires 2 December 1996 [Page 36] Internet Draft HTML internationalization 27 May 1996 Expires 2 December 1996 [Page 37] Internet Draft HTML internationalization 27 May 1996 Bibliography [BRYAN88] M. Bryan, "SGML -- An Author's Guide to the Standard Generalized Markup Language", Addison-Wesley, Reading, 1988. [ERCS] Extended Reference Concrete Syntax for SGML. [GOLD90] C. F. Goldfarb, "The SGML Handbook", Y. Rubinsky, Ed., Oxford University Press, 1990. [HTTP-1.1] R.T. Fielding, H. Frystyk Nielsen, and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", Work in progress (draft-ietf-http-v11-spec-03.txt), MIT/LCS, May 1996. [ISO-639] ISO 639:1988. Codes pour la représentation des noms de langue. Technical content in [ISO-3166] ISO 3166:1993. Codes pour la représentation des noms de pays. [ISO-8601] ISO 8601:1988. Éléments de données et formats d'échange -- Échange d'information -- Représentation de la date et de l'heure. [ISO-8859-1] ISO 8859-1:1987. International Standard -- Informa- tion Processing -- 8-bit Single-Byte Coded Graphic Character Sets -- Part 1: Latin Alphabet No. 1. [ISO-8879] ISO 8879:1986. International Standard -- Information Processing -- Text and Office Systems -- Standard Gen- eralized Markup Language (SGML). Expires 2 December 1996 [Page 38] Internet Draft HTML internationalization 27 May 1996 [ISO-10646] ISO/IEC 10646-1:1993. International Standard -- Infor- mation technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane. [NICOL] G.T. Nicol, "The Multilingual World Wide Web", Elec- tronic Book Technologies, 1995, [NICOL2] G.T. Nicol, "MIME Header Supplemented File Type", Work in progress, , EBT, October 1995. [RFC1345] K. Simonsen, "Character Mnemonics & Character Sets", RFC 1345, Rationel Almen Planlaegning, June 1992. [RFC1468] J. Murai, M. Crispin and E. van der Poel, "Japanese Character Encoding for Internet Messages", RFC 1468, Keio University, Panda Programming, June 1993. [RFC1521] N. Borenstein and N. Freed, "MIME (Multipurpose Inter- net Mail Extensions) Part One: Mechanisms for Specify- ing and Describing the Format of Internet Message Bod- ies", RFC 1521, Bellcore, Innosoft, September 1993. [RFC1641] D. Goldsmith, M.Davis, "Using Unicode with MIME", RFC 1641, Taligent inc., July 1994. [RFC1642] D. Goldsmith, M. Davis, "UTF-7: A Mail-safe Transfor- mation Format of Unicode", RFC 1642, Taligent inc., July 1994. [RFC1738] T. Berners-Lee, L. Masinter, and M. McCahill, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox PARC, University of Minnesota, October 1994. [RFC1766] H. Alverstrand, "Tags for the Identification of Lan- guages", RFC 1766, UNINETT, March 1995. [RFC1866] T. Berners-Lee and D. Connolly, "Hypertext Markup Lan- guage - 2.0", RFC 1866, MIT/W3C, November 1995. [RFC1867] E. Nebel and L. Masinter, "Form-based File Upload in HTML", RFC 1867, Xerox Corporation, November 1995. [RFC1942] D. Raggett, "HTML Tables", RFC 1942, W3C, May 1996. Expires 2 December 1996 [Page 39] Internet Draft HTML internationalization 27 May 1996 [RFC1945] T. Berners-Lee, R.T. Fielding, and H. Frystyk Nielsen, "Hypertext Transfer Protocol -- HTTP/1.0", RFC 1945, MIT/LCS, UC Irvine, May 1996. [SQ91] SoftQuad, "The SGML Primer", 3rd ed., SoftQuad Inc., 1991. [TAKADA] Toshihiro Takada, "Multilingual Information Exchange through the World-Wide Web", Computer Networks and ISDN Systems, Vol. 27, No. 2, Nov. 1994 , p. 235-241. [TEI] TEI Guidelines for Electronic Text Encoding and Inter- change. [UNICODE] The Unicode Consortium, "The Unicode Standard -- Worldwide Character Encoding -- Version 1.0", Addison- Wesley, Volume 1, 1991, Volume 2, 1992, and Technical Report #4, 1993. The BIDI algorithm is in appendix A of volume 1, with corrections in appendix D of volume 2. [UTF-8] ISO/IEC 10646-1:1993 AMENDMENT 2 (1996). UCS Transfor- mation Format 8 (UTF-8). [VANH90] E. van Hervijnen, "Practical SGML", Kluwer Academicq Publishers Group, Norwell and Dordrecht, 1990. Authors' Addresses François Yergeau Alis Technologies 100, boul. Alexis-Nihon, bureau 600 Montréal QC H4M 2P2 Canada Tel: +1 (514) 747-2547 Fax: +1 (514) 747-2561 EMail: fyergeau@alis.com Gavin Thomas Nicol Electronic Book Technologies, Japan 1-29-9 Tsurumaki, Setagaya-ku, Tokyo Japan Tel: +81-3-3230-8161 Expires 2 December 1996 [Page 40] Internet Draft HTML internationalization 27 May 1996 Fax: +81-3-3230-8163 EMail: gtn@ebt.com, gtn@twics.co.jp Glenn Adams Spyglass 118 Magazine Street Cambridge, MA 02139 U.S.A. Tel: +1 (617) 864-5524 Fax: +1 (617) 864-4965 EMail: glenn@spyglass.com Martin J. Duerst Multimedia-Laboratory Department of Computer Science University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland Tel: +41 1 257 43 16 Fax: +41 1 363 00 35 E-mail: mduerst@ifi.unizh.ch Expires 2 December 1996 [Page 41]