www/webidl2tests/readme.en.html

<!DOCTYPE HTML>
<html lang=en>
<title>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</title>
<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
<link rel=license href="#license">

<h1>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</h1>

<div class=section id=intro>
<h2>What is this?</h2>

<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
IDL fragment conforming to the WebIDL specification.  It generates
test cases to check whether the DOM ECMAScript binding of a Web
browser conforms to the <a href="#standards">Web IDL specification</a>
and the given IDL fragment, as well as a test harness for the
automated testing against the generated test cases.

<p>In addition, a test result submission script is available.  Test
results can be submitted from the test harness.  The browser support
matrix for the test suite is automatically generated from the
submitted results.

</div>

<div class=section id=usage>
<h2>Usage</h2>

<ol>

<li id=prepare-idl>

<p>First, prepare an IDL fragment that represents interfaces and
exceptions that should be implemented by Web browsers.  The IDL
fragment must conform to the <a href="#standards">Web IDL
specification</a>.

<p>Ensure that the IDL fragment is conforming.  <em>A broken input
might generate broken test cases.</em> Use conformance checker such as
<a href="http://suika.fam.cx/gate/2007/html/cc/">WebHACC</a> to detect
any possible error.

<li id=prepare-json>

<p>Prepare a JSON file that contains supplemental information on
interfaces and exceptions to be tested.  The JSON file can contain two
kinds of information: <a href="#instantiate">instantiation
instructions</a> and <a href="#crash">crash data</a>.

<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
fragment does not describe possible ways to instantiate interfaces or
exceptions defined in the fragment, it is necessary to tell the script
how to create host objects and host exception objects implementing
those interfaces and exceptions.

<p>The JSON file can contain instantiation instruction data as follows:
<pre class="json syntax code"><code>
{
  "<var>InterfaceName</var>": [
    {
      "id": "<var>create-object-1</var>",
      "code": "<var>code.to.createObject (1);</var>"
    }
    <var>, ...</var>
  ]
  <var>, ...</var>
}
</code></pre>

<p>... where <var>InterfaceName</var> is the name of the interface or
exception as used in the ECMAScript binding,
<var>create-object-1</var> is the identifier that distinguish
different ways to create an object, and <var>code.to.createObject
(1);</var> is a right-hand part of ECMAScript assignment statement to
create an object that implements the interface or exception.  Note
that the identifier (i.e. <var>create-object-1</var>) must be unique
for an interface or exception.  It must also be unique globally,
except when the code (i.e. <var>code.to.createObject (1);</var>) is
not different each other.

<p>For example, the JSON file for an IDL fragment that contains the
definition for the <code>NodeList</code> interface would contain:

<pre class="example json code"><code>
{
  "NodeList": [
    {
      "id": "child-nodes",
      "code": "document.createElement ('p').childNodes;",
    },
    {
      "id": "get-elements",
      "code": "document.getElementsByTagName ('*');"
    }
  ]
}
</code></pre>

<p>If the instantiation instruction for an interface or exception is
not provided, then any test case that requires an instance of the
interface or exception will not be generated.

<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
browser.  Since the existence of a crashing test case prevents the use
of automated test harness, a way to avoid execution of test case that
is known to crash a certain browser is provided.

The crash data can be put into the JSON file with the
<code>_crash</code> key as follows:

<pre class="json syntax code"><code>
{
  <var>... instantiation instruction ...</var>
  "_crash": {
    "<var>test-case-id</var>": [
      "<var>UserAgentNameRegularExpression</var>"
      <var>, ...</var>
    ]
    <var>, ...</var>
  }
}
</code></pre>

<p>... where <var>test-case-id</var> is the identifier of the relevant
test case (i.e. the file name of the test case, except for the
<code>.html</code> extension) and
<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
expression, which will be evaluated by ECMAScript 
<code>new RegExp (<var>regexp</var>)</code> constructor, representing
<code>User-Agent</code> strings of Web browsers that are known to
crash.

<li id=execute-wttjs>

<p>Then, generate the test files by executing the
<code>wttjs.pl</code> script as follows:

<pre>
  $ <kbd>perl wttjs.pl \
        --idl-file-name <var>input.idl</var> \
        --instances-file-name <var>input.json</var> \
        --test-dir-name <var>output-dir/</var> \
        --testset-id <var>testname</var></kbd>
</pre>

<p>... where <var>input.idl</var> is the name of the IDL fragment file
prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
is the name of the JSON file prepared in <a href="#prepare-json">Step
2</a>, <var>output-dir/</var> is the name of the directory in which a
directory for the test files is created, and <var>testname</var> is a
short identifier for the test suite.  For more information on
command-line arguments, see <a href="bin/wttjs.html">the documentation
for the <code>wttjs.pl</code> script</a>.

<li>

<p>At this time, there should be a set of test files for the IDL
fragment, in the directory specified as the command-line argument.

<p>Open <code>all.html</code> in the set by a Web browser that is to
be tested.  The test is automatically started and the progress is
shown in the page until all the tests are done.

<li id=test-done>

<p>Examine the test results.  Lists of failed tests and skipped tests
are shown in the page, as well as the numbers of passed, failed, and
skipped tests.

<p>The test result can be submitted to the <a href="#results">test
result form</a> using a submission button that appears in the page.

</ol>

</div>

<div class=section id=results>
<h2>Test results</h2>

<p>Test results can be submitted so that results of Web browsers can
be compared.

<p>A result table can be prepared for each test suite.  To create a
table, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
where <var>testset-id</var> is the identifier of the test suite (the
one <a href="#execute-wttjs">specified</a> to the
<code>wttjs.pl</code> script) and fill in the form.

<p>Then, submit test results using <a href="#test-done">the submission
button in the test harness</a>.

<p>Finally, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
which lists the result of various browsers.

<p>Note that the source code of the server-side script for the test
result submission is also <a href="#download">available</a>.

</div>

<div class=section id=standards>
<h2>Supported standards</h2>

<p>This script supports:

<ul>

<li>ECMA 262 Third Edition.

<li><a
href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&amp;content-type=text/html;%20charset=utf-8">Web
IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.

</ul>

<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
WebIDL interface/exception constraints.  For example, whether an
attribute returns a value of IDL type <code>long</code> or not is
<em>not</em> tested.  Likewise, whether an attribute with extended
attribute <code>[Null=Empty]</code> really coverts <code>null</code>
to an empty string is <em>not</em> tested, either.  Therefore,
<em>passing all tests generated by the <code>wttjs.pl</code> script
does not necessarily mean the implementation is conforming</em>.

</div>

<div class=section id=download>
<h2>Download</h2>

<p>Latest version of scripts are available via the CVS repository: 

<ul>

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
<code>webroot/www/webidl2tests/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
tarball</a>).

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
submission: <code>webroot/gate/test-results/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
tarball</a>).

</ul>


</div>

<div class=section id=dependency>
<h2>Dependency</h2>

<p>For <code>wttjs.pl</code>: Perl5 or later, <a
href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
and the <code>JSON</code> module.  See <a
href="bin/wttjs.html#dependency">the document</a> for details.

<p>For the test result script: CGI-enabled HTTP server, Perl5 or
later, and <code>Message::CGI::HTTP</code> and
<code>Message::DOM::DOMImplementation</code> modules from the <a
href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
package.

</div>

<div class=section id=example>
<h2>Examples</h2>

<ul>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
DOM Core WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
Level 2 WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
<em>(incomplete)</em>) &mdash; <a
href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
Table</a>

</ul>

</div>

<div class=section id=author>
<h2>Author</h2>

<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.

</div>

<div class=section id=license>
<h2>License</h2>

<p>License of the <em>scripts and documentations</em> (not including
generated files) is as follows:

<blockquote>
<p>Copyright 2008 <a
href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
class=mail>&lt;<a
href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>

<p>This is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
</blockquote>

<p>Note that you may distribute files you generate by these scripts in
any license as you like.

</div>
1	wakaba	1.1	<!DOCTYPE HTML>
2			<html lang=en>
3			<title>wttjs — WebIDL ECMAScript Binding Test Suite Generator</title>
4			<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
5			<link rel=license href="#license">
6
7			<h1>wttjs — WebIDL ECMAScript Binding Test Suite Generator</h1>
8
9			<div class=section id=intro>
10			<h2>What is this?</h2>
11
12			<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
13			IDL fragment conforming to the WebIDL specification. It generates
14			test cases to check whether the DOM ECMAScript binding of a Web
15			browser conforms to the <a href="#standards">Web IDL specification</a>
16			and the given IDL fragment, as well as a test harness for the
17			automated testing against the generated test cases.
18
19			<p>In addition, a test result submission script is available. Test
20			results can be submitted from the test harness. The browser support
21			matrix for the test suite is automatically generated from the
22			submitted results.
23
24			</div>
25
26			<div class=section id=usage>
27			<h2>Usage</h2>
28
29			<ol>
30
31			<li id=prepare-idl>
32
33			<p>First, prepare an IDL fragment that represents interfaces and
34			exceptions that should be implemented by Web browsers. The IDL
35			fragment must conform to the <a href="#standards">Web IDL
36			specification</a>.
37
38			<p>Ensure that the IDL fragment is conforming. <em>A broken input
39			might generate broken test cases.</em> Use conformance checker such as
40	wakaba	1.2	<a href="http://suika.fam.cx/gate/2007/html/cc/">WebHACC</a> to detect
41			any possible error.
42	wakaba	1.1
43			<li id=prepare-json>
44
45			<p>Prepare a JSON file that contains supplemental information on
46			interfaces and exceptions to be tested. The JSON file can contain two
47			kinds of information: <a href="#instantiate">instantiation
48			instructions</a> and <a href="#crash">crash data</a>.
49
50			<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
51			fragment does not describe possible ways to instantiate interfaces or
52			exceptions defined in the fragment, it is necessary to tell the script
53			how to create host objects and host exception objects implementing
54			those interfaces and exceptions.
55
56			<p>The JSON file can contain instantiation instruction data as follows:
57			<pre class="json syntax code"><code>
58			{
59			"<var>InterfaceName</var>": [
60			{
61			"id": "<var>create-object-1</var>",
62			"code": "<var>code.to.createObject (1);</var>"
63			}
64			<var>, ...</var>
65			]
66			<var>, ...</var>
67			}
68			</code></pre>
69
70			<p>... where <var>InterfaceName</var> is the name of the interface or
71			exception as used in the ECMAScript binding,
72			<var>create-object-1</var> is the identifier that distinguish
73			different ways to create an object, and <var>code.to.createObject
74			(1);</var> is a right-hand part of ECMAScript assignment statement to
75			create an object that implements the interface or exception. Note
76			that the identifier (i.e. <var>create-object-1</var>) must be unique
77			for an interface or exception. It must also be unique globally,
78			except when the code (i.e. <var>code.to.createObject (1);</var>) is
79			not different each other.
80
81			<p>For example, the JSON file for an IDL fragment that contains the
82			definition for the <code>NodeList</code> interface would contain:
83
84			<pre class="example json code"><code>
85			{
86			"NodeList": [
87			{
88			"id": "child-nodes",
89			"code": "document.createElement ('p').childNodes;",
90			},
91			{
92			"id": "get-elements",
93			"code": "document.getElementsByTagName ('*');"
94			}
95			]
96			}
97			</code></pre>
98
99			<p>If the instantiation instruction for an interface or exception is
100			not provided, then any test case that requires an instance of the
101			interface or exception will not be generated.
102
103			<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
104			browser. Since the existence of a crashing test case prevents the use
105			of automated test harness, a way to avoid execution of test case that
106			is known to crash a certain browser is provided.
107
108			The crash data can be put into the JSON file with the
109			<code>_crash</code> key as follows:
110
111			<pre class="json syntax code"><code>
112			{
113			<var>... instantiation instruction ...</var>
114			"_crash": {
115			"<var>test-case-id</var>": [
116			"<var>UserAgentNameRegularExpression</var>"
117			<var>, ...</var>
118			]
119			<var>, ...</var>
120			}
121			}
122			</code></pre>
123
124			<p>... where <var>test-case-id</var> is the identifier of the relevant
125			test case (i.e. the file name of the test case, except for the
126			<code>.html</code> extension) and
127			<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
128			expression, which will be evaluated by ECMAScript
129			<code>new RegExp (<var>regexp</var>)</code> constructor, representing
130			<code>User-Agent</code> strings of Web browsers that are known to
131			crash.
132
133			<li id=execute-wttjs>
134
135			<p>Then, generate the test files by executing the
136			<code>wttjs.pl</code> script as follows:
137
138			<pre>
139			$ <kbd>perl wttjs.pl \
140			--idl-file-name <var>input.idl</var> \
141			--instances-file-name <var>input.json</var> \
142			--test-dir-name <var>output-dir/</var> \
143			--testset-id <var>testname</var></kbd>
144			</pre>
145
146			<p>... where <var>input.idl</var> is the name of the IDL fragment file
147			prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
148			is the name of the JSON file prepared in <a href="#prepare-json">Step
149			2</a>, <var>output-dir/</var> is the name of the directory in which a
150			directory for the test files is created, and <var>testname</var> is a
151			short identifier for the test suite. For more information on
152			command-line arguments, see <a href="bin/wttjs.html">the documentation
153			for the <code>wttjs.pl</code> script</a>.
154
155			<li>
156
157			<p>At this time, there should be a set of test files for the IDL
158			fragment, in the directory specified as the command-line argument.
159
160			<p>Open <code>all.html</code> in the set by a Web browser that is to
161			be tested. The test is automatically started and the progress is
162			shown in the page until all the tests are done.
163
164			<li id=test-done>
165
166			<p>Examine the test results. Lists of failed tests and skipped tests
167			are shown in the page, as well as the numbers of passed, failed, and
168			skipped tests.
169
170			<p>The test result can be submitted to the <a href="#results">test
171			result form</a> using a submission button that appears in the page.
172
173			</ol>
174
175			</div>
176
177			<div class=section id=results>
178			<h2>Test results</h2>
179
180			<p>Test results can be submitted so that results of Web browsers can
181			be compared.
182
183			<p>A result table can be prepared for each test suite. To create a
184			table, access <code
185			class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
186			where <var>testset-id</var> is the identifier of the test suite (the
187			one <a href="#execute-wttjs">specified</a> to the
188			<code>wttjs.pl</code> script) and fill in the form.
189
190			<p>Then, submit test results using <a href="#test-done">the submission
191			button in the test harness</a>.
192
193			<p>Finally, access <code
194			class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
195			which lists the result of various browsers.
196
197			<p>Note that the source code of the server-side script for the test
198			result submission is also <a href="#download">available</a>.
199
200			</div>
201
202			<div class=section id=standards>
203			<h2>Supported standards</h2>
204
205			<p>This script supports:
206
207			<ul>
208
209			<li>ECMA 262 Third Edition.
210
211			<li><a
212			href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&content-type=text/html;%20charset=utf-8">Web
213			IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.
214
215			</ul>
216
217			<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
218			WebIDL interface/exception constraints. For example, whether an
219			attribute returns a value of IDL type <code>long</code> or not is
220			<em>not</em> tested. Likewise, whether an attribute with extended
221			attribute <code>[Null=Empty]</code> really coverts <code>null</code>
222			to an empty string is <em>not</em> tested, either. Therefore,
223			<em>passing all tests generated by the <code>wttjs.pl</code> script
224			does not necessarily mean the implementation is conforming</em>.
225
226			</div>
227
228			<div class=section id=download>
229			<h2>Download</h2>
230
231			<p>Latest version of scripts are available via the CVS repository:
232
233			<ul>
234
235			<li><a
236			href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
237			<code>webroot/www/webidl2tests/</code></a> (<a
238			href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
239			tarball</a>).
240
241			<li><a
242			href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
243			submission: <code>webroot/gate/test-results/</code></a> (<a
244			href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
245			tarball</a>).
246
247			</ul>
248
249
250			</div>
251
252			<div class=section id=dependency>
253			<h2>Dependency</h2>
254
255			<p>For <code>wttjs.pl</code>: Perl5 or later, <a
256			href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
257			and the <code>JSON</code> module. See <a
258			href="bin/wttjs.html#dependency">the document</a> for details.
259
260			<p>For the test result script: CGI-enabled HTTP server, Perl5 or
261			later, and <code>Message::CGI::HTTP</code> and
262			<code>Message::DOM::DOMImplementation</code> modules from the <a
263			href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
264			package.
265
266			</div>
267
268			<div class=section id=example>
269			<h2>Examples</h2>
270
271			<ul>
272
273			<li><a
274			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
275			DOM Core WebIDL Conformance</a> (<a
276			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
277			<a
278			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
279			— <a
280			href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
281			Table</a>
282
283			<li><a
284			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
285			Level 2 WebIDL Conformance</a> (<a
286			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
287			<a
288			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
289			— <a
290			href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
291			Table</a>
292
293			<li><a
294			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
295			WebIDL Conformance</a> (<a
296			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
297			<a
298			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
299			<em>(incomplete)</em>) — <a
300			href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
301			Table</a>
302
303			</ul>
304
305			</div>
306
307			<div class=section id=author>
308			<h2>Author</h2>
309
310			<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.
311
312			</div>
313
314			<div class=section id=license>
315			<h2>License</h2>
316
317			<p>License of the <em>scripts and documentations</em> (not including
318			generated files) is as follows:
319
320			<blockquote>
321			<p>Copyright 2008 <a
322			href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
323			class=mail><<a
324			href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>
325
326			<p>This is free software; you can redistribute it and/or modify it
327			under the same terms as Perl itself.
328			</blockquote>
329
330			<p>Note that you may distribute files you generate by these scripts in
331			any license as you like.
332
333			</div>