www/webidl2tests/readme.en.html

<!DOCTYPE HTML>
<html lang=en>
<title>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</title>
<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
<link rel=license href="#license">

<h1>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</h1>

<div class=section id=intro>
<h2>What is this?</h2>

<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
IDL fragment conforming to the WebIDL specification.  It generates
test cases to check whether the DOM ECMAScript binding of a Web
browser conforms to the <a href="#standards">Web IDL specification</a>
and the given IDL fragment, as well as a test harness for the
automated testing against the generated test cases.

<p>In addition, a test result submission script is available.  Test
results can be submitted from the test harness.  The browser support
matrix for the test suite is automatically generated from the
submitted results.

</div>

<div class=section id=usage>
<h2>Usage</h2>

<ol>

<li id=prepare-idl>

<p>First, prepare an IDL fragment that represents interfaces and
exceptions that should be implemented by Web browsers.  The IDL
fragment must conform to the <a href="#standards">Web IDL
specification</a>.

<p>Ensure that the IDL fragment is conforming.  <em>A broken input
might generate broken test cases.</em> Use conformance checker such as
<a href="http://suika.fam.cx/gate/2007/html/cc/">WebHACC</a> to detect
any possible error.

<li id=prepare-json>

<p>Prepare a JSON file that contains supplemental information on
interfaces and exceptions to be tested.  The JSON file can contain two
kinds of information: <a href="#instantiate">instantiation
instructions</a> and <a href="#crash">crash data</a>.

<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
fragment does not describe possible ways to instantiate interfaces or
exceptions defined in the fragment, it is necessary to tell the script
how to create host objects and host exception objects implementing
those interfaces and exceptions.

<p>The JSON file can contain instantiation instruction data as follows:
<pre class="json syntax code"><code>
{
  "<var>InterfaceName</var>": [
    {
      "id": "<var>create-object-1</var>",
      "code": "<var>code.to.createObject (1);</var>"
    }
    <var>, ...</var>
  ]
  <var>, ...</var>
}
</code></pre>

<p>... where <var>InterfaceName</var> is the name of the interface or
exception as used in the ECMAScript binding,
<var>create-object-1</var> is the identifier that distinguish
different ways to create an object, and <var>code.to.createObject
(1);</var> is a right-hand part of ECMAScript assignment statement to
create an object that implements the interface or exception.  Note
that the identifier (i.e. <var>create-object-1</var>) must be unique
for an interface or exception.  It must also be unique globally,
except when the code (i.e. <var>code.to.createObject (1);</var>) is
not different each other.

<p>For example, the JSON file for an IDL fragment that contains the
definition for the <code>NodeList</code> interface would contain:

<pre class="example json code"><code>
{
  "NodeList": [
    {
      "id": "child-nodes",
      "code": "document.createElement ('p').childNodes;",
    },
    {
      "id": "get-elements",
      "code": "document.getElementsByTagName ('*');"
    }
  ]
}
</code></pre>

<p>If the instantiation instruction for an interface or exception is
not provided, then any test case that requires an instance of the
interface or exception will not be generated.

<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
browser.  Since the existence of a crashing test case prevents the use
of automated test harness, a way to avoid execution of test case that
is known to crash a certain browser is provided.

The crash data can be put into the JSON file with the
<code>_crash</code> key as follows:

<pre class="json syntax code"><code>
{
  <var>... instantiation instruction ...</var>
  "_crash": {
    "<var>test-case-id</var>": [
      "<var>UserAgentNameRegularExpression</var>"
      <var>, ...</var>
    ]
    <var>, ...</var>
  }
}
</code></pre>

<p>... where <var>test-case-id</var> is the identifier of the relevant
test case (i.e. the file name of the test case, except for the
<code>.html</code> extension) and
<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
expression, which will be evaluated by ECMAScript 
<code>new RegExp (<var>regexp</var>)</code> constructor, representing
<code>User-Agent</code> strings of Web browsers that are known to
crash.

<li id=execute-wttjs>

<p>Then, generate the test files by executing the
<code>wttjs.pl</code> script as follows:

<pre>
  $ <kbd>perl wttjs.pl \
        --idl-file-name <var>input.idl</var> \
        --instances-file-name <var>input.json</var> \
        --test-dir-name <var>output-dir/</var> \
        --testset-id <var>testname</var></kbd>
</pre>

<p>... where <var>input.idl</var> is the name of the IDL fragment file
prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
is the name of the JSON file prepared in <a href="#prepare-json">Step
2</a>, <var>output-dir/</var> is the name of the directory in which a
directory for the test files is created, and <var>testname</var> is a
short identifier for the test suite.  For more information on
command-line arguments, see <a href="bin/wttjs.html">the documentation
for the <code>wttjs.pl</code> script</a>.

<li>

<p>At this time, there should be a set of test files for the IDL
fragment, in the directory specified as the command-line argument.

<p>Open <code>all.html</code> in the set by a Web browser that is to
be tested.  The test is automatically started and the progress is
shown in the page until all the tests are done.

<li id=test-done>

<p>Examine the test results.  Lists of failed tests and skipped tests
are shown in the page, as well as the numbers of passed, failed, and
skipped tests.

<p>The test result can be submitted to the <a href="#results">test
result form</a> using a submission button that appears in the page.

</ol>

</div>

<div class=section id=results>
<h2>Test results</h2>

<p>Test results can be submitted so that results of Web browsers can
be compared.

<p>A result table can be prepared for each test suite.  To create a
table, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
where <var>testset-id</var> is the identifier of the test suite (the
one <a href="#execute-wttjs">specified</a> to the
<code>wttjs.pl</code> script) and fill in the form.

<p>Then, submit test results using <a href="#test-done">the submission
button in the test harness</a>.

<p>Finally, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
which lists the result of various browsers.

<p>Note that the source code of the server-side script for the test
result submission is also <a href="#download">available</a>.

</div>

<div class=section id=standards>
<h2>Supported standards</h2>

<p>This script supports:

<ul>

<li>ECMA 262 Third Edition.

<li><a
href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&amp;content-type=text/html;%20charset=utf-8">Web
IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.

</ul>

<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
WebIDL interface/exception constraints.  For example, whether an
attribute returns a value of IDL type <code>long</code> or not is
<em>not</em> tested.  Likewise, whether an attribute with extended
attribute <code>[Null=Empty]</code> really coverts <code>null</code>
to an empty string is <em>not</em> tested, either.  Therefore,
<em>passing all tests generated by the <code>wttjs.pl</code> script
does not necessarily mean the implementation is conforming</em>.

</div>

<div class=section id=download>
<h2>Download</h2>

<p>Latest version of scripts are available via the CVS repository: 

<ul>

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
<code>webroot/www/webidl2tests/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
tarball</a>).

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
submission: <code>webroot/gate/test-results/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
tarball</a>).

</ul>


</div>

<div class=section id=dependency>
<h2>Dependency</h2>

<p>For <code>wttjs.pl</code>: Perl5 or later, <a
href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
and the <code>JSON</code> module.  See <a
href="bin/wttjs.html#dependency">the document</a> for details.

<p>For the test result script: CGI-enabled HTTP server, Perl5 or
later, and <code>Message::CGI::HTTP</code> and
<code>Message::DOM::DOMImplementation</code> modules from the <a
href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
package.

</div>

<div class=section id=example>
<h2>Examples</h2>

<ul>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
DOM Core WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
Level 2 WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
<em>(incomplete)</em>) &mdash; <a
href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
Table</a>

</ul>

</div>

<div class=section id=author>
<h2>Author</h2>

<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.

</div>

<div class=section id=license>
<h2>License</h2>

<p>License of the <em>scripts and documentations</em> (not including
generated files) is as follows:

<blockquote>
<p>Copyright 2008 <a
href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
class=mail>&lt;<a
href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>

<p>This is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
</blockquote>

<p>Note that you may distribute files you generate by these scripts in
any license as you like.

</div>
1	<!DOCTYPE HTML>
2	<html lang=en>
3	<title>wttjs — WebIDL ECMAScript Binding Test Suite Generator</title>
4	<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
5	<link rel=license href="#license">
6
7	<h1>wttjs — WebIDL ECMAScript Binding Test Suite Generator</h1>
8
9	<div class=section id=intro>
10	<h2>What is this?</h2>
11
12	<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
13	IDL fragment conforming to the WebIDL specification. It generates
14	test cases to check whether the DOM ECMAScript binding of a Web
15	browser conforms to the <a href="#standards">Web IDL specification</a>
16	and the given IDL fragment, as well as a test harness for the
17	automated testing against the generated test cases.
18
19	<p>In addition, a test result submission script is available. Test
20	results can be submitted from the test harness. The browser support
21	matrix for the test suite is automatically generated from the
22	submitted results.
23
24	</div>
25
26	<div class=section id=usage>
27	<h2>Usage</h2>
28
29	<ol>
30
31	<li id=prepare-idl>
32
33	<p>First, prepare an IDL fragment that represents interfaces and
34	exceptions that should be implemented by Web browsers. The IDL
35	fragment must conform to the <a href="#standards">Web IDL
36	specification</a>.
37
38	<p>Ensure that the IDL fragment is conforming. <em>A broken input
39	might generate broken test cases.</em> Use conformance checker such as
40	<a href="http://suika.fam.cx/gate/2007/html/cc/">WebHACC</a> to detect
41	any possible error.
42
43	<li id=prepare-json>
44
45	<p>Prepare a JSON file that contains supplemental information on
46	interfaces and exceptions to be tested. The JSON file can contain two
47	kinds of information: <a href="#instantiate">instantiation
48	instructions</a> and <a href="#crash">crash data</a>.
49
50	<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
51	fragment does not describe possible ways to instantiate interfaces or
52	exceptions defined in the fragment, it is necessary to tell the script
53	how to create host objects and host exception objects implementing
54	those interfaces and exceptions.
55
56	<p>The JSON file can contain instantiation instruction data as follows:
57	<pre class="json syntax code"><code>
58	{
59	"<var>InterfaceName</var>": [
60	{
61	"id": "<var>create-object-1</var>",
62	"code": "<var>code.to.createObject (1);</var>"
63	}
64	<var>, ...</var>
65	]
66	<var>, ...</var>
67	}
68	</code></pre>
69
70	<p>... where <var>InterfaceName</var> is the name of the interface or
71	exception as used in the ECMAScript binding,
72	<var>create-object-1</var> is the identifier that distinguish
73	different ways to create an object, and <var>code.to.createObject
74	(1);</var> is a right-hand part of ECMAScript assignment statement to
75	create an object that implements the interface or exception. Note
76	that the identifier (i.e. <var>create-object-1</var>) must be unique
77	for an interface or exception. It must also be unique globally,
78	except when the code (i.e. <var>code.to.createObject (1);</var>) is
79	not different each other.
80
81	<p>For example, the JSON file for an IDL fragment that contains the
82	definition for the <code>NodeList</code> interface would contain:
83
84	<pre class="example json code"><code>
85	{
86	"NodeList": [
87	{
88	"id": "child-nodes",
89	"code": "document.createElement ('p').childNodes;",
90	},
91	{
92	"id": "get-elements",
93	"code": "document.getElementsByTagName ('*');"
94	}
95	]
96	}
97	</code></pre>
98
99	<p>If the instantiation instruction for an interface or exception is
100	not provided, then any test case that requires an instance of the
101	interface or exception will not be generated.
102
103	<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
104	browser. Since the existence of a crashing test case prevents the use
105	of automated test harness, a way to avoid execution of test case that
106	is known to crash a certain browser is provided.
107
108	The crash data can be put into the JSON file with the
109	<code>_crash</code> key as follows:
110
111	<pre class="json syntax code"><code>
112	{
113	<var>... instantiation instruction ...</var>
114	"_crash": {
115	"<var>test-case-id</var>": [
116	"<var>UserAgentNameRegularExpression</var>"
117	<var>, ...</var>
118	]
119	<var>, ...</var>
120	}
121	}
122	</code></pre>
123
124	<p>... where <var>test-case-id</var> is the identifier of the relevant
125	test case (i.e. the file name of the test case, except for the
126	<code>.html</code> extension) and
127	<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
128	expression, which will be evaluated by ECMAScript
129	<code>new RegExp (<var>regexp</var>)</code> constructor, representing
130	<code>User-Agent</code> strings of Web browsers that are known to
131	crash.
132
133	<li id=execute-wttjs>
134
135	<p>Then, generate the test files by executing the
136	<code>wttjs.pl</code> script as follows:
137
138	<pre>
139	$ <kbd>perl wttjs.pl \
140	--idl-file-name <var>input.idl</var> \
141	--instances-file-name <var>input.json</var> \
142	--test-dir-name <var>output-dir/</var> \
143	--testset-id <var>testname</var></kbd>
144	</pre>
145
146	<p>... where <var>input.idl</var> is the name of the IDL fragment file
147	prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
148	is the name of the JSON file prepared in <a href="#prepare-json">Step
149	2</a>, <var>output-dir/</var> is the name of the directory in which a
150	directory for the test files is created, and <var>testname</var> is a
151	short identifier for the test suite. For more information on
152	command-line arguments, see <a href="bin/wttjs.html">the documentation
153	for the <code>wttjs.pl</code> script</a>.
154
155	<li>
156
157	<p>At this time, there should be a set of test files for the IDL
158	fragment, in the directory specified as the command-line argument.
159
160	<p>Open <code>all.html</code> in the set by a Web browser that is to
161	be tested. The test is automatically started and the progress is
162	shown in the page until all the tests are done.
163
164	<li id=test-done>
165
166	<p>Examine the test results. Lists of failed tests and skipped tests
167	are shown in the page, as well as the numbers of passed, failed, and
168	skipped tests.
169
170	<p>The test result can be submitted to the <a href="#results">test
171	result form</a> using a submission button that appears in the page.
172
173	</ol>
174
175	</div>
176
177	<div class=section id=results>
178	<h2>Test results</h2>
179
180	<p>Test results can be submitted so that results of Web browsers can
181	be compared.
182
183	<p>A result table can be prepared for each test suite. To create a
184	table, access <code
185	class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
186	where <var>testset-id</var> is the identifier of the test suite (the
187	one <a href="#execute-wttjs">specified</a> to the
188	<code>wttjs.pl</code> script) and fill in the form.
189
190	<p>Then, submit test results using <a href="#test-done">the submission
191	button in the test harness</a>.
192
193	<p>Finally, access <code
194	class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
195	which lists the result of various browsers.
196
197	<p>Note that the source code of the server-side script for the test
198	result submission is also <a href="#download">available</a>.
199
200	</div>
201
202	<div class=section id=standards>
203	<h2>Supported standards</h2>
204
205	<p>This script supports:
206
207	<ul>
208
209	<li>ECMA 262 Third Edition.
210
211	<li><a
212	href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&content-type=text/html;%20charset=utf-8">Web
213	IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.
214
215	</ul>
216
217	<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
218	WebIDL interface/exception constraints. For example, whether an
219	attribute returns a value of IDL type <code>long</code> or not is
220	<em>not</em> tested. Likewise, whether an attribute with extended
221	attribute <code>[Null=Empty]</code> really coverts <code>null</code>
222	to an empty string is <em>not</em> tested, either. Therefore,
223	<em>passing all tests generated by the <code>wttjs.pl</code> script
224	does not necessarily mean the implementation is conforming</em>.
225
226	</div>
227
228	<div class=section id=download>
229	<h2>Download</h2>
230
231	<p>Latest version of scripts are available via the CVS repository:
232
233	<ul>
234
235	<li><a
236	href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
237	<code>webroot/www/webidl2tests/</code></a> (<a
238	href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
239	tarball</a>).
240
241	<li><a
242	href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
243	submission: <code>webroot/gate/test-results/</code></a> (<a
244	href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
245	tarball</a>).
246
247	</ul>
248
249
250	</div>
251
252	<div class=section id=dependency>
253	<h2>Dependency</h2>
254
255	<p>For <code>wttjs.pl</code>: Perl5 or later, <a
256	href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
257	and the <code>JSON</code> module. See <a
258	href="bin/wttjs.html#dependency">the document</a> for details.
259
260	<p>For the test result script: CGI-enabled HTTP server, Perl5 or
261	later, and <code>Message::CGI::HTTP</code> and
262	<code>Message::DOM::DOMImplementation</code> modules from the <a
263	href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
264	package.
265
266	</div>
267
268	<div class=section id=example>
269	<h2>Examples</h2>
270
271	<ul>
272
273	<li><a
274	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
275	DOM Core WebIDL Conformance</a> (<a
276	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
277	<a
278	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
279	— <a
280	href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
281	Table</a>
282
283	<li><a
284	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
285	Level 2 WebIDL Conformance</a> (<a
286	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
287	<a
288	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
289	— <a
290	href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
291	Table</a>
292
293	<li><a
294	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
295	WebIDL Conformance</a> (<a
296	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
297	<a
298	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
299	<em>(incomplete)</em>) — <a
300	href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
301	Table</a>
302
303	</ul>
304
305	</div>
306
307	<div class=section id=author>
308	<h2>Author</h2>
309
310	<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.
311
312	</div>
313
314	<div class=section id=license>
315	<h2>License</h2>
316
317	<p>License of the <em>scripts and documentations</em> (not including
318	generated files) is as follows:
319
320	<blockquote>
321	<p>Copyright 2008 <a
322	href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
323	class=mail><<a
324	href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>
325
326	<p>This is free software; you can redistribute it and/or modify it
327	under the same terms as Perl itself.
328	</blockquote>
329
330	<p>Note that you may distribute files you generate by these scripts in
331	any license as you like.
332
333	</div>