www/webidl2tests/readme.en.html

<!DOCTYPE HTML>
<html lang=en>
<title>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</title>
<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
<link rel=license href="#license">

<h1>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</h1>

<div class=section id=intro>
<h2>What is this?</h2>

<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
IDL fragment conforming to the WebIDL specification.  It generates
test cases to check whether the DOM ECMAScript binding of a Web
browser conforms to the <a href="#standards">Web IDL specification</a>
and the given IDL fragment, as well as a test harness for the
automated testing against the generated test cases.

<p>In addition, a test result submission script is available.  Test
results can be submitted from the test harness.  The browser support
matrix for the test suite is automatically generated from the
submitted results.

</div>

<div class=section id=usage>
<h2>Usage</h2>

<ol>

<li id=prepare-idl>

<p>First, prepare an IDL fragment that represents interfaces and
exceptions that should be implemented by Web browsers.  The IDL
fragment must conform to the <a href="#standards">Web IDL
specification</a>.

<p>Ensure that the IDL fragment is conforming.  <em>A broken input
might generate broken test cases.</em> Use conformance checker such as
<a
href="http://suika.fam.cx/www/markup/html/whatpm/readme">WebHACC</a>
to detect any possible error.

<li id=prepare-json>

<p>Prepare a JSON file that contains supplemental information on
interfaces and exceptions to be tested.  The JSON file can contain two
kinds of information: <a href="#instantiate">instantiation
instructions</a> and <a href="#crash">crash data</a>.

<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
fragment does not describe possible ways to instantiate interfaces or
exceptions defined in the fragment, it is necessary to tell the script
how to create host objects and host exception objects implementing
those interfaces and exceptions.

<p>The JSON file can contain instantiation instruction data as follows:
<pre class="json syntax code"><code>
{
  "<var>InterfaceName</var>": [
    {
      "id": "<var>create-object-1</var>",
      "code": "<var>code.to.createObject (1);</var>"
    }
    <var>, ...</var>
  ]
  <var>, ...</var>
}
</code></pre>

<p>... where <var>InterfaceName</var> is the name of the interface or
exception as used in the ECMAScript binding,
<var>create-object-1</var> is the identifier that distinguish
different ways to create an object, and <var>code.to.createObject
(1);</var> is a right-hand part of ECMAScript assignment statement to
create an object that implements the interface or exception.  Note
that the identifier (i.e. <var>create-object-1</var>) must be unique
for an interface or exception.  It must also be unique globally,
except when the code (i.e. <var>code.to.createObject (1);</var>) is
not different each other.

<p>For example, the JSON file for an IDL fragment that contains the
definition for the <code>NodeList</code> interface would contain:

<pre class="example json code"><code>
{
  "NodeList": [
    {
      "id": "child-nodes",
      "code": "document.createElement ('p').childNodes;",
    },
    {
      "id": "get-elements",
      "code": "document.getElementsByTagName ('*');"
    }
  ]
}
</code></pre>

<p>If the instantiation instruction for an interface or exception is
not provided, then any test case that requires an instance of the
interface or exception will not be generated.

<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
browser.  Since the existence of a crashing test case prevents the use
of automated test harness, a way to avoid execution of test case that
is known to crash a certain browser is provided.

The crash data can be put into the JSON file with the
<code>_crash</code> key as follows:

<pre class="json syntax code"><code>
{
  <var>... instantiation instruction ...</var>
  "_crash": {
    "<var>test-case-id</var>": [
      "<var>UserAgentNameRegularExpression</var>"
      <var>, ...</var>
    ]
    <var>, ...</var>
  }
}
</code></pre>

<p>... where <var>test-case-id</var> is the identifier of the relevant
test case (i.e. the file name of the test case, except for the
<code>.html</code> extension) and
<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
expression, which will be evaluated by ECMAScript 
<code>new RegExp (<var>regexp</var>)</code> constructor, representing
<code>User-Agent</code> strings of Web browsers that are known to
crash.

<li id=execute-wttjs>

<p>Then, generate the test files by executing the
<code>wttjs.pl</code> script as follows:

<pre>
  $ <kbd>perl wttjs.pl \
        --idl-file-name <var>input.idl</var> \
        --instances-file-name <var>input.json</var> \
        --test-dir-name <var>output-dir/</var> \
        --testset-id <var>testname</var></kbd>
</pre>

<p>... where <var>input.idl</var> is the name of the IDL fragment file
prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
is the name of the JSON file prepared in <a href="#prepare-json">Step
2</a>, <var>output-dir/</var> is the name of the directory in which a
directory for the test files is created, and <var>testname</var> is a
short identifier for the test suite.  For more information on
command-line arguments, see <a href="bin/wttjs.html">the documentation
for the <code>wttjs.pl</code> script</a>.

<li>

<p>At this time, there should be a set of test files for the IDL
fragment, in the directory specified as the command-line argument.

<p>Open <code>all.html</code> in the set by a Web browser that is to
be tested.  The test is automatically started and the progress is
shown in the page until all the tests are done.

<li id=test-done>

<p>Examine the test results.  Lists of failed tests and skipped tests
are shown in the page, as well as the numbers of passed, failed, and
skipped tests.

<p>The test result can be submitted to the <a href="#results">test
result form</a> using a submission button that appears in the page.

</ol>

</div>

<div class=section id=results>
<h2>Test results</h2>

<p>Test results can be submitted so that results of Web browsers can
be compared.

<p>A result table can be prepared for each test suite.  To create a
table, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
where <var>testset-id</var> is the identifier of the test suite (the
one <a href="#execute-wttjs">specified</a> to the
<code>wttjs.pl</code> script) and fill in the form.

<p>Then, submit test results using <a href="#test-done">the submission
button in the test harness</a>.

<p>Finally, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
which lists the result of various browsers.

<p>Note that the source code of the server-side script for the test
result submission is also <a href="#download">available</a>.

</div>

<div class=section id=standards>
<h2>Supported standards</h2>

<p>This script supports:

<ul>

<li>ECMA 262 Third Edition.

<li><a
href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&amp;content-type=text/html;%20charset=utf-8">Web
IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.

</ul>

<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
WebIDL interface/exception constraints.  For example, whether an
attribute returns a value of IDL type <code>long</code> or not is
<em>not</em> tested.  Likewise, whether an attribute with extended
attribute <code>[Null=Empty]</code> really coverts <code>null</code>
to an empty string is <em>not</em> tested, either.  Therefore,
<em>passing all tests generated by the <code>wttjs.pl</code> script
does not necessarily mean the implementation is conforming</em>.

</div>

<div class=section id=download>
<h2>Download</h2>

<p>Latest version of scripts are available via the CVS repository: 

<ul>

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
<code>webroot/www/webidl2tests/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
tarball</a>).

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
submission: <code>webroot/gate/test-results/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
tarball</a>).

</ul>


</div>

<div class=section id=dependency>
<h2>Dependency</h2>

<p>For <code>wttjs.pl</code>: Perl5 or later, <a
href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
and the <code>JSON</code> module.  See <a
href="bin/wttjs.html#dependency">the document</a> for details.

<p>For the test result script: CGI-enabled HTTP server, Perl5 or
later, and <code>Message::CGI::HTTP</code> and
<code>Message::DOM::DOMImplementation</code> modules from the <a
href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
package.

</div>

<div class=section id=example>
<h2>Examples</h2>

<ul>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
DOM Core WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
Level 2 WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
<em>(incomplete)</em>) &mdash; <a
href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
Table</a>

</ul>

</div>

<div class=section id=author>
<h2>Author</h2>

<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.

</div>

<div class=section id=license>
<h2>License</h2>

<p>License of the <em>scripts and documentations</em> (not including
generated files) is as follows:

<blockquote>
<p>Copyright 2008 <a
href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
class=mail>&lt;<a
href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>

<p>This is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
</blockquote>

<p>Note that you may distribute files you generate by these scripts in
any license as you like.

</div>
1	<!DOCTYPE HTML>
2	<html lang=en>
3	<title>wttjs — WebIDL ECMAScript Binding Test Suite Generator</title>
4	<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
5	<link rel=license href="#license">
6
7	<h1>wttjs — WebIDL ECMAScript Binding Test Suite Generator</h1>
8
9	<div class=section id=intro>
10	<h2>What is this?</h2>
11
12	<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
13	IDL fragment conforming to the WebIDL specification. It generates
14	test cases to check whether the DOM ECMAScript binding of a Web
15	browser conforms to the <a href="#standards">Web IDL specification</a>
16	and the given IDL fragment, as well as a test harness for the
17	automated testing against the generated test cases.
18
19	<p>In addition, a test result submission script is available. Test
20	results can be submitted from the test harness. The browser support
21	matrix for the test suite is automatically generated from the
22	submitted results.
23
24	</div>
25
26	<div class=section id=usage>
27	<h2>Usage</h2>
28
29	<ol>
30
31	<li id=prepare-idl>
32
33	<p>First, prepare an IDL fragment that represents interfaces and
34	exceptions that should be implemented by Web browsers. The IDL
35	fragment must conform to the <a href="#standards">Web IDL
36	specification</a>.
37
38	<p>Ensure that the IDL fragment is conforming. <em>A broken input
39	might generate broken test cases.</em> Use conformance checker such as
40	<a
41	href="http://suika.fam.cx/www/markup/html/whatpm/readme">WebHACC</a>
42	to detect any possible error.
43
44	<li id=prepare-json>
45
46	<p>Prepare a JSON file that contains supplemental information on
47	interfaces and exceptions to be tested. The JSON file can contain two
48	kinds of information: <a href="#instantiate">instantiation
49	instructions</a> and <a href="#crash">crash data</a>.
50
51	<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
52	fragment does not describe possible ways to instantiate interfaces or
53	exceptions defined in the fragment, it is necessary to tell the script
54	how to create host objects and host exception objects implementing
55	those interfaces and exceptions.
56
57	<p>The JSON file can contain instantiation instruction data as follows:
58	<pre class="json syntax code"><code>
59	{
60	"<var>InterfaceName</var>": [
61	{
62	"id": "<var>create-object-1</var>",
63	"code": "<var>code.to.createObject (1);</var>"
64	}
65	<var>, ...</var>
66	]
67	<var>, ...</var>
68	}
69	</code></pre>
70
71	<p>... where <var>InterfaceName</var> is the name of the interface or
72	exception as used in the ECMAScript binding,
73	<var>create-object-1</var> is the identifier that distinguish
74	different ways to create an object, and <var>code.to.createObject
75	(1);</var> is a right-hand part of ECMAScript assignment statement to
76	create an object that implements the interface or exception. Note
77	that the identifier (i.e. <var>create-object-1</var>) must be unique
78	for an interface or exception. It must also be unique globally,
79	except when the code (i.e. <var>code.to.createObject (1);</var>) is
80	not different each other.
81
82	<p>For example, the JSON file for an IDL fragment that contains the
83	definition for the <code>NodeList</code> interface would contain:
84
85	<pre class="example json code"><code>
86	{
87	"NodeList": [
88	{
89	"id": "child-nodes",
90	"code": "document.createElement ('p').childNodes;",
91	},
92	{
93	"id": "get-elements",
94	"code": "document.getElementsByTagName ('*');"
95	}
96	]
97	}
98	</code></pre>
99
100	<p>If the instantiation instruction for an interface or exception is
101	not provided, then any test case that requires an instance of the
102	interface or exception will not be generated.
103
104	<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
105	browser. Since the existence of a crashing test case prevents the use
106	of automated test harness, a way to avoid execution of test case that
107	is known to crash a certain browser is provided.
108
109	The crash data can be put into the JSON file with the
110	<code>_crash</code> key as follows:
111
112	<pre class="json syntax code"><code>
113	{
114	<var>... instantiation instruction ...</var>
115	"_crash": {
116	"<var>test-case-id</var>": [
117	"<var>UserAgentNameRegularExpression</var>"
118	<var>, ...</var>
119	]
120	<var>, ...</var>
121	}
122	}
123	</code></pre>
124
125	<p>... where <var>test-case-id</var> is the identifier of the relevant
126	test case (i.e. the file name of the test case, except for the
127	<code>.html</code> extension) and
128	<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
129	expression, which will be evaluated by ECMAScript
130	<code>new RegExp (<var>regexp</var>)</code> constructor, representing
131	<code>User-Agent</code> strings of Web browsers that are known to
132	crash.
133
134	<li id=execute-wttjs>
135
136	<p>Then, generate the test files by executing the
137	<code>wttjs.pl</code> script as follows:
138
139	<pre>
140	$ <kbd>perl wttjs.pl \
141	--idl-file-name <var>input.idl</var> \
142	--instances-file-name <var>input.json</var> \
143	--test-dir-name <var>output-dir/</var> \
144	--testset-id <var>testname</var></kbd>
145	</pre>
146
147	<p>... where <var>input.idl</var> is the name of the IDL fragment file
148	prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
149	is the name of the JSON file prepared in <a href="#prepare-json">Step
150	2</a>, <var>output-dir/</var> is the name of the directory in which a
151	directory for the test files is created, and <var>testname</var> is a
152	short identifier for the test suite. For more information on
153	command-line arguments, see <a href="bin/wttjs.html">the documentation
154	for the <code>wttjs.pl</code> script</a>.
155
156	<li>
157
158	<p>At this time, there should be a set of test files for the IDL
159	fragment, in the directory specified as the command-line argument.
160
161	<p>Open <code>all.html</code> in the set by a Web browser that is to
162	be tested. The test is automatically started and the progress is
163	shown in the page until all the tests are done.
164
165	<li id=test-done>
166
167	<p>Examine the test results. Lists of failed tests and skipped tests
168	are shown in the page, as well as the numbers of passed, failed, and
169	skipped tests.
170
171	<p>The test result can be submitted to the <a href="#results">test
172	result form</a> using a submission button that appears in the page.
173
174	</ol>
175
176	</div>
177
178	<div class=section id=results>
179	<h2>Test results</h2>
180
181	<p>Test results can be submitted so that results of Web browsers can
182	be compared.
183
184	<p>A result table can be prepared for each test suite. To create a
185	table, access <code
186	class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
187	where <var>testset-id</var> is the identifier of the test suite (the
188	one <a href="#execute-wttjs">specified</a> to the
189	<code>wttjs.pl</code> script) and fill in the form.
190
191	<p>Then, submit test results using <a href="#test-done">the submission
192	button in the test harness</a>.
193
194	<p>Finally, access <code
195	class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
196	which lists the result of various browsers.
197
198	<p>Note that the source code of the server-side script for the test
199	result submission is also <a href="#download">available</a>.
200
201	</div>
202
203	<div class=section id=standards>
204	<h2>Supported standards</h2>
205
206	<p>This script supports:
207
208	<ul>
209
210	<li>ECMA 262 Third Edition.
211
212	<li><a
213	href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&content-type=text/html;%20charset=utf-8">Web
214	IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.
215
216	</ul>
217
218	<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
219	WebIDL interface/exception constraints. For example, whether an
220	attribute returns a value of IDL type <code>long</code> or not is
221	<em>not</em> tested. Likewise, whether an attribute with extended
222	attribute <code>[Null=Empty]</code> really coverts <code>null</code>
223	to an empty string is <em>not</em> tested, either. Therefore,
224	<em>passing all tests generated by the <code>wttjs.pl</code> script
225	does not necessarily mean the implementation is conforming</em>.
226
227	</div>
228
229	<div class=section id=download>
230	<h2>Download</h2>
231
232	<p>Latest version of scripts are available via the CVS repository:
233
234	<ul>
235
236	<li><a
237	href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
238	<code>webroot/www/webidl2tests/</code></a> (<a
239	href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
240	tarball</a>).
241
242	<li><a
243	href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
244	submission: <code>webroot/gate/test-results/</code></a> (<a
245	href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
246	tarball</a>).
247
248	</ul>
249
250
251	</div>
252
253	<div class=section id=dependency>
254	<h2>Dependency</h2>
255
256	<p>For <code>wttjs.pl</code>: Perl5 or later, <a
257	href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
258	and the <code>JSON</code> module. See <a
259	href="bin/wttjs.html#dependency">the document</a> for details.
260
261	<p>For the test result script: CGI-enabled HTTP server, Perl5 or
262	later, and <code>Message::CGI::HTTP</code> and
263	<code>Message::DOM::DOMImplementation</code> modules from the <a
264	href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
265	package.
266
267	</div>
268
269	<div class=section id=example>
270	<h2>Examples</h2>
271
272	<ul>
273
274	<li><a
275	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
276	DOM Core WebIDL Conformance</a> (<a
277	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
278	<a
279	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
280	— <a
281	href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
282	Table</a>
283
284	<li><a
285	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
286	Level 2 WebIDL Conformance</a> (<a
287	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
288	<a
289	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
290	— <a
291	href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
292	Table</a>
293
294	<li><a
295	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
296	WebIDL Conformance</a> (<a
297	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
298	<a
299	href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
300	<em>(incomplete)</em>) — <a
301	href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
302	Table</a>
303
304	</ul>
305
306	</div>
307
308	<div class=section id=author>
309	<h2>Author</h2>
310
311	<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.
312
313	</div>
314
315	<div class=section id=license>
316	<h2>License</h2>
317
318	<p>License of the <em>scripts and documentations</em> (not including
319	generated files) is as follows:
320
321	<blockquote>
322	<p>Copyright 2008 <a
323	href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
324	class=mail><<a
325	href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>
326
327	<p>This is free software; you can redistribute it and/or modify it
328	under the same terms as Perl itself.
329	</blockquote>
330
331	<p>Note that you may distribute files you generate by these scripts in
332	any license as you like.
333
334	</div>