www/webidl2tests/readme.en.html

<!DOCTYPE HTML>
<html lang=en>
<title>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</title>
<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
<link rel=license href="#license">

<h1>wttjs &mdash; WebIDL ECMAScript Binding Test Suite Generator</h1>

<div class=section id=intro>
<h2>What is this?</h2>

<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
IDL fragment conforming to the WebIDL specification.  It generates
test cases to check whether the DOM ECMAScript binding of a Web
browser conforms to the <a href="#standards">Web IDL specification</a>
and the given IDL fragment, as well as a test harness for the
automated testing against the generated test cases.

<p>In addition, a test result submission script is available.  Test
results can be submitted from the test harness.  The browser support
matrix for the test suite is automatically generated from the
submitted results.

</div>

<div class=section id=usage>
<h2>Usage</h2>

<ol>

<li id=prepare-idl>

<p>First, prepare an IDL fragment that represents interfaces and
exceptions that should be implemented by Web browsers.  The IDL
fragment must conform to the <a href="#standards">Web IDL
specification</a>.

<p>Ensure that the IDL fragment is conforming.  <em>A broken input
might generate broken test cases.</em> Use conformance checker such as
<a
href="http://suika.fam.cx/www/markup/html/whatpm/readme">WebHACC</a>
to detect any possible error.

<li id=prepare-json>

<p>Prepare a JSON file that contains supplemental information on
interfaces and exceptions to be tested.  The JSON file can contain two
kinds of information: <a href="#instantiate">instantiation
instructions</a> and <a href="#crash">crash data</a>.

<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
fragment does not describe possible ways to instantiate interfaces or
exceptions defined in the fragment, it is necessary to tell the script
how to create host objects and host exception objects implementing
those interfaces and exceptions.

<p>The JSON file can contain instantiation instruction data as follows:
<pre class="json syntax code"><code>
{
  "<var>InterfaceName</var>": [
    {
      "id": "<var>create-object-1</var>",
      "code": "<var>code.to.createObject (1);</var>"
    }
    <var>, ...</var>
  ]
  <var>, ...</var>
}
</code></pre>

<p>... where <var>InterfaceName</var> is the name of the interface or
exception as used in the ECMAScript binding,
<var>create-object-1</var> is the identifier that distinguish
different ways to create an object, and <var>code.to.createObject
(1);</var> is a right-hand part of ECMAScript assignment statement to
create an object that implements the interface or exception.  Note
that the identifier (i.e. <var>create-object-1</var>) must be unique
for an interface or exception.  It must also be unique globally,
except when the code (i.e. <var>code.to.createObject (1);</var>) is
not different each other.

<p>For example, the JSON file for an IDL fragment that contains the
definition for the <code>NodeList</code> interface would contain:

<pre class="example json code"><code>
{
  "NodeList": [
    {
      "id": "child-nodes",
      "code": "document.createElement ('p').childNodes;",
    },
    {
      "id": "get-elements",
      "code": "document.getElementsByTagName ('*');"
    }
  ]
}
</code></pre>

<p>If the instantiation instruction for an interface or exception is
not provided, then any test case that requires an instance of the
interface or exception will not be generated.

<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
browser.  Since the existence of a crashing test case prevents the use
of automated test harness, a way to avoid execution of test case that
is known to crash a certain browser is provided.

The crash data can be put into the JSON file with the
<code>_crash</code> key as follows:

<pre class="json syntax code"><code>
{
  <var>... instantiation instruction ...</var>
  "_crash": {
    "<var>test-case-id</var>": [
      "<var>UserAgentNameRegularExpression</var>"
      <var>, ...</var>
    ]
    <var>, ...</var>
  }
}
</code></pre>

<p>... where <var>test-case-id</var> is the identifier of the relevant
test case (i.e. the file name of the test case, except for the
<code>.html</code> extension) and
<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
expression, which will be evaluated by ECMAScript 
<code>new RegExp (<var>regexp</var>)</code> constructor, representing
<code>User-Agent</code> strings of Web browsers that are known to
crash.

<li id=execute-wttjs>

<p>Then, generate the test files by executing the
<code>wttjs.pl</code> script as follows:

<pre>
  $ <kbd>perl wttjs.pl \
        --idl-file-name <var>input.idl</var> \
        --instances-file-name <var>input.json</var> \
        --test-dir-name <var>output-dir/</var> \
        --testset-id <var>testname</var></kbd>
</pre>

<p>... where <var>input.idl</var> is the name of the IDL fragment file
prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
is the name of the JSON file prepared in <a href="#prepare-json">Step
2</a>, <var>output-dir/</var> is the name of the directory in which a
directory for the test files is created, and <var>testname</var> is a
short identifier for the test suite.  For more information on
command-line arguments, see <a href="bin/wttjs.html">the documentation
for the <code>wttjs.pl</code> script</a>.

<li>

<p>At this time, there should be a set of test files for the IDL
fragment, in the directory specified as the command-line argument.

<p>Open <code>all.html</code> in the set by a Web browser that is to
be tested.  The test is automatically started and the progress is
shown in the page until all the tests are done.

<li id=test-done>

<p>Examine the test results.  Lists of failed tests and skipped tests
are shown in the page, as well as the numbers of passed, failed, and
skipped tests.

<p>The test result can be submitted to the <a href="#results">test
result form</a> using a submission button that appears in the page.

</ol>

</div>

<div class=section id=results>
<h2>Test results</h2>

<p>Test results can be submitted so that results of Web browsers can
be compared.

<p>A result table can be prepared for each test suite.  To create a
table, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
where <var>testset-id</var> is the identifier of the test suite (the
one <a href="#execute-wttjs">specified</a> to the
<code>wttjs.pl</code> script) and fill in the form.

<p>Then, submit test results using <a href="#test-done">the submission
button in the test harness</a>.

<p>Finally, access <code
class=uri>&lt;http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
which lists the result of various browsers.

<p>Note that the source code of the server-side script for the test
result submission is also <a href="#download">available</a>.

</div>

<div class=section id=standards>
<h2>Supported standards</h2>

<p>This script supports:

<ul>

<li>ECMA 262 Third Edition.

<li><a
href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&amp;content-type=text/html;%20charset=utf-8">Web
IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.

</ul>

<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
WebIDL interface/exception constraints.  For example, whether an
attribute returns a value of IDL type <code>long</code> or not is
<em>not</em> tested.  Likewise, whether an attribute with extended
attribute <code>[Null=Empty]</code> really coverts <code>null</code>
to an empty string is <em>not</em> tested, either.  Therefore,
<em>passing all tests generated by the <code>wttjs.pl</code> script
does not necessarily mean the implementation is conforming</em>.

</div>

<div class=section id=download>
<h2>Download</h2>

<p>Latest version of scripts are available via the CVS repository: 

<ul>

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
<code>webroot/www/webidl2tests/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
tarball</a>).

<li><a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
submission: <code>webroot/gate/test-results/</code></a> (<a
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
tarball</a>).

</ul>


</div>

<div class=section id=dependency>
<h2>Dependency</h2>

<p>For <code>wttjs.pl</code>: Perl5 or later, <a
href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
and the <code>JSON</code> module.  See <a
href="bin/wttjs.html#dependency">the document</a> for details.

<p>For the test result script: CGI-enabled HTTP server, Perl5 or
later, and <code>Message::CGI::HTTP</code> and
<code>Message::DOM::DOMImplementation</code> modules from the <a
href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
package.

</div>

<div class=section id=example>
<h2>Examples</h2>

<ul>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
DOM Core WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
Level 2 WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
&mdash; <a
href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
Table</a>

<li><a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
WebIDL Conformance</a> (<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
<a
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
<em>(incomplete)</em>) &mdash; <a
href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
Table</a>

</ul>

</div>

<div class=section id=author>
<h2>Author</h2>

<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.

</div>

<div class=section id=license>
<h2>License</h2>

<p>License of the <em>scripts and documentations</em> (not including
generated files) is as follows:

<blockquote>
<p>Copyright 2008 <a
href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
class=mail>&lt;<a
href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>

<p>This is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
</blockquote>

<p>Note that you may distribute files you generate by these scripts in
any license as you like.

</div>
1	wakaba	1.1	<!DOCTYPE HTML>
2			<html lang=en>
3			<title>wttjs — WebIDL ECMAScript Binding Test Suite Generator</title>
4			<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml">
5			<link rel=license href="#license">
6
7			<h1>wttjs — WebIDL ECMAScript Binding Test Suite Generator</h1>
8
9			<div class=section id=intro>
10			<h2>What is this?</h2>
11
12			<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an
13			IDL fragment conforming to the WebIDL specification. It generates
14			test cases to check whether the DOM ECMAScript binding of a Web
15			browser conforms to the <a href="#standards">Web IDL specification</a>
16			and the given IDL fragment, as well as a test harness for the
17			automated testing against the generated test cases.
18
19			<p>In addition, a test result submission script is available. Test
20			results can be submitted from the test harness. The browser support
21			matrix for the test suite is automatically generated from the
22			submitted results.
23
24			</div>
25
26			<div class=section id=usage>
27			<h2>Usage</h2>
28
29			<ol>
30
31			<li id=prepare-idl>
32
33			<p>First, prepare an IDL fragment that represents interfaces and
34			exceptions that should be implemented by Web browsers. The IDL
35			fragment must conform to the <a href="#standards">Web IDL
36			specification</a>.
37
38			<p>Ensure that the IDL fragment is conforming. <em>A broken input
39			might generate broken test cases.</em> Use conformance checker such as
40			<a
41			href="http://suika.fam.cx/www/markup/html/whatpm/readme">WebHACC</a>
42			to detect any possible error.
43
44			<li id=prepare-json>
45
46			<p>Prepare a JSON file that contains supplemental information on
47			interfaces and exceptions to be tested. The JSON file can contain two
48			kinds of information: <a href="#instantiate">instantiation
49			instructions</a> and <a href="#crash">crash data</a>.
50
51			<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL
52			fragment does not describe possible ways to instantiate interfaces or
53			exceptions defined in the fragment, it is necessary to tell the script
54			how to create host objects and host exception objects implementing
55			those interfaces and exceptions.
56
57			<p>The JSON file can contain instantiation instruction data as follows:
58			<pre class="json syntax code"><code>
59			{
60			"<var>InterfaceName</var>": [
61			{
62			"id": "<var>create-object-1</var>",
63			"code": "<var>code.to.createObject (1);</var>"
64			}
65			<var>, ...</var>
66			]
67			<var>, ...</var>
68			}
69			</code></pre>
70
71			<p>... where <var>InterfaceName</var> is the name of the interface or
72			exception as used in the ECMAScript binding,
73			<var>create-object-1</var> is the identifier that distinguish
74			different ways to create an object, and <var>code.to.createObject
75			(1);</var> is a right-hand part of ECMAScript assignment statement to
76			create an object that implements the interface or exception. Note
77			that the identifier (i.e. <var>create-object-1</var>) must be unique
78			for an interface or exception. It must also be unique globally,
79			except when the code (i.e. <var>code.to.createObject (1);</var>) is
80			not different each other.
81
82			<p>For example, the JSON file for an IDL fragment that contains the
83			definition for the <code>NodeList</code> interface would contain:
84
85			<pre class="example json code"><code>
86			{
87			"NodeList": [
88			{
89			"id": "child-nodes",
90			"code": "document.createElement ('p').childNodes;",
91			},
92			{
93			"id": "get-elements",
94			"code": "document.getElementsByTagName ('*');"
95			}
96			]
97			}
98			</code></pre>
99
100			<p>If the instantiation instruction for an interface or exception is
101			not provided, then any test case that requires an instance of the
102			interface or exception will not be generated.
103
104			<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web
105			browser. Since the existence of a crashing test case prevents the use
106			of automated test harness, a way to avoid execution of test case that
107			is known to crash a certain browser is provided.
108
109			The crash data can be put into the JSON file with the
110			<code>_crash</code> key as follows:
111
112			<pre class="json syntax code"><code>
113			{
114			<var>... instantiation instruction ...</var>
115			"_crash": {
116			"<var>test-case-id</var>": [
117			"<var>UserAgentNameRegularExpression</var>"
118			<var>, ...</var>
119			]
120			<var>, ...</var>
121			}
122			}
123			</code></pre>
124
125			<p>... where <var>test-case-id</var> is the identifier of the relevant
126			test case (i.e. the file name of the test case, except for the
127			<code>.html</code> extension) and
128			<var>UserAgentNameRegularExpression</var> is an ECMAScript regular
129			expression, which will be evaluated by ECMAScript
130			<code>new RegExp (<var>regexp</var>)</code> constructor, representing
131			<code>User-Agent</code> strings of Web browsers that are known to
132			crash.
133
134			<li id=execute-wttjs>
135
136			<p>Then, generate the test files by executing the
137			<code>wttjs.pl</code> script as follows:
138
139			<pre>
140			$ <kbd>perl wttjs.pl \
141			--idl-file-name <var>input.idl</var> \
142			--instances-file-name <var>input.json</var> \
143			--test-dir-name <var>output-dir/</var> \
144			--testset-id <var>testname</var></kbd>
145			</pre>
146
147			<p>... where <var>input.idl</var> is the name of the IDL fragment file
148			prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var>
149			is the name of the JSON file prepared in <a href="#prepare-json">Step
150			2</a>, <var>output-dir/</var> is the name of the directory in which a
151			directory for the test files is created, and <var>testname</var> is a
152			short identifier for the test suite. For more information on
153			command-line arguments, see <a href="bin/wttjs.html">the documentation
154			for the <code>wttjs.pl</code> script</a>.
155
156			<li>
157
158			<p>At this time, there should be a set of test files for the IDL
159			fragment, in the directory specified as the command-line argument.
160
161			<p>Open <code>all.html</code> in the set by a Web browser that is to
162			be tested. The test is automatically started and the progress is
163			shown in the page until all the tests are done.
164
165			<li id=test-done>
166
167			<p>Examine the test results. Lists of failed tests and skipped tests
168			are shown in the page, as well as the numbers of passed, failed, and
169			skipped tests.
170
171			<p>The test result can be submitted to the <a href="#results">test
172			result form</a> using a submission button that appears in the page.
173
174			</ol>
175
176			</div>
177
178			<div class=section id=results>
179			<h2>Test results</h2>
180
181			<p>Test results can be submitted so that results of Web browsers can
182			be compared.
183
184			<p>A result table can be prepared for each test suite. To create a
185			table, access <code
186			class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>,
187			where <var>testset-id</var> is the identifier of the test suite (the
188			one <a href="#execute-wttjs">specified</a> to the
189			<code>wttjs.pl</code> script) and fill in the form.
190
191			<p>Then, submit test results using <a href="#test-done">the submission
192			button in the test harness</a>.
193
194			<p>Finally, access <code
195			class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>,
196			which lists the result of various browsers.
197
198			<p>Note that the source code of the server-side script for the test
199			result submission is also <a href="#download">available</a>.
200
201			</div>
202
203			<div class=section id=standards>
204			<h2>Supported standards</h2>
205
206			<p>This script supports:
207
208			<ul>
209
210			<li>ECMA 262 Third Edition.
211
212			<li><a
213			href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&content-type=text/html;%20charset=utf-8">Web
214			IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>.
215
216			</ul>
217
218			<p>Note that the <code>wttjs.pl</code> cannot generate tests for all
219			WebIDL interface/exception constraints. For example, whether an
220			attribute returns a value of IDL type <code>long</code> or not is
221			<em>not</em> tested. Likewise, whether an attribute with extended
222			attribute <code>[Null=Empty]</code> really coverts <code>null</code>
223			to an empty string is <em>not</em> tested, either. Therefore,
224			<em>passing all tests generated by the <code>wttjs.pl</code> script
225			does not necessarily mean the implementation is conforming</em>.
226
227			</div>
228
229			<div class=section id=download>
230			<h2>Download</h2>
231
232			<p>Latest version of scripts are available via the CVS repository:
233
234			<ul>
235
236			<li><a
237			href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>:
238			<code>webroot/www/webidl2tests/</code></a> (<a
239			href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download
240			tarball</a>).
241
242			<li><a
243			href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result
244			submission: <code>webroot/gate/test-results/</code></a> (<a
245			href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download
246			tarball</a>).
247
248			</ul>
249
250
251			</div>
252
253			<div class=section id=dependency>
254			<h2>Dependency</h2>
255
256			<p>For <code>wttjs.pl</code>: Perl5 or later, <a
257			href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>,
258			and the <code>JSON</code> module. See <a
259			href="bin/wttjs.html#dependency">the document</a> for details.
260
261			<p>For the test result script: CGI-enabled HTTP server, Perl5 or
262			later, and <code>Message::CGI::HTTP</code> and
263			<code>Message::DOM::DOMImplementation</code> modules from the <a
264			href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a>
265			package.
266
267			</div>
268
269			<div class=section id=example>
270			<h2>Examples</h2>
271
272			<ul>
273
274			<li><a
275			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web
276			DOM Core WebIDL Conformance</a> (<a
277			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>,
278			<a
279			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>)
280			— <a
281			href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result
282			Table</a>
283
284			<li><a
285			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code>
286			Level 2 WebIDL Conformance</a> (<a
287			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>,
288			<a
289			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>)
290			— <a
291			href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result
292			Table</a>
293
294			<li><a
295			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5
296			WebIDL Conformance</a> (<a
297			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>,
298			<a
299			href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a>
300			<em>(incomplete)</em>) — <a
301			href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result
302			Table</a>
303
304			</ul>
305
306			</div>
307
308			<div class=section id=author>
309			<h2>Author</h2>
310
311			<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>.
312
313			</div>
314
315			<div class=section id=license>
316			<h2>License</h2>
317
318			<p>License of the <em>scripts and documentations</em> (not including
319			generated files) is as follows:
320
321			<blockquote>
322			<p>Copyright 2008 <a
323			href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code
324			class=mail><<a
325			href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code>
326
327			<p>This is free software; you can redistribute it and/or modify it
328			under the same terms as Perl itself.
329			</blockquote>
330
331			<p>Note that you may distribute files you generate by these scripts in
332			any license as you like.
333
334			</div>