| 1 |
<!DOCTYPE HTML> |
| 2 |
<html lang=en> |
| 3 |
<title>wttjs — WebIDL ECMAScript Binding Test Suite Generator</title> |
| 4 |
<link rel=stylesheet href="http://suika.fam.cx/www/style/html/xhtml"> |
| 5 |
<link rel=license href="#license"> |
| 6 |
|
| 7 |
<h1>wttjs — WebIDL ECMAScript Binding Test Suite Generator</h1> |
| 8 |
|
| 9 |
<div class=section id=intro> |
| 10 |
<h2>What is this?</h2> |
| 11 |
|
| 12 |
<p><dfn>wttjs</dfn> is a tool to generate automated test suite from an |
| 13 |
IDL fragment conforming to the WebIDL specification. It generates |
| 14 |
test cases to check whether the DOM ECMAScript binding of a Web |
| 15 |
browser conforms to the <a href="#standards">Web IDL specification</a> |
| 16 |
and the given IDL fragment, as well as a test harness for the |
| 17 |
automated testing against the generated test cases. |
| 18 |
|
| 19 |
<p>In addition, a test result submission script is available. Test |
| 20 |
results can be submitted from the test harness. The browser support |
| 21 |
matrix for the test suite is automatically generated from the |
| 22 |
submitted results. |
| 23 |
|
| 24 |
</div> |
| 25 |
|
| 26 |
<div class=section id=usage> |
| 27 |
<h2>Usage</h2> |
| 28 |
|
| 29 |
<ol> |
| 30 |
|
| 31 |
<li id=prepare-idl> |
| 32 |
|
| 33 |
<p>First, prepare an IDL fragment that represents interfaces and |
| 34 |
exceptions that should be implemented by Web browsers. The IDL |
| 35 |
fragment must conform to the <a href="#standards">Web IDL |
| 36 |
specification</a>. |
| 37 |
|
| 38 |
<p>Ensure that the IDL fragment is conforming. <em>A broken input |
| 39 |
might generate broken test cases.</em> Use conformance checker such as |
| 40 |
<a href="http://suika.fam.cx/gate/2007/html/cc/">WebHACC</a> to detect |
| 41 |
any possible error. |
| 42 |
|
| 43 |
<li id=prepare-json> |
| 44 |
|
| 45 |
<p>Prepare a JSON file that contains supplemental information on |
| 46 |
interfaces and exceptions to be tested. The JSON file can contain two |
| 47 |
kinds of information: <a href="#instantiate">instantiation |
| 48 |
instructions</a> and <a href="#crash">crash data</a>. |
| 49 |
|
| 50 |
<p><dfn id=instantiate>Instantiation instructions</dfn>: Since an IDL |
| 51 |
fragment does not describe possible ways to instantiate interfaces or |
| 52 |
exceptions defined in the fragment, it is necessary to tell the script |
| 53 |
how to create host objects and host exception objects implementing |
| 54 |
those interfaces and exceptions. |
| 55 |
|
| 56 |
<p>The JSON file can contain instantiation instruction data as follows: |
| 57 |
<pre class="json syntax code"><code> |
| 58 |
{ |
| 59 |
"<var>InterfaceName</var>": [ |
| 60 |
{ |
| 61 |
"id": "<var>create-object-1</var>", |
| 62 |
"code": "<var>code.to.createObject (1);</var>" |
| 63 |
} |
| 64 |
<var>, ...</var> |
| 65 |
] |
| 66 |
<var>, ...</var> |
| 67 |
} |
| 68 |
</code></pre> |
| 69 |
|
| 70 |
<p>... where <var>InterfaceName</var> is the name of the interface or |
| 71 |
exception as used in the ECMAScript binding, |
| 72 |
<var>create-object-1</var> is the identifier that distinguish |
| 73 |
different ways to create an object, and <var>code.to.createObject |
| 74 |
(1);</var> is a right-hand part of ECMAScript assignment statement to |
| 75 |
create an object that implements the interface or exception. Note |
| 76 |
that the identifier (i.e. <var>create-object-1</var>) must be unique |
| 77 |
for an interface or exception. It must also be unique globally, |
| 78 |
except when the code (i.e. <var>code.to.createObject (1);</var>) is |
| 79 |
not different each other. |
| 80 |
|
| 81 |
<p>For example, the JSON file for an IDL fragment that contains the |
| 82 |
definition for the <code>NodeList</code> interface would contain: |
| 83 |
|
| 84 |
<pre class="example json code"><code> |
| 85 |
{ |
| 86 |
"NodeList": [ |
| 87 |
{ |
| 88 |
"id": "child-nodes", |
| 89 |
"code": "document.createElement ('p').childNodes;", |
| 90 |
}, |
| 91 |
{ |
| 92 |
"id": "get-elements", |
| 93 |
"code": "document.getElementsByTagName ('*');" |
| 94 |
} |
| 95 |
] |
| 96 |
} |
| 97 |
</code></pre> |
| 98 |
|
| 99 |
<p>If the instantiation instruction for an interface or exception is |
| 100 |
not provided, then any test case that requires an instance of the |
| 101 |
interface or exception will not be generated. |
| 102 |
|
| 103 |
<p><dfn id=crash>Crash data</dfn>: A test case sometimes crashes a Web |
| 104 |
browser. Since the existence of a crashing test case prevents the use |
| 105 |
of automated test harness, a way to avoid execution of test case that |
| 106 |
is known to crash a certain browser is provided. |
| 107 |
|
| 108 |
The crash data can be put into the JSON file with the |
| 109 |
<code>_crash</code> key as follows: |
| 110 |
|
| 111 |
<pre class="json syntax code"><code> |
| 112 |
{ |
| 113 |
<var>... instantiation instruction ...</var> |
| 114 |
"_crash": { |
| 115 |
"<var>test-case-id</var>": [ |
| 116 |
"<var>UserAgentNameRegularExpression</var>" |
| 117 |
<var>, ...</var> |
| 118 |
] |
| 119 |
<var>, ...</var> |
| 120 |
} |
| 121 |
} |
| 122 |
</code></pre> |
| 123 |
|
| 124 |
<p>... where <var>test-case-id</var> is the identifier of the relevant |
| 125 |
test case (i.e. the file name of the test case, except for the |
| 126 |
<code>.html</code> extension) and |
| 127 |
<var>UserAgentNameRegularExpression</var> is an ECMAScript regular |
| 128 |
expression, which will be evaluated by ECMAScript |
| 129 |
<code>new RegExp (<var>regexp</var>)</code> constructor, representing |
| 130 |
<code>User-Agent</code> strings of Web browsers that are known to |
| 131 |
crash. |
| 132 |
|
| 133 |
<li id=execute-wttjs> |
| 134 |
|
| 135 |
<p>Then, generate the test files by executing the |
| 136 |
<code>wttjs.pl</code> script as follows: |
| 137 |
|
| 138 |
<pre> |
| 139 |
$ <kbd>perl wttjs.pl \ |
| 140 |
--idl-file-name <var>input.idl</var> \ |
| 141 |
--instances-file-name <var>input.json</var> \ |
| 142 |
--test-dir-name <var>output-dir/</var> \ |
| 143 |
--testset-id <var>testname</var></kbd> |
| 144 |
</pre> |
| 145 |
|
| 146 |
<p>... where <var>input.idl</var> is the name of the IDL fragment file |
| 147 |
prepared in <a href="#prepare-idl">Step 1</a>, <var>input.json</var> |
| 148 |
is the name of the JSON file prepared in <a href="#prepare-json">Step |
| 149 |
2</a>, <var>output-dir/</var> is the name of the directory in which a |
| 150 |
directory for the test files is created, and <var>testname</var> is a |
| 151 |
short identifier for the test suite. For more information on |
| 152 |
command-line arguments, see <a href="bin/wttjs.html">the documentation |
| 153 |
for the <code>wttjs.pl</code> script</a>. |
| 154 |
|
| 155 |
<li> |
| 156 |
|
| 157 |
<p>At this time, there should be a set of test files for the IDL |
| 158 |
fragment, in the directory specified as the command-line argument. |
| 159 |
|
| 160 |
<p>Open <code>all.html</code> in the set by a Web browser that is to |
| 161 |
be tested. The test is automatically started and the progress is |
| 162 |
shown in the page until all the tests are done. |
| 163 |
|
| 164 |
<li id=test-done> |
| 165 |
|
| 166 |
<p>Examine the test results. Lists of failed tests and skipped tests |
| 167 |
are shown in the page, as well as the numbers of passed, failed, and |
| 168 |
skipped tests. |
| 169 |
|
| 170 |
<p>The test result can be submitted to the <a href="#results">test |
| 171 |
result form</a> using a submission button that appears in the page. |
| 172 |
|
| 173 |
</ol> |
| 174 |
|
| 175 |
</div> |
| 176 |
|
| 177 |
<div class=section id=results> |
| 178 |
<h2>Test results</h2> |
| 179 |
|
| 180 |
<p>Test results can be submitted so that results of Web browsers can |
| 181 |
be compared. |
| 182 |
|
| 183 |
<p>A result table can be prepared for each test suite. To create a |
| 184 |
table, access <code |
| 185 |
class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/info></code>, |
| 186 |
where <var>testset-id</var> is the identifier of the test suite (the |
| 187 |
one <a href="#execute-wttjs">specified</a> to the |
| 188 |
<code>wttjs.pl</code> script) and fill in the form. |
| 189 |
|
| 190 |
<p>Then, submit test results using <a href="#test-done">the submission |
| 191 |
button in the test harness</a>. |
| 192 |
|
| 193 |
<p>Finally, access <code |
| 194 |
class=uri><http://suika.fam.cx/gate/test-results/list/<var>testset-id</var>/all></code>, |
| 195 |
which lists the result of various browsers. |
| 196 |
|
| 197 |
<p>Note that the source code of the server-side script for the test |
| 198 |
result submission is also <a href="#download">available</a>. |
| 199 |
|
| 200 |
</div> |
| 201 |
|
| 202 |
<div class=section id=standards> |
| 203 |
<h2>Supported standards</h2> |
| 204 |
|
| 205 |
<p>This script supports: |
| 206 |
|
| 207 |
<ul> |
| 208 |
|
| 209 |
<li>ECMA 262 Third Edition. |
| 210 |
|
| 211 |
<li><a |
| 212 |
href="http://dev.w3.org/cvsweb/~checkout~/2006/webapi/WebIDL/Overview.html?rev=1.96&content-type=text/html;%20charset=utf-8">Web |
| 213 |
IDL specification, revision 1.96 (3 September 2008 Editor's Draft)</a>. |
| 214 |
|
| 215 |
</ul> |
| 216 |
|
| 217 |
<p>Note that the <code>wttjs.pl</code> cannot generate tests for all |
| 218 |
WebIDL interface/exception constraints. For example, whether an |
| 219 |
attribute returns a value of IDL type <code>long</code> or not is |
| 220 |
<em>not</em> tested. Likewise, whether an attribute with extended |
| 221 |
attribute <code>[Null=Empty]</code> really coverts <code>null</code> |
| 222 |
to an empty string is <em>not</em> tested, either. Therefore, |
| 223 |
<em>passing all tests generated by the <code>wttjs.pl</code> script |
| 224 |
does not necessarily mean the implementation is conforming</em>. |
| 225 |
|
| 226 |
</div> |
| 227 |
|
| 228 |
<div class=section id=download> |
| 229 |
<h2>Download</h2> |
| 230 |
|
| 231 |
<p>Latest version of scripts are available via the CVS repository: |
| 232 |
|
| 233 |
<ul> |
| 234 |
|
| 235 |
<li><a |
| 236 |
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/"><code>wttjs.pl</code>: |
| 237 |
<code>webroot/www/webidl2tests/</code></a> (<a |
| 238 |
href="http://suika.fam.cx/gate/cvs/webroot/www/webidl2tests/webidl2tests.tar.gz?tarball=1">download |
| 239 |
tarball</a>). |
| 240 |
|
| 241 |
<li><a |
| 242 |
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/">result |
| 243 |
submission: <code>webroot/gate/test-results/</code></a> (<a |
| 244 |
href="http://suika.fam.cx/gate/cvs/webroot/gate/test-results/test-results.tar.gz?tarball=1">download |
| 245 |
tarball</a>). |
| 246 |
|
| 247 |
</ul> |
| 248 |
|
| 249 |
|
| 250 |
</div> |
| 251 |
|
| 252 |
<div class=section id=dependency> |
| 253 |
<h2>Dependency</h2> |
| 254 |
|
| 255 |
<p>For <code>wttjs.pl</code>: Perl5 or later, <a |
| 256 |
href="http://suika.fam.cx/www/markup/html/whatpm/readme">Whatpm</a>, |
| 257 |
and the <code>JSON</code> module. See <a |
| 258 |
href="bin/wttjs.html#dependency">the document</a> for details. |
| 259 |
|
| 260 |
<p>For the test result script: CGI-enabled HTTP server, Perl5 or |
| 261 |
later, and <code>Message::CGI::HTTP</code> and |
| 262 |
<code>Message::DOM::DOMImplementation</code> modules from the <a |
| 263 |
href="http://suika.fam.cx/www/manakai-core/doc/web/">manakai-core</a> |
| 264 |
package. |
| 265 |
|
| 266 |
</div> |
| 267 |
|
| 268 |
<div class=section id=example> |
| 269 |
<h2>Examples</h2> |
| 270 |
|
| 271 |
<ul> |
| 272 |
|
| 273 |
<li><a |
| 274 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/webdomcore-20081008/all.html">Web |
| 275 |
DOM Core WebIDL Conformance</a> (<a |
| 276 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore-20081008.idl">IDL</a>, |
| 277 |
<a |
| 278 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/webdomcore.json">JSON</a>) |
| 279 |
— <a |
| 280 |
href="http://suika.fam.cx/gate/test-results/list/webdomcore-20081008/all">Result |
| 281 |
Table</a> |
| 282 |
|
| 283 |
<li><a |
| 284 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/xhr2-20081006/all.html"><code>XMLHttpRequest</code> |
| 285 |
Level 2 WebIDL Conformance</a> (<a |
| 286 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr2-20081006.idl">IDL</a>, |
| 287 |
<a |
| 288 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/xhr.json">JSON</a>) |
| 289 |
— <a |
| 290 |
href="http://suika.fam.cx/gate/test-results/list/xhr2-20081006/all">Result |
| 291 |
Table</a> |
| 292 |
|
| 293 |
<li><a |
| 294 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/tests/html5-r2321/all.html">HTML5 |
| 295 |
WebIDL Conformance</a> (<a |
| 296 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html5-r2321.idl">IDL</a>, |
| 297 |
<a |
| 298 |
href="http://suika.fam.cx/~wakaba/-temp/test/dom/js-binding/idls/html.json">JSON</a> |
| 299 |
<em>(incomplete)</em>) — <a |
| 300 |
href="http://suika.fam.cx/gate/test-results/list/html5-r2321/all">Result |
| 301 |
Table</a> |
| 302 |
|
| 303 |
</ul> |
| 304 |
|
| 305 |
</div> |
| 306 |
|
| 307 |
<div class=section id=author> |
| 308 |
<h2>Author</h2> |
| 309 |
|
| 310 |
<p><a href="http://suika.fam.cx/~wakaba/who?" rel=author>Wakaba</a>. |
| 311 |
|
| 312 |
</div> |
| 313 |
|
| 314 |
<div class=section id=license> |
| 315 |
<h2>License</h2> |
| 316 |
|
| 317 |
<p>License of the <em>scripts and documentations</em> (not including |
| 318 |
generated files) is as follows: |
| 319 |
|
| 320 |
<blockquote> |
| 321 |
<p>Copyright 2008 <a |
| 322 |
href="http://suika.fam.cx/~wakaba/who?">Wakaba</a> <code |
| 323 |
class=mail><<a |
| 324 |
href="mailto:w@suika.fam.cx">w@suika.fam.cx</a>></code> |
| 325 |
|
| 326 |
<p>This is free software; you can redistribute it and/or modify it |
| 327 |
under the same terms as Perl itself. |
| 328 |
</blockquote> |
| 329 |
|
| 330 |
<p>Note that you may distribute files you generate by these scripts in |
| 331 |
any license as you like. |
| 332 |
|
| 333 |
</div> |
Parent Directory
| 
Revision Log