wttjs is a tool to generate automated test suite from an IDL fragment conforming to the WebIDL specification. It generates test cases to check whether the DOM ECMAScript binding of a Web browser conforms to the Web IDL specification and the given IDL fragment, as well as a test harness for the automated testing against the generated test cases.
In addition, a test result submission script is available. Test results can be submitted from the test harness. The browser support matrix for the test suite is automatically generated from the submitted results.
Please note that the WebIDL spec implemented by this tool is too old!
First, prepare an IDL fragment that represents interfaces and exceptions that should be implemented by Web browsers. The IDL fragment must conform to the Web IDL specification.
Ensure that the IDL fragment is conforming. A broken input might generate broken test cases. Use conformance checker such as WebHACC to detect any possible error.
Prepare a JSON file that contains supplemental information on interfaces and exceptions to be tested. The JSON file can contain two kinds of information: instantiation instructions and crash data.
Instantiation instructions: Since an IDL fragment does not describe possible ways to instantiate interfaces or exceptions defined in the fragment, it is necessary to tell the script how to create host objects and host exception objects implementing those interfaces and exceptions.
The JSON file can contain instantiation instruction data as follows:
{
"InterfaceName": [
{
"id": "create-object-1",
"code": "code.to.createObject (1);"
}
, ...
]
, ...
}
... where InterfaceName is the name of the interface or exception as used in the ECMAScript binding, create-object-1 is the identifier that distinguish different ways to create an object, and code.to.createObject (1); is a right-hand part of ECMAScript assignment statement to create an object that implements the interface or exception. Note that the identifier (i.e. create-object-1) must be unique for an interface or exception. It must also be unique globally, except when the code (i.e. code.to.createObject (1);) is not different each other.
For example, the JSON file for an IDL fragment that contains the
definition for the NodeList interface would contain:
{
"NodeList": [
{
"id": "child-nodes",
"code": "document.createElement ('p').childNodes;",
},
{
"id": "get-elements",
"code": "document.getElementsByTagName ('*');"
}
]
}
If the instantiation instruction for an interface or exception is not provided, then any test case that requires an instance of the interface or exception will not be generated.
Crash data: A test case sometimes crashes a Web
browser. Since the existence of a crashing test case prevents the use
of automated test harness, a way to avoid execution of test case that
is known to crash a certain browser is provided.
The crash data can be put into the JSON file with the
_crash key as follows:
{
... instantiation instruction ...
"_crash": {
"test-case-id": [
"UserAgentNameRegularExpression"
, ...
]
, ...
}
}
... where test-case-id is the identifier of the relevant
test case (i.e. the file name of the test case, except for the
.html extension) and
UserAgentNameRegularExpression is an ECMAScript regular
expression, which will be evaluated by ECMAScript
new RegExp (regexp) constructor, representing
User-Agent strings of Web browsers that are known to
crash.
Then, generate the test files by executing the
wttjs.pl script as follows:
$ perl wttjs.pl \
--idl-file-name input.idl \
--instances-file-name input.json \
--test-dir-name output-dir/ \
--testset-id testname
... where input.idl is the name of the IDL fragment file
prepared in Step 1, input.json
is the name of the JSON file prepared in Step
2, output-dir/ is the name of the directory in which a
directory for the test files is created, and testname is a
short identifier for the test suite. For more information on
command-line arguments, see the documentation
for the wttjs.pl script.
At this time, there should be a set of test files for the IDL fragment, in the directory specified as the command-line argument.
Open all.html in the set by a Web browser that is to
be tested. The test is automatically started and the progress is
shown in the page until all the tests are done.
Examine the test results. Lists of failed tests and skipped tests are shown in the page, as well as the numbers of passed, failed, and skipped tests.
The test result can be submitted to the test result form using a submission button that appears in the page.
Test results can be submitted so that results of Web browsers can be compared.
A result table can be prepared for each test suite. To create a
table, access <https://suika.suikawiki.org/gate/test-results/list/testset-id/info>,
where testset-id is the identifier of the test suite (the
one specified to the
wttjs.pl script) and fill in the form.
Then, submit test results using the submission button in the test harness.
Finally, access <https://suika.suikawiki.org/gate/test-results/list/testset-id/all>,
which lists the result of various browsers.
Note that the source code of the server-side script for the test result submission is also available.
This script supports:
Note that the wttjs.pl cannot generate tests for all
WebIDL interface/exception constraints. For example, whether an
attribute returns a value of IDL type long or not is
not tested. Likewise, whether an attribute with extended
attribute [Null=Empty] really coverts null
to an empty string is not tested, either. Therefore,
passing all tests generated by the wttjs.pl script
does not necessarily mean the implementation is conforming.
Latest version of the test generation script,
wttjs.pl, is available in the Git repository (Suika
/ GitHub).
Latest version of the test result submission script is available via the CVS repository:
For wttjs.pl: Perl5 or later, Whatpm,
and the JSON module. See the document for details.
For the test result script: CGI-enabled HTTP server, Perl5 or
later, and Message::CGI::HTTP and
Message::DOM::DOMImplementation modules from the manakai-core
package.
XMLHttpRequest
Level 2 WebIDL Conformance (IDL,
JSON)
— Result
Table
License of the scripts and documentations (not including generated files) is as follows:
Copyright 2008-2011 Wakaba
<wakaba@suikawiki.org>This is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Note that you may distribute files you generate by these scripts in any license as you like.