"All your test are belong to us"
Tests are good. More tests are better. Lots and lots and lots of tests are... hard to manage. That's why we wrote Pandokia.
Pandokia is designed as a lightweight test management and reporting system. It consists of a loosely coupled set of components that:
- discover test files
- configure the environment for each test
- invoke external test runners to run tests (with concurrency possible)
- gather and import test results to a database
- identify missing test results
- display browsable (CGI) test reports from the database
Any test runner, reporting tool, or analysis tool that complies with the interface between components can participate in the system.
Pandokia is a command line tool, invoked with a "pdk <verb>" pattern. "pdk --help" shows the available commands.
We assume two primary use cases for Pandokia:
- nightly or continuous-integration batch runs of the whole system
- developer-driven manual runs of parts of the system
Environment, configuration, and disabled tests are managed with special files in the test directory, to make it as easy as possible for the developer. Tests are run individually, so a test that crashes will not abort the entire test run, and may be run concurrently, to make best use of multi-processor systems. Customized email notifications can be set up so that developers receive email only when there are problems in tests they care about.
In addition to the usual Pass/Fail/Error? statuses, pandokia also supports the status of Disabled and Missing. A Missing test is a test that was expected, but for which no test result was received.
It is often useful to record more information about a test than its completion status. Pandokia allows the test author to define Test Definition Attributes and Test Result Attributes (TDAs and TRAs) which will also be stored in the database and accessible through the report generator. Typically, TDAs will include input parameters or thresholds for a test, and TRAs will include the results of calculations performed by a test.
Browser screen shots
These are static snapshots, so the links don't go anywhere, but they illustrate various aspects of the pandokia reporting system:
- screen1: Shows daily report, tabular form. Test sets are grouped by project, and further subdivided by host and status.
- screen2: Shows treewalker partly drilled down to a single project. Tests are subdivided by the first element of the hierarchical test name, and by status. The same display can be further narrowed by host, or by clicking through to the test name or status links.
- screen3: Shows a single test result record, aka the "detailed" report for a single test. Clicking through the tree hierarchy will eventually reach this point. This test shows a large number of test definition and result attributes that have been defined.
- screen4: Shows the test summary page with attributes shown, and demonstrates how common attributes (that have the same value for all tests in a set) are shown. Quickly identifying commonalities within a set of failing tests can be helpful in determining the source of the problem.