FarragoTests

Unit Test Overview

The Farrago unit test framework is based on JUnit. All test cases subclass net.sf.farrago.test.FarragoTestCase, which provides Farrago-specific infrastructure methods (e.g. starting up and shutting down a database instance automatically).

Unit tests can be divided into a number of categories:

• Component tests are standard JUnit tests which exercise a single component in isolation (e.g. FennelTupleTest for package net.sf.farrago.fennel.tuple).
• API tests (e.g. net.sf.farrago.test.FarragoJdbcTest for the JDBC API) are standard black-box JUnit tests.
• SQL-based tests have no associated Java code (other than a generic harness class). Instead, they are implemented as SQL script files together with reference files containing the expected results. More details on their operation are provided below. SQL-based tests can be used to test classes which are very difficult to test in isolation due to large numbers of dependencies. For example, EXPLAIN PLAN is a good way to verify that optimizer transformation rules are firing as expected.

---

Diff-based Tests

In many cases, desired functionality can be verified by producing a string representing the test result and comparing this with another string which defines the expected results. org.eigenbase.test.DiffTestCase provides generic diff-based testing infrastructure. A specific test case calls the DiffTestCase.openTestLog() method to get a java.io.Writer, and writes its output to this Writer. The test case then calls the DiffTestCase.diffTestLog() method to compare the results. Normally, the test output is written to the directory dev/farrago/testlog/TestClassName, with the filename testMethodName.log. The expected results must be checked into source control as a file with the same name except with the extension .ref instead of .log. If the test passes, the .log file is deleted automatically. If the test fails, the .log file is left behind for diagnostic purposes. (Old log files are automatically deleted before testing starts.) No .dif file is ever generated; the comparison is performed by Java code.

TBD: org.eigenbase.test.DiffRepository as an alternative.

When designing diff-based tests, it's important to control for all variables which could cause spurious failures. Possible problems are algorithm non-determinism (for example, a non-stable sort may produce different sort orders on different machines) and environmental differences (e.g. a reference to the local machine's name). Whenever possible, a diff-based test must prevent these (e.g. by specifying an ORDER BY clause with a full key for each SQL statement). (If there are too many, it's a sign that diff-based testing is inappropriate for the test case.) FarragoTestCase also provides a workaround in the form of the addDiffMask method, which allows the caller to specify a regular expression to be masked out before diffing. This should only be used sparingly.

---

SQL Tests

The harness class net.sf.farrago.test.FarragoSqlTest works together with the ant build to find and execute the set of all SQL scripts to be run. The ant build defines a fileset containing all .sql files under the dev/farrago/unitsql directory. This is passed as a multi-line string of filenames to FarragoSqlTest via the property net.sf.farrago.fileset.unitsql. For each script, FarragoSqlTest invokes <a class="wiki_link_ext" href="http://sqlline.sf.net" rel="nofollow">SqlLine</a> to produce a .log file. This is diffed against the .ref file using the diff-based testing support described above. However, the .log and .ref files for SQL tests are stored in the same directory as the source .sql file (rather than under the src directory).

Here's an example test script and its expected output:

set schema sales;
-- test DISTINCT on a single column
select distinct gender from emps order by 1;

GENDER

F

M

For tests dealing with Unicode data, see this thread regarding rules for .sql/.ref file encodings.

Specialized Tests

• Matrix tests are intended to be run multiple times with different configurations. For example, the SQL-based tests under the regressionsql directory are run twice, once with the Fennel calculator implementation and once with the Java calculator implementation.
• Concurrency tests run with multiple threads. A generic framework for concurrent SQL testing is provided by package net.sf.farrago.test.concurrent. The .mtsql scripts under the unitsql/concurrent directory are interpreted by this framework. There is also an mtsql utility for running them as client programs.

---

Test Context

Persistent test fixtures such as the schema SALES are created and populated by all developer builds. See script dev/farrago/initsql/createSalesSchema.sql for an example. Such objects can be referenced (but should not be modified) by any test case as global fixtures. The complete list of fixtures is provided in this table:

Type	Name
Schema	LOCALDB.SALES
Schema	LOCALDB.SQLJ
Schema	LOCALDB.INFORMATION_SCHEMA
Data Wrapper	SYS_*
User	_SYSTEM
User	sa
Role	PUBLIC

At the beginning of each test suite (but not each test case), FarragoTestCase takes care of dropping (with CASCADE) all top-level objects other than these fixtures. This means that tests can rely on being able to use any other object names, and do not have to worry about dropping test objects on exit. Tests should not use these fixtures as parents for new objects, since children of permanent fixtures will not be dropped automatically.

For SQL-based tests, each script is its own test suite. For Java-based tests, each test class is normally a single suite, while each method is a test case. Besides dropping objects at the beginning of each suite, FarragoTestCase also restores system parameters to their pre-test state at the end of each test.

---

Dependence on Native Code

Tests must run successfully when the Fennel native code library is available. Without Fennel, many tests are expected to fail.

---

Test Invocation

The ant test target rebuilds the catalog and runs all unit tests. You can also run a single unit test via the junitSingle script located in dev/farrago. This script takes a single argument. If the argument is the name of an existing file, the file is assumed to be an SQL script and is executed as a diff-based test. Otherwise, the argument is assumed to be the unqualified name of a JUnit test class to be run.

Test results and error details are written to the dev/farrago/testlog directory; a summary of tests run is available in the ant console output.

The junitDebug script operates like the junitSingle script but brings up the JSwat debugger as well.

IDE's with built-in JUnit integration (e.g. Eclipse and IDEA) can also be used to run or debug most tests. In order to allow for IDE execution of SQL scripted tests, the build generates a test class net.sf.farrago.test.FarragoSqlTestWrapper with one method for each .sql file under unitsql. This class is generated during ant createCatalog; it can be regenerated at any time with ant generateSqlTestWrapper.

Once you do that, you can further run tests using naked java by:
java -classpath `cat classpath.gen` junit.textui.TestRunner net.sf.farrago.test.FarragoSqlTestWrapper

If you want to run tests using custom JVM settings, you can do this with by creating a file named dev/farrago/customBuild.properties and adding a line as follows (this example sets the maximum heap size to 512 megabytes):

farrago.test.jvmargs= -Xmx512M

Note that this only affects tests run from ant (including Blackhawk); it does not take affect for commands such as sqllineEngine.

See FarragoTestMachineSharing for how to set up a sandbox to avoid port conflicts with other Farrago sandboxes on the same test machine.

---

Checkin Acceptance Tests

The FarragoCheckinAcceptance page describes the tests which must be successfully run by developers before any checkin. Currently the full test suite is used; we plan to develop a faster acceptance suite which runs only a subset of the full suite.

---

Test Tracing

Besides test log output, test execution also produces trace information in the normal location, dev/farrago/trace/FarragoTrace.log. (This trace log is truncated automatically at the beginning of a full run.) FarragoTestCase inserts trace messages for the beginning and ending of each test case, making it easier to find a problem in the middle of a long run.

---

Test Data

Some tests may require external data; there are several ways to handle this:

• Check in external datafiles. For an example, see dev/farrago/unitsql/med/csv.sql, which reads data from example.csv in the same directory. But please don't check in large datafiles.
• Generate external datafiles from the build. For an example, see dev/farrago/unitsql/ddl/med.sql, which reads data from the mdr files in the same directory (generated by ant createMdrTestData). This option isn't a good general solution since it couples the build with testing (TODO: decouple it and make it easier to use).
• Generate external datafiles from the test itself. This is the recommended approach whenever it makes sense (i.e. generated data can provide the required coverage). For an example, see net.sf.farrago.test.regression.FarragoSorterTest. This test also demonstrates how a Java-based test can use a companion .sql script for setup-only purposes. Data should be generated in a test-specific directory under dev/farrago/testgen.

Test Metadata

(This section is currently only a proposal)

Some tests may be relevant only to a particular configuration. Configuration aspects can include:

• personality {farrago, luciddb, aspen}
• platform {linux, windows}
• test thoroughness {checkin, nightly}
• component choice during matrix testing
  • calculator {javacalc, fennelcalc, autocalc}
  • optimizer {hep, volcano}
• availability of SQL/MED test datasource {sqlserver, oracle, mysql, postgresql, ... }
• repository implementation {mdr, hibernate}
• repository storage {hsqldb,mysql}
• Farrago default character set {ISO-8859-1, UCS-2}
• JVM {JRockit, Hotspot}
• JVM default character set encoding

We may want these to be registered explicitly somewhere (properties file included by build?) so that we can catch typos in metadata.

In the past, we have discussed using Blackhawk's metadata capability for test selection filtering purposes. However, so far we haven't invested in an effort to base Farrago testing on Blackhawk. If we want to start adding test metadata without moving to Blackhawk (or an equivalent such as TestNG), we'll need some other mechanism.

• For JUnit tests, Julian suggested using annotations. This could look something like the next example (which also uses JUnit4 annotations for test declarations, although that may not be required):

    @Test
    @TestForPersonalities('farrago,luciddb')
    @TestForPlatforms('!windows')
    public void testBackupRestore()
    {
        ...
    }

This would mean the test should only be run only for Farrago and LucidDB personalities, on any platform other than Windows. Alternatively, we could just lump everything into generic tags:

    @Test
    @TestForTags('farrago,luciddb,!windows')
    public void testBackupRestore()
    {
        ...
    }

Either way, implementing these annotations requires suite filtering. Note that the presence of multiple tags does not mean to run the test more than once; it just means that the test is compatible with those tags. A matrix test runner could iterate through various predefined combinations of tags (like Mondrian's megatest).

• For SQL-based tests, Julian suggested using comments. We might require the tags to appear in the first block of SQL-style comments in a .sql file; the test runner would process this block looking for tags, and skip the rest of the test as needed.

-- $Id:...$
-- Backup/restore tests
-- @TestForPersonalities('farrago,luciddb')
-- @TestForPlatforms('!windows')

...

Existing tests which mix personalities will need to be refactored, e.g. so that LucidDB-specific tests aren't run against SQLstream. When doing this, care should be taken to coordinate on integrations via farrago-dev to avoid causing unnecessary conflicts.

Test invocation methods such as ant test and junitSingle will need a way to configure desired test tags, e.g.

ant -Dfarrago.test.tags=javacalc,aspen test

--Szuercher 12:05, 10 November 2008 (EST) Perhaps for extra credit: Allow tests to be tagged with bug report ID and provide a mode to run all tests related to a particular bug, overriding the fact that the test is disabled. Use-case: There are three tests disabled due to bug FRG-111. I'm working on a fix and want to run all the related tests.