testcase

Introduction

To aid with functional testing, ALFA includes built-in support to define tests alongside model definitions. Furthermore, tests can be executed and code coverage reports produced to provide quality assurance guarantees.

Testing Areas

Models expressed in ALFA have several areas that can be of interest for testing. This list is not exhaustive, but highlights some areas worth considering for data model testing.

1. Creating a new user-defined type.
   • E.g. Valid numeric values, valid enumeration constants
2. Valid data-type constraints.
   • E.g. Valid string size, valid collection size, valid value ranges.
   • string(3,3), list<int>(1,*), int(0,100)
3. Validate correct assert behaviour.
4. Validate library function behaviour.
5. Correct service function operation.
6. Edge-case values - 00:00, 23:59:59:999, leap years etc.
7. Handing unexpected values.
   • Empty strings
   • Empty collections ( list< T >, set< T >, map< K, V > )
   • Negative or zero values
   • Optional types with no values
   • try< T > type values with errors

Support for Testing

1. Defining a testcase

A testcase definition allow tests to be expressed containing ‘Scenarios’ containing predefined conditions and expected behaviour.

A testcase definition is syntactically identical to a library. All functions within a testcase accept one parameter of type alfa.test.Scenario. The Scenario service methods are invoked to assert the expected behaviour.

entity Order key( ... ) {
    ...
}

testcase OrderTest {
    testScenario1( s : alfa.test.Scenario ) : void {
        // invoke methods on argument 's' to perform testing
    }

    testScenario2( s : alfa.test.Scenario ) : void {
        // invoke methods on argument 's' to perform testing
    }
}

Example Test

testcase PersonTest {
    divideByZero( s : alfa.test.Scenario ) : void {
        // Assert that 10 divide 0 causes a failure
        s::fails("Divide by zero", ( ) => MathLib::divide( 10, 0 ) )
    }

    validatePerson( s : alfa.test.Scenario ) : void {
        // Assert a new Person object can be created
        s::pass("A Person object can be created", ( ) =>  new Example.Person( "Bob", 20 )  )

        // Assert a new Person with age set to 150 fails ( due to constraints set )
        s::failsWith("Person object age constraint upto 120", ( ) =>  new Person("Bob", 150 ), "Invalid age 150" )
    }
}

2. Test naming conventions

1. *Test

   For testcases ending in ‘Test’, ALFA enforces the prefix name matches a type definition in the same namespace. This enables test coverage to report if certain types do not have tests.

2. *TestSuite

   A testcase name ending TestSuite has no convention for the prefix, and can be used to test any set of types, service, library etc.

3. Test Scenario service

The alfa.test.Scenario service exposes number of methods that can be used to compose tests.

// This service is passed into methods that are part of a 'test' user-defined type. Number of method are available to
// formula data required for a test, and execute functionality and assert expected results.
service alfa.test.Scenario() {

    // The specified entity object is available within the test to be accessed via the 'query' or 'lookup' functions.
    given( description : string, data : $entity ) : void

    // Load JSON files from the path specified (file, or directory), and those entity objects are available within the
    // test to be accessed via the 'query' or 'lookup' functions.
    givenAll( description : string, path : string ) : void

    // The specified service is available in the context of this Scenario as a mocked service.
    // When invoked, returns a random value from the results list.
    with( description : string, srv : $service, results : map< FunctionName : string, ReturnValues : list< string > > ) : void

    // When the testBody is executed, no error is expected and completes normally
    pass( description : string, testBody : func<(), void > ) : void

    // When the testBody is executed, an error is expected.
    // This method should be used when the exact error message expected can be implementation specific (E.g.Java, Python etc)
    fail( description : string, testBody : func<(), void > ) : void

    // When the testBody is executed, an error is expected on the field specified by the 'errorFieldPath' field
    failOn( description : string, testBody : func<(), void >, expectedErrorFieldPath : string ) : void

    // When the testBody is executed, an error is expected with the message being reported
    failWith( description : string, testBody : func<(), void >, expectedErrorMessage : string ) : void

    // Assert the lambda returns true
    assertTrue( description : string, testBody : func<(), boolean > ) : void

    // Create a random object of the given type
    random( typeName : string ) : $udt

    // Create a random object using the partially filled in object (randomize the rest)
    randomWith( partiallyFilledObject : $entity ) : $udt
}

Note the testBody argument in testBody : func<(), void > is a lambda expression that is executed to validate the behaviour of the test. This can be a single expression or a code block as { ... }.

Note the alfa.test.Scenario object is valid only for the duration of the method. Each method call will have a new instance of the Scenario object created.

Users are free to choose a naming strategy for tests. For improving coverage, a testcase per user-defined type can be followed as a convention. Tools to enforce availability of tests and test coverage expectations are currently being developed.

Test Coverage

1. ALFA Test Coverage Tool can run these tests and show the results and code coverage output.
2. Using the javatest exporter will generate testcase definitions as Java unit tests that can be executed in JUnit.