flibs/ftnunit(n) 1.2 "flibs"
flibs/ftnunit - Unit testing
TABLE OF CONTENTS
SYNOPSIS
DESCRIPTION
OUTPUT
ROUTINES
CUSTOMISATION
GENERATING TESTS FROM A TABLE
TODO
RELATED WORK
COPYRIGHT
| runtests.bat |
| runtests.sh |
| runtests.tcl |
| ftnunitgui.tcl |
| call runtests( testproc ) |
| call runtests_init |
| call runtests_final( stop ) |
| call test( proc, text, ignore ) |
| call assert_true( cond, text ) |
| call assert_false( cond, text ) |
| call assert_equal( value1, value2, text ) |
| call assert_comparable( value1, value2, margin, text ) |
| call assert_inbetween( value, vmin, vmax, text ) |
| call assert_files_comparable( filename1, filename2, margin, text ) |
| exists = ftnunit_file_exists( filename ) |
| call ftnunit_get_lun( lun ) |
| call ftnunit_remove_file( filename ) |
| call ftnunit_make_empty_file( filename ) |
| cmd call ftnunit_hook_test_start( text ) |
| cmd call ftnunit_hook_test_stop( text ) |
| call ftnunit_hook_test_failure( text, assert_text, failure_text ) |
| call ftnunit_hook_test_complete |
|
JUnit is a well-known facility for defining and running unit
tests in Java programs. The ftnunit framework was inspired by
that facility. It is not as good-looking as JUnit, by no means:
-
It has a simple graphical user-interface that does not group the tests.
-
As Fortran does not allow introspection, the test routines can not
be detected automatically, instead as a programmer you need to set up a
high-level routine yourself that collects all the unit tests.
-
A runtime error, like division by zero, may lead to a termination of
the program. There is no (portable) way to catch these. Instead, the
framework relies on a batch file or shell script to repeatedly start the
program until all tests are run.
Despite these limitations, ftnunit can be a great help:
-
The Tcl program gentabletest.tcl generates a complete test program based
on a simple input file (see GENERATING TESTS FROM A TABLE).
-
The code to test the various components (subroutines, functions, tasks
consisting of several program units) can be combined with the program
itself, without interfering with the ordinary code.
This is achieved by defining a single routine (test_all, say) that runs
all the unit tests and that is called via the provided routine
runtests:
| |
program myprog
...
!
! The routine runtests will check if unit tests are requested
! If not, it will return immediately. This way we make sure
! the unit tests remain part of the program.
!
! The routine test_all runs all unit tests
! (see the dataproc_testing module)
!
call runtests( test_all )
!
! Ordinary processing
!
...
end program
|
The routine runtests checks if there is a file "ftnunit.run". If there is
such a file, it will run the given subroutine. Otherwise it will return
and the rest of the program is executed.
-
Because the test code is incorporated in the program itself, it is less
likely that they evolve independently: changes in the argument lists of
the subroutines and functions may lead to compile errors in the test
code.
-
There is no need to set up a whole new program for testing portions of
the program.
The source file "test_ftnunit.f90" illustrates how to use the ftnunit
framework:
-
The main program calls the routine "runtests" and passes it the argument
"test_all", a routine defined in a module called "dataproc_testing".
-
The routine "test_all" consists of nothing but calls to the generic
routine "test":
| |
subroutine test_all
call test( test_no_file, "Read non-existent file" )
call test( test_empty_file, "Read an empty file" )
call test( test_invalid_file, "Read an invalid file" )
call test( test_ordinary_file, "Read an ordinary file" )
end subroutine test_all
|
-
The module includes a source file "ftnunit_test.f90". This is a remnant
of a previous version. Please ignore this.
-
The generic routine "test" checks whether a particular unit
test needs to be run (via the test number) and then runs the subroutine
that was passed as one of its arguments. One such routine looks like
this:
| |
subroutine test_no_file
integer :: nodata
real :: vmean, vmin, vmax
call ftnunit_remove_file( 'no_such_file' )
call write_name( 'no_such_file' )
call open_files
call process_data( nodata, vmean, vmax, vmin )
call assert_true( nodata == 0, "No data read" )
end subroutine test_no_file
|
The assertion is used to check that the result is as expected.
-
The program contains some deliberate errors and the resulting
log file looks like this:
| |
Test: Read non-existent file
Test: Read an empty file
Test: Read an invalid file
forrtl: severe (59): list-directed I/O syntax error, unit 11, file c:\arjen\flibs\tests\ftnunit\invalid_file
Image PC Routine Line Source
test_ftnunit.exe 004151B9 Unknown Unknown Unknown
test_ftnunit.exe 00415017 Unknown Unknown Unknown
test_ftnunit.exe 004141F4 Unknown Unknown Unknown
test_ftnunit.exe 00414629 Unknown Unknown Unknown
test_ftnunit.exe 00409C05 Unknown Unknown Unknown
test_ftnunit.exe 004095FB Unknown Unknown Unknown
test_ftnunit.exe 0040144B Unknown Unknown Unknown
test_ftnunit.exe 00401FE9 Unknown Unknown Unknown
test_ftnunit.exe 00401A2C Unknown Unknown Unknown
test_ftnunit.exe 00401BB3 Unknown Unknown Unknown
test_ftnunit.exe 0040294A Unknown Unknown Unknown
test_ftnunit.exe 0040232E Unknown Unknown Unknown
test_ftnunit.exe 0044A1E9 Unknown Unknown Unknown
test_ftnunit.exe 00433519 Unknown Unknown Unknown
kernel32.dll 7C816D4F Unknown Unknown Unknown
Incrementally linked image--PC correlation disabled.
Test: Read an ordinary file
Number of failed assertions: 0
Number of runs needed to complete the tests: 3
|
The program is run via one of the following files:
- runtests.bat
-
A batch file for use under MS Windows
- runtests.sh
-
A Bourne shell script for use under UNIX/Linux or similar systems, like
Cygwin or Mingw.
- runtests.tcl
-
A Tcl program that presents a simple graphical user-interface
- ftnunitgui.tcl
-
A Tcl-based graphical user-interface that allows you to run tests one at a time
When you run a program in test mode via the batch file/shell script,
it produces two output files:
-
runtests.log contains all the output that was written to the screen.
-
ftnunit.html is an HTML file with a summary of the tests and a colour
encoding to indicate the test cases that succeeded or failed.
The module ftnunit contains the following subroutines and functions:
- call runtests( testproc )
-
Routine to start the unit tests. It checks if the file "ftnunit.run"
exists. If so, it will call the subroutine testproc that was
passed. Otherwise it will simply return, so that the ordinary program
execution may continue.
If the subroutine testproc returns, the program stops, unless you have
called the subroutine runtests_init before runtests.
- subroutine testproc
-
Subroutine that calls the individual test routines. It takes no
arguments. It wil generally exist of a series of calls to the
routine test - see below.
- call runtests_init
-
Routine to initialise the ftnunit system, so that you call runtests
more than once. To complete the tests, call runtests_final, as
this will print the final statistics and stop the program.
- call runtests_final( stop )
-
Routine to finalise the ftnunit system: it will print the final statistics
and stop the program, but only if the file "ftnunit.run" is present.
- logical, optional, intent(in) stop
-
If present and set to true, the routine will not stop the program.
Instead it returns and the program continues working.
- call test( proc, text, ignore )
-
Routine to run the individual unit test routine (emph proc). It decides
if the test has not run yet and if so, the test routine is called.
Otherwise it is skipped.
test takes care of all administrative details.
- subroutine proc
-
Subroutine that implements an individual unit test. It takes no
arguments. Within each such subroutine the complete unit test is run.
- character(len=*), intent(in) text
-
Text describing the particular unit test. It is printed in the log
file.
- logical, optional, intent(in) ignore
-
If present and set to true, the test routine will not be actually run.
Instead it is shown as "ignored". This feature is useful if the code to
be tested is not yet ready for testing.
file.
- call assert_true( cond, text )
-
Routine to check that a condition is true. If not, a message is printed
in the log file and the number of failures is increased.
- logical cond
-
The condition to be checked
- character(len=*), intent(in) text
-
Text describing the condition
- call assert_false( cond, text )
-
Routine to check that a condition is false. If not, a message is
printed in the log file and the number of failures is increased.
- logical cond
-
The condition to be checked
- character(len=*), intent(in) text
-
Text describing the condition
- call assert_equal( value1, value2, text )
-
Routine to check that two logicals, two strings, or two integers are equal
or if two one-dimensional integer, logical or string arrays are equal. If not, a message is
printed, along with the values that were different.
- <type> value1
-
The first integer/logical/string value
- <type> value2
-
The second integer/logical/string value
- character(len=*), intent(in) text
-
Text describing the condition
Or:
- integer [, dimension(:)] value1
-
The first integer value or array
- integer [, dimension(:)] value2
-
The second integer value or array
- character(len=*), intent(in) text
-
Text describing the condition
- call assert_comparable( value1, value2, margin, text )
-
Routine to check that two reals are almost equal or if two one-dimensional
real arrays (single or double precision) are almost equal. If not, a message
is printed, along with the values that were different.
The margin is taken as a relative tolerance. Two values are
considered almost equal if:
| | abs( v1 - v2 ) < margin * (abs(v1)+abs(v2)) / 2 |
- real [, dimension(:)] value1
-
The first real value or array
- real [, dimension(:)] value2
-
The second real value or array
- character(len=*), intent(in) text
-
Text describing the condition
- call assert_inbetween( value, vmin, vmax, text )
-
Routine to check that a (single or double precision) real value lies between
two given bounds. This establishes an absolute range for the values, rather than
a relative.
- real value
-
The real value to be tested
- real vmin
-
The minimum value that is allowed
- real vmax
-
The maximum value that is allowed
- call assert_files_comparable( filename1, filename2, margin, text )
-
Routine to check that two files are equal or almost equal. The files are scanned
line by line and item by item. If an item can be interpreted as a number, then
the comparison is done using a margin, otherwise the corresponding items are
considered strings and should be exactly equal.
A report is produced of all differing lines.
The margin is taken as a relative tolerance. Two numerical values are
considered almost equal if:
| |
abs( v1 - v2 ) < margin * (abs(v1)+abs(v2)) / 2 |
- character(len=*) filename1
-
The name of the first file to read
- character(len=*) filename2
-
The name of the second file to read
- character(len=*), intent(in) text
-
Text describing the condition
- exists = ftnunit_file_exists( filename )
-
Logical function to check that a particular file exists
- character(len=*), intent(in) filename
-
Name of the file to be checked
- call ftnunit_get_lun( lun )
-
Subroutine to get a free LU-number
- integer, intent(out) lun
-
Next free LU-number
- call ftnunit_remove_file( filename )
-
Subroutine to remove (delete) a file
- character(len=*), intent(in) filename
-
Name of the file to be removed
- call ftnunit_make_empty_file( filename )
-
Subroutine to make a new, empty file
- character(len=*), intent(in) filename
-
Name of the file to be created
The module ftnunit_hooks can be used to customise the output
of ftnunit to a certain extent. It provides four routines
that you can easily adapt to your needs:
- cmd call ftnunit_hook_test_start( text )
-
]
Routine to signal the start of a test.
- string text
-
The descriptive text passed to the routine test.
- cmd call ftnunit_hook_test_stop( text )
-
]
Routine to signal the end of a test.
- string text
-
The descriptive text passed to the routine test.
- call ftnunit_hook_test_failure( text, assert_text, failure_text )
-
Routine to signal that an assertion has failed (and therefore the test).
- string text
-
The descriptive text passed to the routine test.
- string assert_text
-
Text describing the assertion
- string failure_text
-
The text describing the reason of the failure
- call ftnunit_hook_test_complete
-
Routine called after the tests have been run (for instance to start and
Internet browser to view the HTML report).
Note that the default implementation consists of empty routine.
As an example of customisation, under Windows, you could start the
default Internet browser showing the HTML file like this:
subroutine ftnunit_hook_test_complete
call system( "ftnunit.html" )
end subroutine ftnunit_hook_test_complete
provided an association for files with extension "html" exists.
The Tcl program "gentabletest.tcl" reads the test specifications from an
input file and generates a complete Fortran program. The ideas from
Bil Kleb's "Toward Scientific Numerical Modeling"
ftp://ftp.rta.nato.int/PubFullText/RTO/MP/RTO-MP-AVT-147/RTO-MP-AVT-147-P-17-Kleb.pdf
were used for the set-up.
To do: provide a detailed description. For the moment: see example.tbl, including below.
| |
! Example of generating test code via a table
! -------------------------------------------
! The routine to be tested determines the minimum oxygen concentration
! in a river, based on the Streeter-Phelps model:
!
! dBOD/dt = -k * BOD
!
! dO2/dt = -k * BOD + ka * (O2sat-O2) / H
!
! where
! BOD - biological oxygen demand (mg O2/l)
! O2 - oxygen concentration (mg O2/l)
! O2sat - saturation concentration of oxygen (mg O2/l)
! k - decay rate of BOD (1/day)
! ka - reareation rate of oxygen (m/day)
! H - depth of the river
!
! We need boundary (initial) conditions for BOD and oxygen and
! the equations describe the concentrations of BOD and oxygen in a
! packet of water as it flows along the river.
!
! Note:
! It is a very simple model, it is not meant as a realistic
! representation.
!
! The routine simply continues the solution until a minimum is found.
! The results are: oxymin and time
!
!
! The keyword DECLARATIONS introduces the declarations we need for the
! complete generated code
!
DECLARATIONS
use streeter_phelps
real :: bod, oxy
real :: k, ka, h, oxysat, dt, oxymin, time
!
! The keyword CODE introduces the code fragment required to run the
! routine or routines. The results and possible checking of error
! conditions are separated.
!
CODE
call compute_min_oxygen( bod, oxy, k, ka, h, oxysat, dt, oxymin, time )
!
! The keyword RESULT indicates which arguments/variables hold the
! interesting results. Specify one name per line (you can not currently
! use array elements) and the allowed margin (taken as absolute, if
! followed by "%" as a percentage)
!
RESULT
oxymin 0.001 ! Minimum oxygen concentration
time 0.01% ! Time the minimum is reached
!
! The keyword ERROR is used for a code fragment that checks if the
! routine has correctly found an error in the input (that is, some
! parameter value is out of range). The code is invoked when any of
! result variables in a table entry has the keyword ERROR instead of
! a proper value.
! Use the subroutine "error" to indicate the correctly reported error
! condition.
!
ERROR
if ( time == -999.0 ) then
call error
endif
!
! The keyword RANGES specifies that the variables are to be taken
! from a uniform or a normal distribution. The generated program will
! simply select values at random and run the code with them. The report
! consists of the detailed output as well as a summary.
!
RANGES
oxy 10.0 2.0 Uniform ! Name of the variable, the mean and the margin (uniform)
! Normal: mean and standard deviation followed by Normal
! Note: all parameters must be given!
!
! The keyword TABLE indicates the beginning of a table of input data and
! expected values. The first (non-comment) line contains the names of
! the variables as used in the code fragments and all others are the
! values expected.
!
! There are two special values:
! ? - indicating an unknown value for result variables and a "do not
! care" value for input variables
! It is useful to generate a table that does contain the (computed)
! results (see the file table.out) or to indicate situations
! where one or more input variables are out of range and this
! should lead to an error
! ERROR - indicating that the entry should cause the routine to be
! tested to flag an error condition.
!
TABLE
dt oxy bod oxysat h k ka oxymin time
0.1 10 1 10 10 0.1 1.0 10.0 2.0
1.0 10 1 10 10 0.1 1.0 ? ?
!
! This case is unacceptable: time step must be positive
0.0 ? ? ? ? ? ? ? ERROR
1.0 0. 10 10 10 0.1 1.0 ? ?
|
The following things are still left to do:
-
Proper inclusion of the routine prolog and epilog
-
Extension of the set of assertion routines
There are at least two similar initiatives with regard to a unit testing
framework for Fortran:
(Note: To avoid confusion, I have renamed my original module "funit" to
ftnunit)
Copyright © 2009 Arjen Markus <arjenmarkus@sourceforge.net>