{1 OUnit: xUnit testing framework for OCaml}

{2 What is unit Testing?}

A test-oriented methodology for software development is most
effective when tests are easy to create, change, and execute. The
JUnit tool pioneered test-first development in Java. OUnit is
an adaptation of JUnit to OCaml.

With OUnit, as with JUnit, you can easily create tests, name them,
group them into suites, and execute them, with the framework
checking the results automatically.

{!modules: OUnit2 OUnit}

{2 Getting Started}

The basic principle of a OUnit test suite is to have a {i test.ml} file which
will contain the tests, and an OCaml module under test, here named {i foo.ml}.

File {i foo.ml}:
{[
(* The functions we wish to test *)
let unity x = x;;
let funix ()= 0;;
let fgeneric () = failwith "Not implemented";;
]}

The main point of a test is to check that the function under test has the
expected behavior. You check the behavior using assert functions. The simplest
one is {!OUnit2.assert_equal}. This function compares the result of the
function under test with an expected result.

Some useful functions include:
- {!OUnit2.assert_equal} the basic assert function
- {!OUnit2.(>:::)} to define a list of tests
- {!OUnit2.(>::)} to name a test
- {!OUnit2.run_test_tt_main} to run the test suite you define
- {!OUnit2.bracket_tmpfile} that create a temporary filename.
- {!OUnit2.bracket_tmpdir} that create a temporary directory.

File {i test.ml}:
{[
open OUnit2;;

let test1 test_ctxt = assert_equal "x" (Foo.unity "x");;

let test2 test_ctxt = assert_equal 100 (Foo.unity 100);;

(* Name the test cases and group them together *)
let suite =
"suite">:::
 ["test1">:: test1;
  "test2">:: test2]
;;

let () =
  run_test_tt_main suite
;;
]}

And compile the module

{[
$ ocamlfind ocamlc -o test -package oUnit -linkpkg -g foo.ml test.ml
]}

A executable named "test" will be created. When run it produces the
following output.

{[
$ ./test
..
Ran: 2 tests in: 0.00 Seconds
OK
]}

When using {!OUnit2.run_test_tt_main}, a non-zero exit code signals that the
test suite failed.

{2 Extra features}

OUnit supports colored output and JUnit/HTML generation. They are command line
flags or environment variables that you can set before running OUnit test
suites.

{ul
{- Colored output:
{ul
  {- Set the environment variable [OUNIT_CI=true]}
  {- Use the command line flag [-ci true]}
}}
{- JUnit generation:
{ul
  {- Set the environment variable [OUNIT_OUTPUT_JUNIT_FILE=fn]}
  {- Use the command line flag [-output-junit-file fn]}
}}
{- HTML report
{ul
  {- Set the environment variable [OUNIT_OUTPUT_HTML_DIR=fn]}
  {- Use the command line flag [-output-html-dir fn]}
}}}

{2 Advanced usage}

This section is only for advanced users who wish to uncover the power of OUnit.

{!modules: OUnit2}

{3 Error reporting}

The error reporting part of OUnit is quite important. If you want to identify
the failure, you should tune the display of the value and the test.

Here is a list of things you can display:
- name of the test: OUnit uses numbers to define path's test. But an error
  reporting about a failed test "0:1:2" is less explicit than
  "OUnit:0:comparator:1:float_comparator:2"
- [~msg] parameter: it allows you to define, say, which assert has failed in your
  test. When you have more than one assert in a test, you should provide a
  [~msg] to differentiate them
- [~printer] parameter: {!OUnit2.assert_equal} allows you to define a printer for
  compared values. A message ["abcd" is not equal to "defg"] is better than [not
  equal]

{[
open OUnit2;;

let _ =
  "mytest">::
  (fun test_ctxt ->
    assert_equal
      ~msg:"int value"
      ~printer:string_of_int
      1
      (Foo.unity 1))
;;
]}

{3 Command-line arguments}

{!OUnit2.run_test_tt_main} already provides a set of command-line arguments to
help users run only the tests they want:
- [-only-test]: skip all the tests except this one, you can use this flag
  several time to select more than one test to run
- [-list-test]: list all the available tests and exit
- [-help]: display help message and exit

It is also possible to add your own command-line arguments, environment
variables and config file variables. You should do it if you want to define some
extra arguments.

For example:

{[
open OUnit2;;

let my_program =
  Conf.make_exec "my_program"
;;

let test1 test_ctxt =
  assert_command (my_program test_ctxt) []
;;

let () =
  run_test_tt_main ("test1" >:: test1)
;;
]}

The [Conf.make_*] creates a command-line argument, an environment variable and
a config file variable.

{3 Skip and todo tests}

Tests are not always meaningful and can even fail because something is missing
in the environment. In order to handle this, you can define a skip condition
that will skip the test.

If you start by defining your tests rather than implementing the functions
under test, you know that some tests will just fail. You can mark these tests
as pending todo tests. This way they will be reported differently in your test suite.

{[
open OUnit2;;

let _ =
  "allfuns" >:::
  [
    "funix">::
    (fun test_ctxt ->
      skip_if (Sys.os_type = "Win32") "Don't work on Windows";
      assert_equal
        0
        (Foo.funix ()));

    "fgeneric">::
    (fun test_ctxt ->
      todo "fgeneric not implemented";
      assert_equal
        0
        (Foo.fgeneric ()));
  ]
;;
]}

{3 OUnit2.Threads}

This module provide thread related utilities. In particular, it provides a
"thread" runner, that allows to run concurrently tests using OCaml threads.
This should provide a good level of parallelism on Windows, for UNIX systems
it is recommended to use the standard "process" runner.

To install the extra thread runner:

{[
let () = OUnitThreads.init ()
]}

{!modules: OUnitThreads}

{3 Effective OUnit}

This section has general tips about unit testing and OUnit. It is the
result of some years using OUnit in real-world applications.

- test everything: the more you create tests, the better chance you have to
  catch errors in your program early. Every submitted bug to your application
  should have a matching test. This is a good practice, but it is not always
  easy to implement.
- test only what is really exported: on the long term, you have to maintain your
  test suite. If you test low-level functions, you'll have a lot of tests to
  rewrite. You should focus on creating tests for functions for which the
  behavior shouldn't change.
- test fast: the best test suite is the one that runs after every single build.
  You should set your default Makefile target to run the test suite. It means
  that your test suite should be fast to run, typically, a 10s test suite is
  fine.
- test long: contrary to the former tip, you should also have a complete test
  suite which can be very long to run. The best way to achieve both tips, is to
  define a command-line argument [-long] and skip the tests that are too long in
  your test suite according to it. When you do a release, you should run
  your long test suite.
- family tests: when testing behavior, most of the time you call exactly the
  same code with different arguments. In this case [List.map] and
  {!OUnit2.(>:::)} are your friends. For example:

{[
open OUnit2;;

let _ =
  "Family">:::
  (List.map
    (fun (arg,res) ->
      let title =
        Printf.sprintf "%s->%s" arg res
      in
        title >::
        (fun test_ctxt ->
          assert_equal res (Foo.unity arg)))
      ["abcd", "abcd";
       "defg", "defg";
       "wxyz", "wxyz"])
;;
]}

- test failures and successes: the most obvious thing you want to test are
  successes, i.e. that you get the expected behavior in the normal case. But
  most of the errors arise in corner cases and in the code of the test itself.
  For example, you can have a partial application of your {!OUnit2.assert_equal}
  and never encounter any errors, just because the [assert_equal] is not called.
  In this case, if you test errors as well as the "happy path", you will have
  a notice the missing errors as well.
- set up and clean your environment in the test: you should not set up and clean
  your test environment outside the test. Ideally, if you run no tests, the
  program should do nothing. This also ensures  that you are always testing in a
  clean environment, not polluted by the result of failed tests of an earlier
  test run. This includes the process environment, like current working
  directory.

{[
open OUnit2;;

let _ =
  (* We need to call a function in a particular directory *)
  "change-dir-and-run">::
  (fun test_ctxt ->
    assert_command ~chdir:"/foo/test" "ls" [])
;;
]}
- separate your tests: OUnit test code should live outside the code under a
  directory called {i test}. This allow to drop the dependency on OUnit when
  distributing your library/application. This also enables people to easily
  make a difference from what really matters (the main code) and what are only
  tests. It is also possible to have the tests directly in the code, like in
  Quickcheck-style tests.

The unit testing scope is always hard to define. Unit testing should be about
testing a single feature. But OUnit can also help you to test higher-level
behavior, by running a full program for example. While it isn't real unit
testing, you can use OUnit to do it and should not hesitate to do it.

In terms of lines of codes, a test suite can represent from 10% to 150% of the
code under test. With time, your test suite will grow faster than your
program/library. A good ratio is 33%.

{3 OUnit2.Advanced}

These modules should only be used when building low-level OUnit features. They
allow to create your own process runner or logger.

Modules available in [ounit2.advanced]:
{!modules: OUnitAssert OUnitBracket OUnitCache OUnitCheckEnv OUnitChooser OUnitConf OUnitCore OUnitDiff OUnitLogger OUnitLoggerCI OUnitLoggerHTML OUnitLoggerJUnit OUnitLoggerStd OUnitPlugin OUnitPropList OUnitResultSummary OUnitRunner OUnitRunnerProcesses OUnitShared OUnitState OUnitTest OUnitTestData OUnitUtils}

@author Maas-Maarten Zeeman
@author Sylvain Le Gall
