Test cases¶
Writing and running a first test¶
Tests are written in classes inheriting TestCase. A test
case consists of:
- some set-up which prepares the environment and the resources required for the test to run,
- a list of assertions, usually as a list of checks that must be verified to mark the test as successful,
- some finalization code which cleans the resources used during the test. It should revert the environment back to its state before the set-up.
Let’s look at a minimal example:
class MinimalExample(asynctest.TestCase):
def test_that_true_is_true(self):
self.assertTrue(True)
In this example, we created a test which contains only one assertion: it
ensures that True is, well, true.
assertTrue() is a method of
TestCase. If the test is successful, it does nothing. Else,
it raises an AssertionError.
The documentation of unittest lists assertion methods implemented by unittest.TestCase.
asynctest.TestCase adds some more for asynchronous code.
We can run it by creating an instance of our test case, its constructor takes the name of the test method as argument:
>>> test_case = MinimalExample("test_that_true_is_true")
>>> test_case.run()
<unittest.result.TestResult run=1 errors=0 failures=0>
To make things more convenient, unittest provides a test runner
script. The runner discovers test methods in a module (or package, or class) by
looking up methods with a name prefixed by test_ in
TestCase subclasses:
$ python -m unittest test_cases
.
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
The runner will create and run an instance of the test case (as shown in the
code above) for each method that it finds. This means that you can add as many
test methods to your TestCase class as you want.
Test setup¶
Let’s work on a slightly more complex example:
class AnExampleWithSetup(asynctest.TestCase):
async def a_coroutine(self):
return "I worked"
def test_that_a_coroutine_runs(self):
my_loop = asyncio.new_event_loop()
try:
result = my_loop.run_until_complete(self.a_coroutine())
self.assertIn("worked", result)
finally:
my_loop.close()
Here, we create a loop that will run a coroutine, ensure that the result of
this coroutine is as expected (it should return an object containing the string
"worked" somewhere). Then we close the loop, even if an exception was
raised.
If we happen to write several test methods, the set-up and clean-up will likely be repeated several times. It’s probably more convenient to move these parts into their own methods.
We can override two methods of the TestCase class:
setUp() and tearDown() which
will be respectively called before the test method and after the test method:
class AnExampleWithSetupMethod(asynctest.TestCase):
async def a_coroutine(self):
return "I worked"
def setUp(self):
self.my_loop = asyncio.new_event_loop()
def test_that_a_coroutine_runs(self):
result = self.my_loop.run_until_complete(self.a_coroutine())
self.assertIn("worked", result)
def tearDown(self):
self.my_loop.close()
Both examples are very similar: TestCase will run
tearDown() even if an exception is raised in the test
method.
However, if an exception is raised in setUp(), the
test execution is aborted and tearDown() will never
run. If the setup fails in between the initialization of several resources,
some of them will never be cleaned.
This problem can be solved by registering clean-up callbacks which will always
be executed. A clean-up callback is a function without (required) arguments
that is passed to addCleanup().
Using this feature, we can rewrite our previous example:
class AnExampleWithSetupAndCleanup(asynctest.TestCase):
async def a_coroutine(self):
return "I worked"
def setUp(self):
self.my_loop = asyncio.new_event_loop()
self.addCleanup(self.my_loop.close)
def test_that_a_coroutine_runs(self):
result = self.my_loop.run_until_complete(self.a_coroutine())
self.assertIn("worked", result)
Tests should always run isolated from the others, this is why tests should only rely on local resources created for the test itself. This ensures that a test will not impact the execution of other tests, and can greatly help to get an accurate diagnostic when debugging a failing test.
It’s also worth noting that the order in which tests are executed by the test runner is undefined. It can lead to unpredictable behaviors if tests share some resources.
Testing asynchronous code¶
Speaking of tests isolation, it’s usually preferable to create one loop per test. If the loop is shared, one test could (for instance) schedule a task and never await its result, the task would then run (and possibly trigger unexpected side effects) in another test.
asynctest.TestCase will create (and clean) an event loop for each test
that will run. This loop is set in the loop
attribute. We can use this feature and rewrite the previous example:
class AnExampleWithTestCaseLoop(asynctest.TestCase):
async def a_coroutine(self):
return "I worked"
def test_that_a_coroutine_runs(self):
result = self.loop.run_until_complete(self.a_coroutine())
self.assertIn("worked", result)
Tests functions can be coroutines. TestCase will schedule
them on the loop.
class AnExampleWithTestCaseAndCoroutines(asynctest.TestCase):
async def a_coroutine(self):
return "I worked"
async def test_that_a_coroutine_runs(self):
self.assertIn("worked", await self.a_coroutine())
setUp() and tearDown() can
also be coroutines, they will all run in the same loop.
class AnExampleWithAsynchronousSetUp(asynctest.TestCase):
async def setUp(self):
self.queue = asyncio.Queue(maxsize=1)
await self.queue.put("I worked")
async def test_that_a_lock_is_acquired(self):
self.assertTrue(self.queue.full())
async def tearDown(self):
while not self.queue.empty():
await self.queue.get()
Note
The functions setUpClass(),
setUpModule() and their tearDown counterparts can not be
coroutine. This is because the loop only exists in an instance of
TestCase.
In practice, these methods should be avoided because they will not allow to reset the environment between tests.
Automated checks¶
Asynchronous code introduces a class of subtle bugs which can be hard to detect. In particular, clean-up of resources is often performed asynchronously and can be missed in tests.
TestCase can check and fail if some callbacks or resources
are still pending at the end of a test.
These checks can be configured with the decorator fail_on().
class AnExempleWhichDetectsPendingCallbacks(asynctest.TestCase):
def i_must_run(self):
pass # do something
@asynctest.fail_on(active_handles=True)
async def test_missing_a_callback(self):
self.loop.call_later(1, self.i_must_run)
This test will fail because the test don’t wait long enough or doesn’t cancel
the callback i_must_run(), scheduled to run in 1 second:
======================================================================
FAIL: test_missing_a_callback (tutorial.test_cases.AnExempleWhichDetectsPendingCallbacks)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/martius/Code/python/asynctest/asynctest/case.py", line 300, in run
self._tearDown()
File "/home/martius/Code/python/asynctest/asynctest/case.py", line 262, in _tearDown
self._checker.check_test(self)
File "/home/martius/Code/python/asynctest/asynctest/_fail_on.py", line 90, in check_test
getattr(self, check)(case)
File "/home/martius/Code/python/asynctest/asynctest/_fail_on.py", line 111, in active_handles
case.fail("Loop contained unfinished work {!r}".format(handles))
AssertionError: Loop contained unfinished work (<TimerHandle when=3064.258340775 AnExempleWhichDetectsPendingCallbacks.i_must_run()>,)
----------------------------------------------------------------------
Some convenient decorators can be used to enable of disable all checks:
strict() and lenient().
All decorators can be used on a class or test function.