I had an interesting discussion about how to test applications through the user interface on JavaZone. I promised to sum up my experiences, so here it is:
Scenario: You have an application with a lot of pages and want to test something a page twelve.
First try: Make your GUI test tool fill out eleven pages and finally do your stuff at page twelve.
- The test takes a long time to run, because of the eleven pages that needs to be filled out.
- The test is fragile, because an error on any of the eleven pages will break the test
- The test is a maintenance nightmare, changes in the chain of pages will require the test to be updated. A lot of tests like this will require a lot of unnecessary work.
Make a setup page in your application that can be used to generate test data. Rewrite the test so that it first uses the setup page to generate test data, and then takes the shortest route to page number twelve. The Mock Client pattern is a good starting point for implementing the setup page. By simulating a client directly against the service layer you also have a pretty nice integration test. It’s important that the mock clients run in Continuous Integration, if they break you want to know it as soon as possible. Remember to secure the setup page, so that it is unreachable in production.
Object per page
Scenario: You are starting out with automatic GUI-testing and are writing tests with the low level API that comes with the tool.
Problem: The the name of a button has changed and a lot of tests are breaking. Because the low level API has been used in different tests, we need to change the name of the button several places.
Solution: Make your own high level API of your application, on object per page. The page object could have methods like:
- page.add_row( “col value 1”, “col value 2” )
- The tests are easier to read
- The tests are faster to write
- A change in a page will only result in a change in the page object
Problem: It’s difficult to run tests because other systems needs to be up and running and test data must be available and synchronised.
Solution: Make an integration page where you could change live integration points with mocks. When the tests run in Continuous Integration use mock integration, but keep the test data as close to live testing as possible. If key test data is equal in both live and mock testing, the tests can be reused for live testing. At the end of the iteration, turn to live integration, and run the tests to verify expected behaviour. Remember to secure the integration page, so that it is unreachable in production.
Words of advice
- Define a maximum test time for your suite, minutes are acceptable, hours are not
- The number of good tests are limited, to many tests will burden the project
- Keep track of tests that fails when the application isn’t broken, fragile tests belong in the garbage bin
- Migrate your tests into integration tests if you can, sounds difficult, but challenge yourself and the team. GUI testing is more expensive than integration testing.