In the book Dreaming in Code , the author mentions Axe Sharpening, specifically how development teams can spend too much time sharpening their axe and not enough time cutting down trees.
“Give a person six hours to cut down a tree, the saying goes, and she will spend the first four hours sharpening the axe. In other words, most of us would rather spend time improving the tools that make a job easier than getting on with the job itself”.
However what if you axe becomes completely blunt?
The problem – Too much unwieldy code
The Jira team uses JWebUnit and HttpUnit to help us do functional testing of Jira.
The current Jira web test framework design is based on a class called JiraWebTest, which is derived from WebTestCaseWrapper, which is derived from WebTestCase, which is itself a delegate/mixin for the WebTester class, and finally is derived from the JUnit TestCase class.
JiraWebTest is 7300 lines long (is this a bad code metric?), while the WebTestCaseWrapper is a thin little thing at 1100 lines long.
And just for good measure, we have approximately 230 actual “Test” classes all in the one package. Just to keep things interesting 🙂
These tests consist of approximately 80,000 lines of code, which has grown from 20,000 lines of code in 2 years. And because we are trying to get bettter at testing, it will continue to grow. According to Clover, our combined unit test plus functional test coverage is 72% and its growing at 0.1% per fortnight.
What else is bad about it?
None of the code above is unit tested, which may explain why assertTextSequence() has gone through several iterations, each time causing passing tests to fail and failing tests to pass.
So, will we hit the magic 10,000 lines of code in one class? Can we top 500 classes in one package?
What does the functional test framework do?
The code consists of a bunch of methods that perform the following functional areas.
- General Assertions
- Text Assertions
- Table Assertions
- Link Assertions
- Page Assertions
- Jira general administration operations
- Jira navigation operations
- Jira data setup operations
- Database restore for test setup
- Logging and HTML failure dump processing
How could this happen?
Everything seems perfectly obvious in hindsight.
- Obviously the size of the base classes are too big.
- Obviously the problem domain concerns of the code is too concentrated.
- Obviously the code needs refactoring?
- Obviously packages should be used to organize the classes.
What happened is that the functional test framework code is a little “unloved”. The functional test code grew over time but we didn’t take the time to ensure that the framework code was robust, refactored and re-usable.
Because of the large number of methods in a main classes, new team members have trouble “getting to know” the framework and what methods they can re-use. And hence we end up with multiple methods doing almost the same thing.
We didnt spend enough time sharpening our “functional testing” axe. And now we are beginning to pay for it in. Its proving harder to re-use the functional test code, its proving hard to improve the functional test code and its proving hard for the team to know about the features the functional test code offers.
Whats good about it?
Even with its current unwieldy code design, its still works. It runs 1-20 times a day in Bamboo and runs every night in our nightly build.
The use of web functional tests is all about “insurance” for the future. It has saved our bacon on a numerous times where we have regressed Jira behavior as new features are added or when we have fixed other bugs. It helps enormously to catch these problems during development rather than after release.
What are we going to do about this?
The big question is how should we proceed in the future without destroying the work of the past? We cant afford to refactor the current code but we don’t want to continue “growing” the functional test code based on this framework.
The answer is to create a new functional test framework that can exist along side the old framework. We keep the old tests and begin writing new functional tests using the new framework. This minimize the impact of the change and also ensure Jira continues to be “tested” according to the existing “rules”.
An alternative would be to go back and refactor the old test code to use the new framework. The risk is that you cant “faithfully reproduce” the exact testing rules and hence the “testability” (is there such a word?) of Jira goes down. And we don’t want to take that risk.
Where to from now?
We are currently in the early stage of implementing the new web functional test framework. We hope to do the following things:
- Have web test classes derive from a class as close as possible to JUnit TestCase.
- Break out the assertions from the helper methods and rely instead on the base JUnit Assertions directly.
- Re-organize the code into meaningful packages
- Provide “helpers” that do things like “HTML DOM inspection”, HTML Table inspection, HTML link inspection
- Also provide more “high level helpers” that do Jira administration and navigation operations.
The goal is to then write all new tests using the new framework and to slowly “retire” the old code. It will still exist and be run to test Jira but it wont be actively changed.
By following a few more coding best practices (god I hate that term), in essence give the functional test code a bit more “love”, we hope that it will be better used by development team members, more maintainable over time and able to cope with our expected test growth.