One of the greatest things about our software development process is our need and desire to test everything. Unfortunately, this doesn’t always translate into absolute best practices in the test code, coverage, or design, but we are always improving. Something I have been been meaning to get around to is property based (or automated specification based) testing.

The traditional approach to unit testing (and TDD) is to write a test, see red, and then go fix it. If doing TDD religiously, this can lead to some annoying interstitial states where the code is patently absurd, for example returning the constant used in the unit test may satisfy the test until a second test is added.

Another problem with using constants in tests is that we usually don’t explore the edge-cases in our code. We’ll only use characters from the US-ASCII charset as test strings for example, and therefore have little hope of finding encoding issues. We’ll often need to be very diligent and verbose in our tests in order to cover all these aspects, and when push comes to shove and the pressure is on to just ship it, test coverage is an easy thing to drop.

So, techniques that increase both coverage and reuse can be highly valuable.

With all that in mind, we recently started using ScalaCheck to try out property based testing. The results have been excellent!

Arbitrariness

The general idea is that you replace a fixed property of type A with an arbitrary one, created by a generator (basically a Factory in GoF speak). So if I have a test where I need some String for a value, rather that use only the specific one I rack my brain for (and usually come up with “fred”):

[cc lang=’scala’ ]
def pushAddsOne = {
val stack = new Stack[String]
stack push “fred”
stack.peek must beEqualTo(“fred”)
}
[/cc]

I declare that my test should work for all Strings:

[cc lang=’scala’ ]
def pushAddsOne = Props.forAll { s: String =>
val stack = new Stack[String]
stack push s
stack.peek must beEqualTo(s)
}
[/cc]

Props.forAll takes a function for whom it can find an Arbitrary instance for all the arguments, and then runs the test by default 100 times with different arbitrary inputs! Furthermore, once it finds a failure case, it then tries variations of those patterns to see if it can reduce the failure down to the canonical inputs that cause the problem, saving you some of the trouble to work out what is the underlying cause.

Using it in the Real World™

We are writing a Storage API, with the main implementation being file-system based. I wanted some comprehensive tests of this implementation that simulated real world usages. This means storing some large data in it. We are storing a mix of binary and char data, but largely character data which is what I wanted to test.

Here’s an example test:

[cc lang=’scala’ ]
def findNone =
Prop.forAll { hash: Hash =>
execute { Storage.get(hash, _.isDefined) } must beRight like {
case Right(f) => f must beFalse
}
}
[/cc]

To make this work I need to define an Arbitrary[Hash] which I do by asking for an arbitrary[String] (which gives me a Gen[String]) and hashing it by mapping the hash function across it:

[cc lang=’scala’ ]
implicit def ArbitraryHash: Arbitrary[Hash] =
Arbitrary { arbitrary[String].map(hash.SHA1) }
[/cc]

Similarly, I want to put largish, multiline Strings in my file store. So I need some kit to generate these for me:

[cc lang=’scala’ ]
sealed trait MultiLine
type MultiLineString = String @@ MultiLine

def genMultiLineString: Gen[MultiLineString] = {
for {
a b c } yield s”$a\n$b\n$c”
}.map(Tag.apply)

implicit def ArbitraryMultiLineString = Arbitrary(genMultiLineString)
[/cc]

This says, firstly subtype String using unboxed tagged types so we can know that we are dealing with the set of Strings that have multiple lines in them. Then implement a generator that combines three arbitrary Strings with newlines and then tags them as multiline, and finally construct our Arbitrary[MultiLineString] instance from that.

Here’s a test that uses them:
[cc lang=’scala’ ]
def setAndGet =
Prop.forAll { str: MultiLineString =>
execute {
for {
adr get } yield get
} must beRight.like { case Some(s) => s must beEqualTo(str) }
}
[/cc]

Predictably, my tests started failing in Bamboo with errors like1:
[cc lang=’scala’ ]
[error] A counter-example is ‘
[error] 졦
[error] 蠲’ (after 1 try)
[error] ‘Right(Some(
[error] ?
[error] ?))’ is Right but ‘
[error] ?
[error] ?’ is not equal to ‘
[error] 졦
[error] 蠲’
[/cc]

An encoding bug found! I’d not have found this one until after we went into production.

Corollary

This is only the tip of the iceberg. We have since been using this kind of thing to efficiently test other things. My favorite so far was a test that for an abstract trampolining monad interpreter:

[cc lang=’scala’ ]
class InterpretedSpec extends ScalaCheckSpec {
import box.Box
implicit def arbitraryBox[A: Arbitrary]: Arbitrary[Box[A]] =
Arbitrary {
Gen { _ => implicitly[Arbitrary[A]].arbitrary.sample.map { Box.apply } }
}

def is =
s2″””
The monadic interpreter should
implement Monad ${checkAll(monad.laws[Box])}
not blow out the stack ${noStackOverflow}
“””
def noStackOverflow =
Loop[Box](10000) must beEqualTo(10000)
}
[/cc]

Two lines generates about a thousand tests! And of course, as per usual, you can use this stuff to test your Java code as well.

So what are you waiting for? Get cracking!

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now