Go Tricks: Unit Testing with testing/quick
The testing/quick package in the Go standard library provides a couple of utility functions for black box testing your code. I recently experimented with it to test a simple function in one of my services in order to learn how to use it. Along the way, I also discovered some issues with the functional code; while I could have discovered these through greater creativity in writing my test cases, using this method takes away the tunnel vision that I think naturally occurs when you know how a function is supposed to be used.
Here is the code that I'm testing:
const characterSet = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
func generateString(numChars int) string {
rand.Seed(time.Now().UnixNano())
randomChars := make([]byte, numChars)
for i := range randomChars {
randomChars[i] = characterSet[rand.Intn(len(characterSet))]
}
return "anonymized_" + string(randomChars)
}
This function is used as part of anonymizing user data in a backend system. All it really does is generate a string that follows the pattern “anonymized_[some combination of random numbers and letters of the specified length]". It looks pretty simple to test; the only real challenge here is that we can't control the seed used by random, so I can't assert directly against expected output. Instead, I'll just test the length is what I expect, and that the string starts with “anonymize_".1 For a little bit of extra safety, I'll also run the function more than once with the same input and assert we get different output for the randomized portion of the string.
Here's what that test ends up looking like:
func Test_generateString(t *testing.T) {
tests := []struct {
name string
numChars int
}{
{
name: "basic test - 10 characters",
numChars: 10,
},
{
name: "basic test - 5 characters",
numChars: 5,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
wantPrefix := "anonymized_"
wantLength := len(wantPrefix) + tt.numChars
got1 := generateString(tt.numChars)
got2 := generateString(tt.numChars)
if got1 == got2 {
t.Errorf("received same output more than once: %s, %s", got1, got2)
}
for _, got := range []string{got1, got2} {
if len(got) != wantLength {
t.Errorf("unexpected length of generated string, got %d, wanted %d", len(got), wantLength)
}
if !strings.HasPrefix(got, wantPrefix) {
t.Errorf("missing prefix %s, got %s", wantPrefix, got)
}
}
})
}
}
This isn't a terrible set of tests. We validate that the function does what we think it does and we have multiple input values tested, so we know the function actually is using the input. It's weakness is that I wrote these tests knowing how I intend to use the function; the function itself does not really do anything to guarantee that another developer is using it in that same way. The testing/quick package is helpful here in that it will use your function any way that it possibly can, so it can really help make your code much more defensive.
I'm going to use quick.Check() to test this. This function takes an interface{} they call f and a pointer to a quick.Config object. For my testing, I don't need to override the default config so I can just pass a nil there. The value for f is a little tricky, because even though it is typed an empty interface, it actually is required that it is a function that returns a bool. So a really basic test using quick looks like:
func Test_generateString_quick(t *testing.T) {
// create a function that takes whatever values you want quick to
// randomly create and returns a bool
f := func(x int) bool {
fmt.Println("input is:", x)
// execute the function under test
s := generateString(x)
// return whether successful or not
return len(s) == x+len("anonymized_")
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
If I run this test, I get a real quick panic:
input is: -1190708547876777949
--- FAIL: Test_generateString_quick (0.00s)
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range
Right! I forgot! Numbers can be negative! Since I'm not checking anything about numChars aside from the fact that it is a whole number, when we try to make a slice of that length called randomChars, the runtime complains if I ask it to make a slice of length -5. This is pretty obvious in retrospect, but in tunnel vision about how the function ought to be used I completely missed this panic causing oversight. Let's add an error to the function signature and a check that the value is greater than zero before we make a slice:
func generateString(numChars int) (string, error) {
if numChars <= 0 {
return "", fmt.Errorf("numChars must be greater than 0, received: %d", numChars)
}
rand.Seed(time.Now().UnixNano())
randomChars := make([]byte, numChars)
for i := range randomChars {
randomChars[i] = characterSet[rand.Intn(len(characterSet))]
}
return "anonymized_" + string(randomChars), nil
}
And now update our unit tests to match. First the original:
func Test_generateString(t *testing.T) {
tests := []struct {
name string
numChars int
}{
{
name: "basic test - 10 characters",
numChars: 10,
},
{
name: "basic test - 5 characters",
numChars: 5,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
wantPrefix := "anonymized_"
wantLength := len(wantPrefix) + tt.numChars
got1, err := generateString(tt.numChars)
if err != nil {
t.Errorf("unexpected error: %s", err.Error())
return
}
got2, err := generateString(tt.numChars)
if err != nil {
t.Errorf("unexpected error: %s", err.Error())
return
}
if got1 == got2 {
t.Errorf("received same output more than once: %s, %s", got1, got2)
}
for _, got := range []string{got1, got2} {
if len(got) != wantLength {
t.Errorf("unexpected length of generated string, got %d, wanted %d", len(got), wantLength)
}
if !strings.HasPrefix(got, wantPrefix) {
t.Errorf("missing prefix %s, got %s", wantPrefix, got)
}
}
})
}
}
And now the quick test:
func Test_generateString_quick(t *testing.T) {
// create a function that takes whatever values you want quick to
// randomly create and returns a bool
f := func(x int) bool {
fmt.Println("input is:", x)
// execute the function under test
s, e := generateString(x)
// return whether successful or not
if e != nil {
if x <= 0 {
return true
}
}
return len(s) == x+len("anonymized_")
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
If I run the quick test again, I learn another important fact about numbers:
input is: 817465805161738976
--- FAIL: Test_generateString_quick (0.00s)
panic: runtime error: makeslice: len out of range [recovered]
panic: runtime error: makeslice: len out of range
Not only can numbers be 0 or even negative, they can also grow quite large. Here, it's complaining that the size of this will be too big. I don't really want people to be trying to make any giganticly long anonymization strings here, so I'll just set a reasonable maximum allowed value. 1024 is a nice number:
func generateString(numChars int) (string, error) {
if numChars <= 0 {
return "", fmt.Errorf("numChars must be greater than 0, received: %d", numChars)
}
if numChars > 1024 {
return "", fmt.Errorf("numChars must not be greater than 1024, received: %d", numChars)
}
rand.Seed(time.Now().UnixNano())
randomString := make([]byte, numChars)
for i := range randomString {
randomString[i] = characterSet[rand.Intn(len(characterSet))]
}
return "anonymized_" + string(randomString), nil
}
If I run the unit test again, now I actually get a real test result, because there is no longer a case where this code panics:
input is: 73690994180292648
--- FAIL: Test_generateString_quick (0.00s)
/home/tmargheim/go/src/local/code-scratchpad/main_test.go:112: #1: failed on input 73690994180292648
FAIL
Note that the first input is line is from our fmt.Println, which I added because we were panicking and not otherwise able to see what was causing our problem. Now that the test assertion is failing, we do get a nice message telling us on which input the test failed. It looks like I need to update my logic around whether the test was successful to know about our new maximum limit:
func Test_generateString_quick(t *testing.T) {
// create a function that takes whatever values you want quick to
// randomly create and returns a bool
f := func(x int) bool {
fmt.Println("input is:", x)
// execute the function under test
s, e := generateString(x)
// return whether successful or not
if e != nil {
if x <= 0 || x > 1024 {
return true
}
}
return len(s) == x+len("anonymized_")
}
if err := quick.Check(f, nil); err != nil {
t.Error(err)
}
}
Now when I run my test, everything passes. Perfect!
The testing/quick package doesn't do anything magic to improve your code. What it does do is anything your signatures allow, which can help identify gaps in your logic. In this case, I'm dealing with an unexported function inside my own code; maybe the defensiveness doesn't provide a ton of value. On the other hand, adding the testing/quick test function really didn't take a whole lot of work either. In production code, I'd probably add a test case representing each failure that quick.Check() identified since I think that that's easier to understand when it fails, but I'd still keep both test functions around to make sure future changes were safe.
-
A quick aside about design here. How could we improve this for testing? I think the most obvious choice would be to refactor to take the seed as an argument to the function. In that way, I could easily control the output and assert more meaningfully in my tests. However, in this case, it's important to me that users of this function do not get duplicate values, and so I'm loathe to open up this function to just an arbitrary seed. Another option would be to wrap the
time.Now()bit in to some sort of injectable helper; again this would allow me to predict the value of my random seed which makes assertion on the output value possible.A final option (for now, I'm sure there's more) would be to set up a method receiver for this function that had the seed as a property. This would allow me to test thoroughly and would probably have some very small performance gain as well, since I'd only need to calculate a seed on initialization. Maybe I'll write a post about benchmark testing and use this code as my test case again. In any case, like all things, it is a trade-off; for our actual use case, the simple tests above were thorough enough to verify the delivered value. ↩︎