Alexander Münch
Alex programmiert seit über 30 Jahren. Bei Senacor ist er Technical Expert für Kotlin, Spring and Java.
When writing tests, it’s not only about whether they really correctly test your code and how much coverage they achieve. One important thing is also how much information you’ll get when there is a failing test.
Ever since I am using Kotlin, I am using the combination Kotest and Strikt to successfully test my code. Strikt is a small and maybe unknown library, that’s why I want to give an introduction to this library by this very blog post.
Source: strikt.io
Introduction
What is a test?
First, let’s look at what a test is basically doing:
fun test() {
// Setup a scenario, ...
val fluxCapacitor = FluxCapacitor()
// ... call the code under test, ...
val actual = fluxCapacitor.enable()
// ... and assert if that actual value matches the expected value.
val expected = 42
val ok = actual == expected
// If not, fail the test.
if (!ok) {
throw AssertionError("Implementation wrong!")
}
}
Of course, nobody writes a test like that.
We use test frameworks like Kotest to help us. Test frameworks facilitate the execution of tests, provide means to structure our tests, give us the ability to avoid duplication (for example define a base for “these tests need a database”) and many more features.
One of a test framework’s features is to generate reports telling us which tests ran successfully and which ones failed and why. In order to quickly see where the problem is and what is the problem, the tests must include this information.
Looking at the simplified example above, a possible test failure would result in a report like this:
Implementation wrong!
java.lang.AssertionError: Implementation wrong!
at com.example.FluxCapacitorTest.test(FluxCapacitorTest.kt:26)
....This doesn’t give up much information.
We could add more information into the test by changing the last block to:
// If not, fail the test.
if (!ok) {
throw AssertionError(
"Implementation wrong! " +
"When enabling the flux capacitor, " +
"the result was $actual, but we expected it to be $expected."
)
}The output would then say:
Implementation wrong! When enabling the flux capacitor, the result was 43, but we expected it to be 42.
java.lang.AssertionError: Implementation wrong! When enabling the flux capacitor, the result was 43, but we expected it to be 42.
at com.example.FluxCapacitorTest.test(FluxCapacitorTest.kt:26)
....This is what we want, but it’s a lot of work to always include the „actual“ and „expected“ data. In this example the data is just a simple number. In reality, the data will be much more complex.
What’s an assertion library?
This is where an assertion library comes into play. It offers the authors of tests to structurally mark what is the „actual“ data, and what is the „expected“ data.
The assertion library will then compare the two pieces of information. If they don’t match, the assertion library will automatically fail the test and output the actual and expected values, and any additional information that was provided in the test.
In this article, we will have a look at Strikt.
Strikt
Strikt is an assertion library written in Kotlin. It makes use of Kotlin’s lambda syntax, nullability safety und extension functions to provide a nice developer experience.
Using Kotest and Strikt, our exemplary test would look something like this:
class FluxCapacitorTest : StringSpec({
"The FluxCapacitor returns the correct numeric value on enable()" {
val fluxCapacitor = FluxCapacitor()
expectThat(fluxCapacitor)
.get("enabling the flux capacitor") {
enable()
}
.isEqualTo(42)
}
})And a possible output on test failure would be:
▼ Expect that FluxCapacitor #882002928:
▼ enabling the flux capacitor:
✗ is equal to 42
found 43We see, Strikt displays all the information at a glance:
- What overall data object was tested? →
FluxCapacitor #882002928 - What data precisely was looked at? →
enabling the flux capacitor - What was the expected value? →
42 - What was the actual value? →
43 - Did expected and actual match? →
✗, i.e. no
Strikt’s Assertions
Strikt comes with a lot of assertions out-of-the-box to handle common cases. It helps verifying collections, strings, numbers and much more. Have a look at the documentation for all the available assertions.
Strikt provides useful tools to test if code behaves correctly with erroneous input, whether the correct Exception is thrown and whether all its fields are set as expected.
If that’s not enough, Strikt provides additional modules for example a Jackson module.
Strikt and Kotlin’s Null-Safety
Kotlin’s typing and null-safety help a lot when writing tests.
For example
expectThat(someString).contains("some")works fine, however
expectThat(exception)
.message // Assertion.Builder<String?>
.contains("file not found") // COMPILE ERRORException’s message could be null. Strikt provides isNotNull() to verify there is no null value:expectThat(exception)
.message // Assertion.Builder<String?>
.isNotNull() // Assertion.Builder<String>
.contains("file not found") // calls Assertion.Builder<String>.contains()Asserting Multiple Things at Once
There is a golden rule „one assertion per test“, guiding you to not test too much at once, but rather focus on only one particular thing.
However, when testing complex business code, we usually have rich objects and want to verify all its properties are set correctly.
Let’s look at an example found while working with the DGS GraphQL framework:
@Test
fun generateDataClassWithBooleanPrimitiveCreatesIsGetter() {
val schema = """
type MyType {
truth: Boolean!
boxedTruth: Boolean
}
""".trimIndent()
val (dataTypes) = CodeGen(
CodeGenConfig(
schemas = setOf(schema),
packageName = basePackageName,
generateIsGetterForPrimitiveBooleanFields = true
)
).generate()
val typeSpec = dataTypes[0].typeSpec
assertThat(typeSpec.fieldSpecs[0].type.toString()).isEqualTo("boolean")
assertThat(typeSpec.methodSpecs[0].returnType.toString()).isEqualTo("boolean")
assertThat(typeSpec.methodSpecs[0].name.toString()).isEqualTo("isTruth")
assertThat(typeSpec.methodSpecs[1].name.toString()).isEqualTo("setTruth")
assertThat(typeSpec.fieldSpecs[1].type.toString()).isEqualTo("java.lang.Boolean")
assertThat(typeSpec.methodSpecs[2].returnType.toString()).isEqualTo("java.lang.Boolean")
assertThat(typeSpec.methodSpecs[2].name.toString()).isEqualTo("getBoxedTruth")
assertThat(typeSpec.methodSpecs[3].name.toString()).isEqualTo("setBoxedTruth")
}The test needs eight assertions to satisfy the developer that everything works correctly. These eight assertions are formulated in long lines making it hard to read the test. Also, if any one of these assertions fails, the whole test fails, and the report would include that particular problem. That means If there are multiple problems only the first one would be reported.
With Strikt the assertions could look like that:
expectThat(dataTypes)
.first()
.get(JavaFile::typeSpec)
.and {
get(TypeSpec::fieldSpecs).and {
elementAt(0).get(FieldSpec::type).hasToString("boolean")
elementAt(1).get(FieldSpec::type).hasToString("java.lang.Boolean")
}
get(TypeSpec::methodSpecs).and {
withElementAt(0) {
get(MethodSpec::returnType).hasToString("boolean")
get(MethodSpec::name).isEqualTo("isTruth")
}
withElementAt(1) {
get(MethodSpec::name).isEqualTo("setTruth")
}
withElementAt(2) {
get(MethodSpec::returnType).hasToString("java.lang.Boolean")
get(MethodSpec::name).isEqualTo("getBoxedTruth")
}
withElementAt(3) {
get(MethodSpec::name).isEqualTo("setBoxedTruth")
}
}
}
The test has become larger, but more structured.
The benefit is visible when comparing the output of a failing test. To demonstrate this, I created a bug in the setter generation. Since the GraphQL type in the test setup contains two fields, two setters are affected by the bug.
With the current code the output looks like:
org.opentest4j.AssertionFailedError:
expected: "setTruth"
but was: "SetTruth"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at com.netflix.graphql.dgs.codegen.CodeGenTest.generateDataClassWithBooleanPrimitiveCreatesIsGetter(CodeGenTest.kt:202)
... 31 lines ...
at java.util.ArrayList.forEach(ArrayList.java:1259)
... 9 lines ...
at java.util.ArrayList.forEach(ArrayList.java:1259)
... 40 lines ...
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)▼ Expect that [package com.netflix.graphql.dgs.codegen.tests.generated.types;
|
|import java.lang.Boolean;
|import java.lang.Object;
|import java.lang.Override;
|import java.lang.String;
|
|public class MyType {
| private boolean truth;
|
| private Boolean boxedTruth;
|
| public MyType() {
| }
|
| public MyType(boolean truth, Boolean boxedTruth) {
| this.truth = truth;
| this.boxedTruth = boxedTruth;
| }
|
| public boolean isTruth() {
| return truth;
| }
|
| public void SetTruth(boolean truth) {
| this.truth = truth;
| }
|
| public Boolean getBoxedTruth() {
| return boxedTruth;
| }
|
| public void SetBoxedTruth(Boolean boxedTruth) {
| this.boxedTruth = boxedTruth;
| }
|
| @Override
| public String toString() {
| return "MyType{" + "truth='" + truth + "'," +"boxedTruth='" + boxedTruth + "'" +"}";
| }
|
| @Override
| public boolean equals(Object o) {
| if (this == o) return true;
| if (o == null || getClass() != o.getClass()) return false;
| MyType that = (MyType) o;
| return truth == that.truth &&
| java.util.Objects.equals(boxedTruth, that.boxedTruth);
| }
|
| @Override
| public int hashCode() {
| return java.util.Objects.hash(truth, boxedTruth);
| }
|
| public static com.netflix.graphql.dgs.codegen.tests.generated.types.MyType.Builder newBuilder() {
| return new Builder();
| }
|
| public static class Builder {
| private boolean truth;
|
| private Boolean boxedTruth;
|
| public MyType build() {
| com.netflix.graphql.dgs.codegen.tests.generated.types.MyType result =
| new com.netflix.graphql.dgs.codegen.tests.generated.types.MyType();
| result.truth = this.truth;
| result.boxedTruth = this.boxedTruth;
| return result;
| }
|
| public com.netflix.graphql.dgs.codegen.tests.generated.types.MyType.Builder truth(
| boolean truth) {
| this.truth = truth;
| return this;
| }
|
| public com.netflix.graphql.dgs.codegen.tests.generated.types.MyType.Builder boxedTruth(
| Boolean boxedTruth) {
| this.boxedTruth = boxedTruth;
| return this;
| }
| }
|}
|]:We also see all assertions of the test, and which ones were fine, and which ones were failing and why:
▼ value of property typeSpec:
▼ value of property fieldSpecs:
▼ element at index 0 private boolean truth;
|:
▼ value of property type:
✓ has toString() value "boolean"
▼ element at index 1 private java.lang.Boolean boxedTruth;
|:
▼ value of property type:
✓ has toString() value "java.lang.Boolean"
▼ value of property methodSpecs:
▼ element at index 0 public boolean isTruth() {
| return truth;
|}
|:
✓ matches method 'boolean isTruth()'
▼ returns type 'boolean':
✓ is equal to "boolean"
▼ has name 'isTruth':
✓ is equal to "isTruth"
▼ element at index 1 public void SetTruth(boolean truth) {
| this.truth = truth;
|}
|:
▼ value of property name:
✗ is equal to "setTruth"
found "SetTruth"
▼ element at index 2 public java.lang.Boolean getBoxedTruth() {
| return boxedTruth;
|}
|:
✓ matches method 'java.lang.Boolean getBoxedTruth()'
▼ returns type 'java.lang.Boolean':
✓ is equal to "java.lang.Boolean"
▼ has name 'getBoxedTruth':
✓ is equal to "getBoxedTruth"
▼ element at index 3 public void SetBoxedTruth(java.lang.Boolean boxedTruth) {
| this.boxedTruth = boxedTruth;
|}
|:
▼ value of property name:
✗ is equal to "setBoxedTruth"
found "SetBoxedTruth"There is even more improvements hidden in this example. If for example dataTypes did not contain any elements (so there would be no first element), the current code would throw a NoSuchElementException giving no further information. With Strikt, we have dataTypes as the subject. Strikt would still output the data in question, we would see the empty list at a glance. With first() vs. an empty list this is rather trivial, but imagine the dev’s joy when having a list with 4 elements where the 5ᵗʰ element is missing. 😀
Without Strikt’s detailed output you have to sit down in a debugger, execute the test, break at the fitting code location, and inspect the values. Using Strikt with the right subject and assertion is a big time-saver.
Custom Assertions
Strikt uses Kotlin’s extension functions, so it’s easy to extend the library with custom assertions to eliminate duplication.
In the above example I created a generic assertion to interface between javapoet’s classes and the string that is used to check for correctness:
fun <T : Any> Assertion.Builder<T>.hasToString(expected: String) =
assertThat("has toString() value %s", expected) {
it.toString() == expected
}We could also imagine more tailored assertions to the use-case at hand. Strikt allows to write composed assertions like this:
fun Assertion.Builder<MethodSpec>.matchesMethod(
expectedReturnTypeAsString: String,
expectedName: String
) =
compose("matches method '$expectedReturnTypeAsString $expectedName()'") {
get("returns type '$expectedReturnTypeAsString'") { it.returnType.toString() }
.isEqualTo(expectedReturnTypeAsString)
get("has name '$expectedName'") { it.name }
.isEqualTo(expectedName)
} then {
if (allPassed) pass() else fail()
}The test could then simplify checking a method’s signature to just a single line:
elementAt(2).matchesMethod("java.lang.Boolean", "getBoxedTruth")The composed assertion will also display its sub-assertions in the report, so again it’s easy to see where the problem is:
✗ matches method 'java.lang.Boolean getBoxedTruth()'
▼ returns type 'java.lang.Boolean':
✓ is equal to "java.lang.Boolean"
▼ has name 'getBoxedTruth':
✗ is equal to "getBoxedTruth"
found "GetBoxedTruth"My Conclusion
Everybody needs to test their code. Strikt does a good job assisting developers to write good tests. It especially helps a lot finding out where’s problem if there is a test failure (and there always will be test failures 😉).
In order to profit from Strikt’s strengths, you of course have to use the library correctly. For example not using the correct subject, or a non-descriptive get {} will take away the helpful information. You, as a test author, have to write your test in a way all information is available in case of a test failure. Strikt gives you the tools for doing that.
Strikt integrates well into the Kotlin world. This article by far did not show everything Strikt has to offer, so if you are using Kotlin, I highly recommend giving Strikt a try. And since Strikt is independent of your other test frameworks you can easily introduce it side-by-side with existing tests. 🤩
Happy testing!
If you want to learn more about Kotlin, check out this article by my colleagues Lukas and Georg.