Testing your REST APIs end to end

I hope you agree, that testing your code is important. If you don't automate it, you'll need to do it manually, which is slow and tedious. That's why I prefer automated testing. I'm just too lazy to manually test every aspect of the applications I'm building, after each tiny change I do. However, this post is not about unit tests, there are many better posts on this topic out there. This post is about how we built a large and maintainable test suite for our countless API endpoints.

Phase 1 - Postman and its shortcomings

The core of apaleo is our API. When we started building it a bit over 2 years ago, we tested our API using Postman, probably like most people out there do. It allows to create complex flows, export them to JSON files and even has a CLI (newman) which can be used on the build server. We had scripts to automatically generate a test environment, which could be used to set up your local development environment, or to recreate the staging environment at any time. Additionally, we tested flows, which required several coordinated API calls. This all worked really nicely, until it didn't. As the complexity of the system grew, it became more and more painful to maintain the blob of those Postman tests. Towards the end, when we finally started replacing them, we had 3 JSON files, with a total of over 5,000 lines. You can imagine, editing those files often lead to merge conflicts.

Phase 2 - Jest with TypeScript

This might sound weird. Jest was developed by facebook to unit test JavaScript projects. Yet we are using it to test (comparingly) long running API calls. Why did we decide to do so?

When we were looking for an alternative to Postman, we had 2 requirements. The alternative should be ...

• ... easy to use and understand.

  We wanted a tool with syntax highlighting and auto complete, like we were used to from C#.

• ... (fairly) dynamic.

  But it shouldn't be C# or Java, because those languages are very static. That's great for production code, but it would be tedious to write tests in those languages.

That's how we stumbled upon Jest and used it together with TypeScript. As TypeScript is just a layer on top of a dynamic programming language, you can easily manipulate objects, to create new test data and expectations. To make it easy to perfom the necessary API calls, we are using Swagger Codegen to create a client SDK in TypeScript for those tests.

Jest with TypeScript is great, it allows us to test even more complex flows very easily. By enabling TSLint we have reasonable certainty, that those tests are also fairly maintainable over time. Again, they don't need to have the same quality as production code, but people need to be able to understand them.

Phase 2.1 - Time Travel

As we were building more and more complex flows in the tests, we found a new limitation. Many flows depend on the date and time of the call. It was impossible to test these throughly, without having a possibility to travel forward in time. So we implemented time traveling. An HTTP header, which is only available in development, which can dictate the time of the current request. That way, we can now test creating a reservation for 2 days and checking it out after that period, within a fraction of a second.

Just like it's predicted with real time traveling, it can cause a lot of confusion. It's important to understand if the issues you find are actual bugs in the system, or just caused by jumping back and forth in time. The latter obviously won't happen in production and is an idication, that you messed up your tests. However, in general, this allows us now to model every single flow our customers might have even over long periods of time.

Phase 2.2 - Testing Callbacks

As with most other systems out there, HTTP requests aren't the only way you can interact with our API. The API can also callback based on web hooks or some deeper integrations. We started testing those webhook callbacks by using RequestBin as the target. This works great when you test if data is being delivered and you don't care too much about the response of the target.

For the case, when we do actually care about the response, we again chose something, which wasn't necessarily designed for that task, but works great regardless. We are using Pact to set up expected interactions between our service and the target and validate if those interactions were met. We used Pact instead of other solutions, because it has TypeScript type definitions, so it is very easy to set up the interactions, as TypeScript already tells you all the required data.

Summary

By switching from Postman to TypeScript with Jest, implementing a custom time traveling header and introducing Pact to our test suites, we are able to test every aspect of our API, we've come across so far. Also a nice plus point is, that our developers seem to enjoy writing those end to end tests, which definitely helps the test coverage.

Thinking back to those 5k lines of JSON data we had with Postman tests, today we have 50k lines of code, just to test our core service. Those 50k lines of code perform over 10k requests within only a few minutes. All the tests with test data and helper functions amount in total to 65k lines of code and are growing.