Existing Solutions

When we began exploring this space, we wanted a contract testing solution that was easy to get started with. We also wanted it to be deployment-aware, meaning that it could quickly identify the contracts that mattered for a given deployment environment. We examined the most popular options, including Pact, PactFlow, Karate, Specmatic, and Spring Cloud Contract. Given the importance of independent deployability, we eventually narrowed our search to PactFlow and Specmatic, since these were the only prominent options that offered a spec-driven approach.

PactFlow

PactFlow is a managed, subscription-based, Software-as-a-Service (SaaS) product for contract testing. It expands on the open-source, consumer-driven Pact framework to provide a service they call "bidirectional" contract testing. Bidirectional supports both spec-driven and provider-driven models by allowing the provider's API spec to be published before or after it has been tested against the provider service.

A notable characteristic of PactFlow is that it uses a backend component known as a broker to store and manage contracts. Having a broker allows PactFlow to keep track of deployments and inform developers of whether or not a service can be safely deployed to a given environment.

A major challenge we see with PactFlow is that even though it is a paid service, it still requires substantial effort to adopt contract testing. Although the authoritative contract is an API spec, PactFlow also uses a consumer contract to represent the consumer service internally. In order to generate this contract, consumer teams either need to write new unit tests with a client library from the Pact ecosystem, or else configure their own third-party tool to generate the consumer contract another way. PactFlow also does not support provider verification out of the box. Provider teams need to supply their own solution for testing the provider's implementation against the API spec.

Specmatic

Specmatic is an open-source offering for spec-driven contract testing. A key feature of Specmatic is that it offers a way to generate the contract by automatically recording the interactions between the consumer and provider services. This means that contract testing can be achieved with a significantly smaller startup cost — there is no need to write new unit tests to get up and running.

Another characteristic of Specmatic is that it does not use a broker — contracts are stored in version control instead. While this comprises a simple solution for managing contracts, it loses out on some useful features that a broker can provide, one of which is deployment-awareness. If users of Specmatic want their CI/CD pipeline to automatically gate deployments, they will have to build their own solution to identify the relevant contracts per environment.

A Need for a New Solution

Existing solutions offer clear trade-offs in terms of contract testing approach and feature set. A recurring theme among many of them is that adopting contract testing is costly, requiring significant investment in writing new unit tests.

Other solutions lack awareness of deployments. Although they can test services against a contract, they don't have a way to know whether that service is compatible with the services that are currently deployed to any given environment.

There are also a limited number of solutions that offer a spec-driven approach. Spec-driven contract testing is desirable to many companies that want to maximize independent deployability. It also aligns most naturally with teams that use a spec-first approach to API design.

Since we could not find a solution with the combination of trade-offs we were looking for, we decided to build our own.