Fast but not furious quality engineering

[Hita]

The Technology Team at Marqeta has always worked hard to maintain a culture of testability and speed. As a sophisticated global payments issuer processor, we have some of the most advanced infrastructure and tools for building highly configurable payment cards. Behind the curtains, this means a growing stack of heterogeneous technologies and microservices that all need to work together.

Growing pains

As a Quality Engineer, the biggest hurdle was spinning up specific versions of several microservices and applications in a single environment for true integration testing, all while still maintaining agility. With over 25,000 automated tests for just our core services, and several other test suites for each application, we needed a way to spin up a full stack environment for all our testing requirements (e.g. regression, performance, feature, and integration testing), and we needed it to be 100% automated.

Solution

Sushi Boat, our one-stop shop for spinning up services and applications in a containerized environment using Docker Compose. Docker allows containers to be linked, granting the linked container shell access and allowing it to run scripts against the application’s container. This link can also be made between the Docker application container and another container which is packaging a service the application needs to run against for a true integration test.

It did exactly what we needed:

1. Build and deploy a specific version of each and every service and application.
2. Automatically seed test data and add dependencies for these applications.
3. Isolate applications and services from stray scripts and workers.
4. Set up run time properties such as environmental variables, commands and ports on each application and service.
5. Ultimately create an environment where we can have true integration testing with all of our applications and services, while still maintaining agility.

Having this in place, we now have an automated way to make complex testing possible, predictable, and fast across multiple applications, APIs, and microservices — all while maintaining parity between the test environments and production environments.

Structure of Sushi Boat

Sushi Boat was designed as a collection of source code clones of all our applications and their dependencies. As the name illustrates, the different applications are sushi’s within one package called the boat. For example if we needed to spin up our core API’s (lets call it app1), we would clone app1 to sushi boat. We would also have to add dependencies like a .yml file for docker, .conf file for configurations, a set-env file to set up environmental variables, log folder for storing logs, bash script to set branch or pull latest from git and spool folders to dump files to be picked by other applications.

An example .yml file:

 An example set-env file:

 An example .conf file:

An example bash script to manage repositories:

add some fun to your build with jokes

Burnt bridges

With several dependencies for each microservice to run, every test environment had multiple services communicating with each other. We achieved that using bridge networking. The caveat of this was, slow response because all local traffic must go through a proxy process due to limitations in linux bridge networking and lacking the means to replicate production scenarios since many applications were living on the same host but with different ports. This was not working out for us when the ports were defined by the consumer for only one application and more than one application was to be consumed.

We found the perfect solution by using Consul. Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform. Applications could reach out to it to procure the host name. The host names could be used alongside ports to connect to the correct container.

To overcome slow response we had few of the containers communicate with each other directly than going through this port forward. This was achieved by creating a network and then have each container use that network to communicate with each using just container name.

$ docker network create myApp
$ docker run --net=myApp <other options>

Twinning and winning!

 When working on multiple projects at once, we encountered a use case where we need to have three instances of the same service but with different branches. To get that up and running, the first step included cloning the repository thrice. This raised an issue of port conflicts. To avoid that we included a port generator that generates random ports and makes sure it is not already existing to avoid port conflicts.

 Finally running docker compose up on all the three containers built applications with their respective branches!

Next steps

Sushi Boat has momentously helped us create several local instances parallely and run tests on each without polluting the others. There are several roadblocks with every new requirement like seeding giant data to simulate some of our customers who have ever growing data points, maintaining several test environments etc. Many of the questions are yet to be answered ¯\_(ツ)_/¯ but with what we have, we have cut down on setup time and delivered test results faster.

This is just the beginning. Future enhancements include running sushi boat on the cloud to automate release certifications, run not just multiple instances of a single service within sushi boat but multiple sushi boats themselves etc. The efforts are moving towards a direction to thoroughly automate and verify our production deploys with confidence and to scale and expand of our products. Additionally there is so much to do besides all that has already been done.

We are always hiring! If you find this exciting and something to be a part of, check out our openings: https://www.marqeta.com/company/careers