How Shopify Writes Scalable Go Services
4 patterns we use to deliver emails well
Author: Ishan Ghimire, Originally published April 23rd, 2021
Preface
“Go will be the server language of the future.” — Tobias Lütke, Shopify CEO
Let’s set the scene a bit. Go is one of the fastest growing programming languages. A 2020 HackerEarth survey of ~17,000 developers, had respondents rank it as their most desired language to learn. As an aside, it also seems to be a language of choice for blockchain development, one of the fastest growing spaces in the industry. Shopify’s Production Engineering teams are intimately familiar with the language as many of our services are built with it. As a Production Engineering Intern (Winter 2021), I’ve spent the last 4 months piecing together why and how.
I’ve also learned over the past 4 months about the benefits of scoping projects down. So, let’s focus this article, since an exposition of the totality of how Shopify thinks about this stuff would be an impossible ordeal. We will cover some of the more interesting patterns I’ve identified among the services I’ve worked with (I know, I know, forgive me for the slightly clickbait title). Quick context : I worked in infrastructure, on a collection of services that configure and deliver emails at Shopify.
We will consider scalability in 2 different ways. The first is the more typically associated meaning — how our services scale in production. Our fourth example touches on this. The second is more pertinent to the content of this article — how the code we write scales. This is important because Shopify wants to be a “100 year company.” There’s a couple unsurprising themes throughout, such as separation of concerns, a concept that lends itself well to scalability.
No code presented below is actual source code at Shopify.
Components
Consider a service that has many functions. In email delivery at Shopify, some of these things include a logger, a Kafka producer, a Redis client, and more. For example, to talk to Redis, it’s good practice to create a client abstraction layer that handles this for us. We want to partition these, to better manage each functionality of the app. Consider the Component interface.
Now, when we instantiate a Service, we can pass in anything as a component, as long as it implements Run(). For example, let's add a database client and a Kafka producer. We can treat these components as black boxes.
Now, we can tie this together
Things get interesting when one considers the flexibility of the Component model. For example, Service also implements Run() . Consider a hypothetical case where you want to run multiple services. For example, we have a configuration service and a delivery service for emails. We can create these services from the components they require, but then also treat each service as a component, since they implement Run(). This pattern is infinitely reusable.
One practice we take part in is using either tomb or context to handle tracking and termination of goroutines that our components run in. Context is favoured for reasons outside the scope of this article, but let us examine how we would expand our example by using tomb, since an example with context would require a little more wiring to explain. tomb.Go runs a passed function in a goroutine, tracking its state. The relevant method can be found here.
Testing with Mocks
We use the mock package. Mocks help us develop a testing suite that is able to treat each functionality of a codebase as a black box. Let’s assume I want to test a component that among many things, loads and stores data from a Redis instance.
The problem is, when I test someComponent.Run() I might just want to test the logic of the code contained there, not the extra "dependencies" (loosely implied meaning) the function relies on, such as the redis client operating correctly. Another potential issue might be that I don't actually want to store or load any data while testing. Mocks come to the rescue here. We can create a DataClient interface and a singleton mock struct that also implements Load() and Store(), and then modify someComponent to hold instances of DataClient as opposed to the specified redisClient.
When we test, we will declare a list of calls that we expect to occur within someComponent.Run() such as redis.Load(), as well as their expected return values. We then match these with the actual calls that occur. The actual calls are recorded with Called() as in the Load() and Store() implementations of mockRedisClient, shown above. Here is what testing would look like.
Filters
Our email delivery system has many checks and balances. We call them filters. When there’s different checks we want to enforce, we decouple them, and then employ them in tandem by using a “parent” model. Consider a chat app that is building censorship. They want to filter out messages that start with the letter Z, and messages that are less than 40 characters. Instead of writing these constraints all at once, we can decouple them and then stack them together.
This pattern is infinitely composable. If we require more filters, we can add a parent parameter to length and modify its Filter implementation.
Queuing Emails
One pattern unique to email delivery is an optimization we’ve developed to cut costs. Emails are queued up on an event stream, and then delivered. The problem is, emails are heavy, and this incurs costs. We can’t queue entire email bodies. An optimization we use is to only queue identifiers to emails, and store the actual content in cloud storage. When it is an email’s turn to be delivered, we use the identifier to retrieve the body from the storage. We treat each email as a job that is enqueued. With a Components model, event driven architecture becomes easier to work with, as components such as a runner that enqueues jobs, or a client that stores data can be used together, and at a high level, without needing much understanding of the internal code.
As a Production Engineering Intern, I’ve learned that building for the long term underscores every design decision at Shopify. And this has been some of the ways Shopify will build for the long term with Go.
Originally published at https://beshup.medium.com on April 23, 2021.
