Advanced Technical Topics: Orchestration

· by

This is the second post in a series of advanced technical topics.

In the previous post, we discussed idempotence, the idea that some code must be written such that it can be run multiple times with the same inputs and deliver the same results and side effects. The post was running a little long, though, so I decided to punt on the very real problem of thread safety (probably a mistake), and use it as a launch pad for discussing orchestration.

So, let’s go back to our example of the e-commerce workflow. Once a user purchases a product, we need to perform the following actions exactly one time each, and in order. No task may start until the previous task has completed:

  • Reserve inventory
  • Charge credit card
  • Generate shipping label
  • Ship merchandise

When Dinosaurs Roamed the Earth

In the old days, we kept track of everything in the database. We’d have a transactions table with a status field, and a single server which ran code that looked something like this (imagine this in whichever language you’d prefer, with an appropriate async-await harness):

function getNextTransactionFromDatabase() {
  transaction = db.query("SELECT * FROM transactions WHERE status='new' ORDER BY id LIMIT 1")
  db.query("UPDATE transactions SET status='in progress' WHERE id=%1", [transaction.id])
  return transaction
}
function handleTransaction() {
  transaction = getNextTransactionFromDatabase()
  if (transaction != null) {
    reserveResult = reserveInventory(transaction)
    chargeResult = chargeCreditCard(reserveResult)
    labelResult = generateShippingLabel(chargeResult)
    shipResult = shipMerchandise(labelResult)
    updateTransaction(transaction.id, shipResult)
  }
}

We’d get burned by idempotence a couple of times, put in some guardrails (as described previously), and eventually get to a point where it worked well enough. Because we were single-threaded, we didn’t have to worry about other processes trying to get in on the action.

Unfortunately, this almost immediately created another problem: some of these steps might take a non-trivial amount of time to complete, and if two or more people tried buying something at the same time, then sales might get blocked for a long time waiting for their turn in the database queue. If one transaction had a problem due to a bug or missing edge case (PO boxes, grr), it might halt the entire line. Beefing up the server or database might help us speed up the process marginally, but we always ended up needing to parallelize the processing infrastructure—both to increase throughput, and to prevent the system from getting stuck on a single failed transaction.

Scaling

Nowadays, everyone knows how to scale horizontally. Just turn on multi-threading, create an auto-scaling group, bump up the max container count, etc. Easy, right? Well, except that throwing multiple threads at a problem (whether in the same process or on different servers) can very easily result in two or more of them picking up the same transaction at the same time. Consider the following:

  • Thread A gets the next task (X) from the database
  • Thread B also gets the next task (X) from the database
  • A marks X as “in process”
  • B also marks X as “in process”
  • A checks to see if we’ve reserved inventory for X (not yet!)
  • B checks to see if we’ve reserved inventory for X (not yet!)
  • A reserves inventory for X
  • B reserves inventory for X
  • and so on

In order to scale, we need a way to guarantee that each task is picked up by exactly one thread. The most common solution to this problem is to use a message queue (or “event queue”, or “event bus”—most people use the terms interchangeably).1 The basic idea is that one process publishes events to a queue, and another process (potentially the same one, but it doesn’t matter) pops messages off the queue later. This has several key advantages:

  • You can depend on the message queue not to lose messages. Once a message has entered the queue, it won’t be removed until it’s been consumed (with the caveat that most message queues have a configurable maximum message lifetime, after which an unread message will be purged—know the default!)
  • Event publishing and consumption are handled asynchronously. If your product depends on service X sending an http request to service Y, then you might lose the request if service Y is down. With message queues, service X can emit the event, and service Y can consume it any time it’s convenient.
  • Messages are only considered “consumed” when they’ve been ACK’d. If Thread A pops a message from the queue and crashes before fully processing it, the message will stay on the queue and be served up again after a configurable amount of time. If the workflow’s been built to be idempotent, this should be fine.
  • Messages are thread-safe. So in the above example, if Thread A pops a message from the queue, Thread B can’t also get that message.

The new system doesn’t look so different:

function handleTransaction() {
  message = getNextTransactionFromMessageQueue()
  if (message != null) {
    reserveInventory(message)
    chargeCreditCard(message)
    generateShippingLabel(message)
    shipMerchandise(message)
    ackMessage(message)
  }
}

We get a message, then process the steps one at a time. We manage idempotence using the mechanisms described in the previous post, and we’re all good. Right?

Mostly? I mean, yes, this will work, but it will also be complex, and create extra work if a thread fails. Imagine, for instance, that you’ve successfully reserved inventory, charged the credit card, and created the shipping label, but the warehouse API is down and you fail to ship the merchandise. In this case, the shipMerchandise() step fails, you don’t ack the message, and it goes back into the queue. When another thread picks up the message again, it will have to figure out what’s been done already. It would be great if we could just skip to the right step.

Event-driven orchestration

There are two main ways to manage steps in a multi-step process. The first way—which I’ve described above—is to have one centralized controller that manages the full process. The code executes one step after another until the process is complete. This is easy to understand, but has several disadvantages.

  • Because the code is centralized, it needs to know everything about the process. All actions related to the process need to be controlled through the central orchestration service.
  • The central orchestrator might be owned by a different team, a different business unit, or a different company: it might be difficult or impossible to get permission to make changes.
  • All steps are tightly coupled, making it harder to update and maintain, and adding risk to any changes.
  • Different parts of the code might need different access, or have different compliance needs (e.g., DSS-PCI for the credit card handling). One step might be I/O bound, another might be memory or CPU bound, etc. You can fix this by having the central controller call other services, at the cost of additional complexity.

Alternatively, you can imagine a flow in which each step knows nothing about the other steps. Each has a specific job, kicked off by receipt of a specific type of message, and at the end of its job emits a new message saying that it’s done. This system might look something like this:

function reserveInventory() {
  message = getNextMessage("sales queue")
  // reserve the inventory, generate resultData
  emitMessage("reserve inventory queue", resultData)
  ackMessage(message)
}
function chargeCreditCard() {
  message = getNextMessage("reserve inventory queue")
  // charge the credit card, generate resultData
  emitMessage("credit card queue", resultData)
  ackMessage(message)
}
function generateShippingLabel() {
  message = getNextMessage("credit card queue")
  // generate shipping label, generate resultData
  emitMessage("shipping label queue", resultData)
  ackMessage(message)
}
function shipMerchandise() {
  message = getNextMessage("shipping label queue")
  // ship the merchandise, generate resultData
  emitMessage("merchandise shipped queue", resultData)
  ackMessage(message)
}

Note the following points:

  • Each function (or lambda, service, etc.) is listening for an event on a queue that represents a certain type of event that’s already happened.
  • When it receives notification of that event, it performs its own specific action, then emits a new event on its own queue with data generated during execution of its action.
  • Each function is loosely coupled. It doesn’t know anything about the other parts of the workflow, and once it’s done, it’s complete. Idempotence is limited to a much smaller scope.
  • Messages can be multiplexed—i.e., it’s possible to set up a mechanism that allows “consumers” to register to get all events from a single “publisher”. Each consumer gets its own queue, and the mechanism automatically copies each published event into each of those queues. This allows new actions to be added without touching the rest of the workflow. For example, after charging the credit card (and emitting an event to that effect) we might have one function that generates a shipping label and another that logs the payment—neither of which know about each other. If we wanted to add a third function to send an email to the user, then we could just create a new consumer, register it with the credit card event publisher, and it will automatically start getting messages. No need for high-risk changes to the original workflow.

Note: we always ACK the message after emitting a “completion” message. If we ACK’d first and emitted second, then a failure between those two lines would catastrophically halt the workflow.

Sounds great, so what’s the catch?

The downside to event-driven orchestration is that it’s hard to trace calls through the system. There’s no direct causal link from one action to another—service 1 emits an event that gets multiplexed to services 2, 3, and 4, each of which might emit their own messages, and on and on. Worse, unknown parts of the codebase might be listening, and there’s no explicitly defined way to know what the full DAG looks like—by design. Trying to backtrace a bug depends on trace IDs, logging, inspection of the infrastructure, and institutional knowledge.

By contrast, the centralized controller is easy to understand. You have a single location that orchestrates all actions, moving procedurally down a list of concrete steps that can be reasoned about comparatively easily.

Which is better? Like most things in life, it depends. Message queues make idempotence easier, decouple the code, add certainty to delivery, and are generally considered a great solution for all or part of a multi-step workflow. Centralized controllers are much easier to understand and debug, but have a lot of limitations. For simple workflows, they can be the natural choice.

The bottom line is that both should be part of your toolkit, and it should be natural for you to reach for one or the other when appropriate.

  1. You might be tempted to write your own message queue (how hard could it be?), but trust me when I tell you that 1) this isn’t where you want to invest your time, and 2) any hand-rolled solution will miss features and edge cases that existing battle-tested solutions have worked out over years of trial, error, and pain. ↩︎

All em-dashes were artisanally created using alt-shift-hyphen. No part of this post was written using AI.

Leave a Reply