Jan 4, 2022
Nov 2, 2020
Min Read

How to Catch Breaking Changes By Watching API Traffic

Kevin Ku
a dog biting a hose
Share This Article

As modern web apps shift to service-oriented architectures, it’s been getting more and more difficult to catch bugs before production. Because it’s hard to simulate production workloads beforehand, functionality that depends on service-service interactions often doesn’t get fully exercised until production. The result is that bugs don’t get uncovered until they are triggered by live user traffic.

As a developer who has also worked in devops, I understand the importance of finding and fixing bugs early in the development cycle. This is one of the reasons I’m excited to be working on change management at Akita. A few weeks ago, our team published a blog post talking about this at a high level. In this blog post, I’m going to go into the nuts and bolts. I’ll show a bug that’s hard to catch with source diffs, linters, or static analysis. Then I’ll show how to use Akita to catch this bug, describing the entire setup from start to finish. Finally, I’ll explain how Akita works under the hood.

Something I’m particularly proud of about Akita is that you can use us without needing to proxy or to make any code changes—and we’ve been working hard so you can set up everything you need to catch breaking changes in just minutes. Try out our private beta to see for yourself!

🕵🏻‍♀️ A particularly sneaky bug

You can find the Go source code for this example on GitHub here.

Let’s say you’re working on an API that returns information about users. To comply with regulations, you omit user phone numbers from the response to prevent callers of your API from storing this information in a scattered fashion that makes it hard to service deletion requests.

For example, your API might look like:

type User struct {
  ID string `json:”id”`

  // Dont return phone number for regulation reasons!
  Phone string `json:”-”`

func main() {
  http.HandleFunc("/users/json", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json")
    enc := json.NewEncoder(w)

Testing your endpoint shows that phone numbers are omitted as expected:

$ curl localhost:8080/users/json

One day, your colleague Aki adds a new version of the endpoint that returns YAML instead of JSON because they want to introduce some competition in the data serialization market.  A very reasonable implementation of the YAML endpoint produces a PR like this:

  http.HandleFunc("/users/yaml", func(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/x-yaml")
    enc := yaml.NewEncoder(w)
    defer enc.Close()

A code reviewer without insider knowledge is very likely to gloss over the fact that yaml.Encode(…) does something different than json.Encode. However, when someone actually uses this new endpoint, they get now users’ phone numbers in the response! 🙊

$ curl localhost:8080/users/yaml
- id: usr_295oDMFK8b1yS5dwlSTdgP
  phone: (123) 456-7890
- id: usr_6NiejyYEVpWfziUXJgovV6
  phone: (777) 888-9999

It turns out that Aki forgot to add the YAML-specific struct tags to the User struct to omit phone numbers from serialization—and now your phone numbers are getting sent somewhere they’re not supposed to go! You don’t realize this until your security team alerts you that they detected this issue in production. By this time, you have to not only roll back the change, but also scrub logs and send out an apology to your users.

This was a particularly subtle bug because it’s very easy to miss by just looking at source diffs This is also something that linters and static analyses won’t be particularly helpful with, unless you’ve configured them with a rule for this exact change.

Akita’s change management would catch this by detecting a new data format:

Pull Request highlighted.png

We’ll dig into this comment after we show you how to set this up for yourself. And this data leak is just one of many kinds of bugs that are hard to catch pre-production, but that you can catch if you have a better model of how your service is interacting with other services.

⚡️ Setting up Akita to fix this bug

To stop these bugs once and for all, we will show how to add Akita to your CI/CD pipeline, so that you get notified how every pull request changes our API. For our particular deployment, we are using CircleCI and GitHub to run Akita. You can easily modify the instructions for your CI/CD pipeline of choice.

The steps for getting up and running are:

  1. Create a new service in Akita, if you don’t have one already.
  2. Connect Akita to GitHub.
  3. Update your CircleCI configuration.
  4. Open a pull request with an API change.

The first thing we need to do is head over to the Akita Console and create a new Akita service. You can do that by clicking on the “New Service” button on the left-hand menu. If you already have a service, you can skip this step.

Once we’ve created and named our new service, we need to connect Akita to Github. To do this, simply click on the “Integrations” menu and then “Integrate” under GitHub. This will take us to GitHub where we can give Akita permission to watch our pull requests.


Now that Akita can post to our GitHub projects, we need to complete the loop by adding Akita to our CI Pipeline. This is relatively straightforward: we just need to add a step to start the Akita Client and another one that stops the client after our tests have completed.

The code to start our client is pretty simple:

  - run:
    name: Start Akita Client
    command: |
      docker run --rm -d \
        --env CI="${CI}" \
        --env CIRCLECI="${CIRCLECI}" \
        --env CIRCLE_BRANCH="${CIRCLE_BRANCH}" \
        --env CIRCLE_SHA1="${CIRCLE_SHA1}" \
        --env AKITA_API_KEY_ID=${AKITA_API_KEY_ID} \
        --network=host \
        --name akita \
        akitasoftware/cli:latest learn \
        --service [YOUR SERVICE NAME HERE] \
        --port [YOUR SERVICE PORT HERE]
    background: true

Then after you have run your integration test, simply add another step to stop the Akita Client, like this:

  - run:
    name: Stop Akita SuperLearn
    command: docker kill --signal=SIGINT akita

You can see a complete example of our CircleCI configuration here.

Once you have merged in the CircleCI changes, you can now test things out by making a quick change to your codebase, commit the change and open a new pull request. If everything went according to plan, once your pipeline completes Akita will leave a comment detailing how your API has changed.

Below is the comment for the example we introduced in the last section.

Pull Request highlighted.png

In this comment you can see:

  • Endpoints Added by this pull request. In this case, we added the YAML endpoint.
  • Endpoints Changed by this pull request. As we expected, we modified the JSON endpoint.
  • Data Formats Added by this pull request. This is where the new US phone number data type appears. Akita automatically detects precise data formats to make this as useful as possible.
  • The Baseline Specification that was used for this comparison. Akita gives you the flexibility to compare against any other test or production spec, so we also show what spec we diffed against.

🌎 Akita across test and production

Now that you’ve supercharged your pull requests with Akita, you’re probably saying to yourself “This is great for testing, but what if my tests don’t cover everything you’d see in production?”

Good news: Akita also allows you to compare test behavior against actual production behavior. To do this, you can use the same start and stop learning commands from your CI in your production environment to create a model of your production behavior. To use it as a baseline for pull requests comparisons, simply mark the production spec as stable in the Akita Console.

Talk to us if this is something you’re interested in!

🔩 Akita nuts and bolts

Under the hood, Akita works by building models of API behavior by watching API traffic.

What I just showed you works by:

  1. Reconstructing HTTP requests/responses from live packet captures. No code changes or proxies. And we only send metadata back to the Akita cloud!
  2. Building models of your API behavior. Expressed in the form of an API spec, annotated with data formats and eventually, implicit API contracts.
  3. Diffing on API behavior. Once we have the API models, it’s straightforward to diff.
YAML Spec.png

And this doesn’t just have to be traffic that already exists in your environments. More on running Akita with automatically generated traffic in future blog posts!

🐕 Try Akita for yourself!

Akita is currently available in a private beta. And I’m working hard every day to help you catch problematic changes more easily. Sign up here to try it out!

Share This Article

Join Our Private Beta now!

Thank you!

Your submission has been sent.
Oops! Something went wrong while submitting the form.