Why webhooks aren’t enough for rapid deployment

At mParticle our mission is straightforward: make it really easy for leading apps and app services to connect. Therefore, partner integrations are central to the mParticle platform, and we’re continually optimizing for quality and maintainability of those integrations. Put simply – our partners’ success is our success.

In the early days, all integrations were built in-house, which afforded us the ability to quickly evolve and expand our product, and avoided the challenges related to supporting a publicly available partner API. But, three years and 80+ integrations later, we opened up integration development to our partners by way of our Firehose API last December. Our goal is to make it fast and easy for any SaaS platform to push data to or pull data from our platform.

It’s common across the web to leverage simple webhooks for inter-service communication – they can allow one system to subscribe for changes in another, and that’s incredibly useful. Rather than needing to poll a system for updates, listeners get updates pushed to them if and when they occur. In the case of mParticle, a partner could receive a callback whenever new data to which they’re subscribed arrives in the platform. Though, whereas webhooks seem simple to set up, we found their simplicity would really mean pushing all of the real work onto our partners.

Firehose provides actual models to work with our schema, an SDK encompassing typical integration use-cases, and is designed specifically for rapid deployment:

Publishing our schema

All partners process data from our platform, which originates as JSON, and translate it into their own structure. To support this, one approach would be to document our API on our doc site and require partners to read and integrate it manually, and update their implementations as we update our API. Instead, we provide actual Java models that are representative of our schema. This means hands-on documentation, compile-time safety, and a clearer way to communicate API updates by way of a versioned SDK.

Bi-directional integration

With versioned models, our partners have a reliable way to receive and process data, but mParticle integrations need to be bi-directional. For example, when an audience list is created in the mParticle platform, we’ll send that audience to an audience API, such as Facebook Custom Audiences. We then verify that the audience was sent to the partner successfully and continually sync the mParticle audience and the matching partner audience. This sort of tight integration provides a more seamless experience for our customers. It also necessitates that we listen and parse dynamic responses from our partners, which is not a typical design pattern in the context of webhooks.

So, we resolved to move beyond webhooks, and make our Firehose SDK mindful of the types of requests and required responses involved in a partner integration. We defined additional models for the different types of responses for each integration touchpoint, as well as an easy-to-extend framework that exposes hooks for when different types of data are received. This goes as far as having partners programmatically describe the capabilities and requirements of their integration, as well as provide descriptions and even branding – all through our API.

Zero infrastructure

Finally, we leverage AWS’s Lambda service to avoid the need for partners to create new infrastructure for their integrations. Rather than deploying new API endpoints and servers to process mParticle data, our partners just write a “lambda function” that implements our SDK. They translate our models into theirs and send data to their existing infrastructure – AWS handles deployment and scaling automatically. A quickstart project with unit tests and a docs tutorial bring it all together.

Firehose – the API, the SDK, and AWS Lambda – is allowing us to add quality integrations at a record pace, and has been crucial in forming a closer relationship with each partner as they’re given ownership over their integrations.