Webhooks – A Conceptual Deep Dive

In simple terms, webhooks are user-defined HTTP callbacks (or small code snippets linked to a web application) that get triggered when specific events take place on an external website or service. They’re especially useful when you’re building notification functions and event-driven responses in applications.

For example, your ecommerce application can use webhooks to ensure it’s notified when a third-party payment gateway receives payment from a customer. Your newsletter campaign might use webhooks to inform your marketeers of any follow-on email activity, and then trigger other functions as necessary.

You as the application provider register your callback API to the external servers that will notify you of an event occurrence. Whenever the trigger event occurs, the webhook in question collects the data from the external system, and sends it to your application, via a URL you specify, in the form of an HTTP POST request.

Webhooks encapsulate the realtime communication channel between two or more applications. Webhooks are highly customizable and can be tailor-made to serve various business needs. The diagram below demonstrates how they work.

Note the important difference between traditional APIs and webhooks: API calls work on a request-based on output mechanism, while webhooks work on events-based output mechanisms. In other words, where APIs rely on user input, webhooks are automated calls from your application to a server.

Webhooks are the HTTP APIs built by the vendors and registered at the company providing the service (service provider) with following the basic standards created by service providers.

Webhooks are important when it comes to establishing an event-based communication channel between your application and your service provider’s application infrastructure. Webhooks keep key data in sync between two remote applications.

As a result, webhooks form the basis of the infrastructure that supports many online activities we take for granted. Take payment notifications. If an e-commerce retailer uses a third-party payment gateway, the ‘payment’ event happens in an environment external to the retailer’s website. Through webhooks, the payment gateway will call the retailer’s register API to provide the payment data as soon as it’s available. The retailer’s server then accepts this data, enabling it to update its database and user-facing screens.

A number of high-profile service providers deliver key capabilities through the use of webhooks, which you can leverage. Let’s look at a couple of examples.

MailChimp is one of many platforms that relies on webhooks to support everyday tasks, such as subscribing to and unsubscribing from email lists, as well as user profile changes. This is useful if you’re using MailChimp as the mechanism to sign website users up to your newsletter, but you also want to pass the data to your CRM system.

SendGrid also offers webhooks for informing users when transactional emails are delivered, opened or unintentionally not delivered to the recipient.

Stripe offers several events that can be used as webhooks within your application. Webhooks provide notifications of successful or failed charges.

GitHub is a great example of a collaboration platform using webhooks to facilitate communication between users. Github uses webhooks for events related to the repository. Actions including code push, repository created and repository deleted all rely on webhooks, which also configure the associated continuous integration/continuous delivery (CI/CD) processes.

Lead-generation platform HubSpot provides webhooks through the Ably platform for notifying users, in real time, about specific events their customers have opted to hear about. This could be company-creation, deal-creation, contact information changes, replies to messages, and so on. Realtime webhooks infrastructure means users can listen to selected notifications and store these in their system or dashboard, to keep their system synchronized with HubSpot’s account changes. Here, realtime is an important factor, because it ensures a delicate interplay of microservices remains in sync.

At a very high level, the process for implementing webhooks is as follows:

• Create a URL on your server that can accept and process a POST request.

• Provide this URL to a webhook provider which then sends the POST request and performs relevant actions, namely making a POST request to specified URLs to update on any changes.

• Your server processes this request and sends a POST request back to the webhook provider, to notify them the requested operation has been completed (or not as the case may be).

With webhooks a basic part of online service provision, the interconnectedness of services makes good webhooks architecture an important part of webhooks’ more general usage – and this is where life can get complicated. If you want your app to scale, webhooks architectures must also be scalable, so that they can handle millions of event occurrences and activate webhooks related to other vendors.

Downtime of your applications and/or infrastructure – both planned and unplanned – can pose a problem when using webhooks at scale. If your systems are down for whatever reason, webhooks won’t be able to deliver event information to your system.

To overcome this, you can use an event messaging queue, such as the open source RabbitMQ, or Amazon’s Simple Queue Service (SQS). Both are specifically designed for this type of use case.

When implemented, any events triggered on the provider’s side will be stored in the event messaging queue. In turn, the queue will attempt to send details of the event to your application. If your systems are live, they will return a ‘success’ message to the event messaging queue, which then removes the event in question. If your systems are down, the queue stores up the events, to be re-sent once your systems are up again. You can configure how the queue’s time-out and retry mechanism works.

This approach ensures no webhook events are lost if your systems are down. However, keep in mind that using external queuing systems can add complexity in other areas.

As with many infrastructure issues the choice whether to engineer solutions yourself or go down the paid proprietary software route depends on your use case and the engineering resources you have available. For example, when it comes to webhooks’ functionality in realtime architecture, issues include implementing message ordering and delivery guarantees, dealing with bursts in usage, idempotent publishing and more.

You can read more about engineering event-driven systems on the Ably Engineering blog.

Ably provides cloud infrastructure and APIs to help developers simplify complex realtime engineering. We make it easy to power and scale realtime features in apps, or distribute data streams to third-party developers as realtime APIs.

A fundamental part of our platform is providing ready-built third-party integrations and data processing pipelines, many using webhooks through our Integrations service. This provides the flexibility to use the services and compute you’re already working with while Ably handles the complexity and scale of doing so. Queue and consume high volumes of realtime messages in bulk, transform or filter messages by executing functions on them, pass messages on to other systems for onward processing or have 3rd parties publishing messages into Ably at scale.

With Incoming Webhooks, one of our latest new features, you can solve all the challenges of scaling realtime data flows across multiple applications using webhooks. Just configure external services and let Ably handle the buffering and load balancing as webhook traffic scales.

We’ve solved many of the issues of deploying webhooks at scale including ordering, rate limiting, and much more.

You can create a free Ably account to get started with realtime messaging and webhook integrations.

• RESThooks

• Webhooks vs WebSub - A comparisonAdd Webhooks to Your API the Right Way - Zapier

• Incoming Webhooks: scale data flow to connected applications without a hitchA Scalable, Reliable Webhook Dispatcher Powered by Kafka