Troubleshooting Webhooks

This blog post contains information about troubleshooting webhooks, which I have been exploring specifically with the Contentstack SaaS headless CMS. Webhooks are just HTTP requests, so troubleshooting them is not unlike troubleshooting any HTTP request.

• Webhook – Wikipedia
• About Webhooks (contentstack.com)

For me, problems that appear significant often have trivial solutions, often involving simple user errors. Webhooks are no exception. Unless you prefer to inspect network traffic, attach a debugger, analyze logs, re-instrument your code, fire up Wireshark, and so forth, always consider the most likely causes first and use the easiest troubleshooting techniques first.

• Is the webhook properly configured in the system that raises it, specifically the:
  • Criteria that trigger the webhook
  • Network protocol (HTTPS)
  • DNS entry
  • URL path
• Is the webhook enabled?
• Are you sure that the event that triggers the webhook is occurring?
• Assuming HTTPS, are all required certificates in place?
• Can the system that invokes the webhook resolve the DNS entry in the URL?
• Can the system that invokes the webhook reach the IP address associated with that DNS entry, or is there a firewall between the two systems?
• Does the webhook meet security requirements to invoke the webhook, specifically identification of the source repository as well as authentication/authorization?
• Is the solution that handles webhooks running and on the correct protocol (HTTPS), with the correct DNS entry, on the correct port, and responding to the correct URL paths?
• Is the webhook receiver configured to respond to the HTTP protocol (GET, PUT, POST, etc.) used by the webhook?
• If there are separate test, development, production, and other solutions, is the issue specific to one environment or its configuration, such as a mixture of test and production configuration?
• If the solution depends on something like ngrok to get webhooks through firewalls, is it running with the correct parameters?

Network issues can delay and block requests. Handlers do not necessarily receive webhooks in the order that the calling system invokes them. Calling systems may invoke webhooks repeatedly in the case of errors.

Sometimes it can help to try the webhook from a browser or preferably a tool such as Postman.

• Postman | The Collaboration Platform for API Development

For webhook listeners inside firewalls and without DNS entries, I use ngrok. I always use HTTPS except locally between ngrok and Azure functions running that cannot auto-generate certificates but run on the same machine anyway. Be sure to specify the appropriate protocol, host, and port on the command line to ngrok. Ngrok works as a proxy, which can be useful in troubleshooting. If ngrok receives the request, then the network is not the issue; something is not calling the webhook handler or it is not responding.

Additional issues occur if the webhook does not include data required or in the format expected by the handler. Beware of logic in the webhook handler that can trigger the webhook, which would invoke the handler and that would trigger the webhook infinitely.

The following list of causes for HTTP status codes comes from API Design Patterns | RESTful API Web Services Design | REST API Design (stoplight.io):

• 200: Successful request, often a GET
• 201: Successful request after a create, usually a POST
• 204: Successful request with no content returned, usually a PUT or PATCH
• 301: Permanently redirect to another endpoint
• 400: Bad request (client should modify the request)
• 401: Unauthorized, credentials not recognized
• 403: Forbidden, credentials accepted but don’t have permission
• 404: Not found, the resource does not exist
• 410: Gone, the resource previously existed but does not now
• 429: Too many requests, used for rate limiting and should include retry headers
• 500: Server error, generic and worth looking at other 500-level errors instead
• 503: Service unavailable, another where retry headers are useful

See also:

• Process Headless CMS Webhooks with Serverless Azure Functions – Deliverystack.net
• Part II: Prototype Contentstack Update Webhook Handler .NET Azure Function to Set Field Values for Validation – Deliverystack.net
• Possible Causes for Contentstack HTTP Errors – Deliverystack.net

That is as far as I have gotten. I do not want to search and arbitrarily link to resources, but if you have specific knowledge or know of an excellent resource and especially if you have experience troubleshooting webhooks, please comment on this blog post.