Process Headless CMS Webhooks with Serverless Azure Functions

This blog post demonstrates a way to implement (serverless) Azure functions that use ASP.NET Core and .NET 5 to process webhooks generated by headless Content Management Systems, including how to run functions locally and use ngrok to invoke those local functions from cloud environments such as SaaS headless CMS. Azure functions provide ASP.NET processing in a serverless context, which can improve performance, efficiency, and scalability while reducing administration and hosting costs. See Serverless Computing Really Means Less Server [Customer Responsibility] – Deliverystack.net for additional information about serverless computing. The example uses webhooks from the Contentstack SaaS headless CMS to index data with Algolia.

Headless content management systems raise webhooks to signal external systems when events occur in the CMS. Webhooks pass JSON to HTTPS service endpoints in external systems. For example, headless content management systems can pass data to URLs after publishing changes. Solutions can use publishing webhooks to update indexes, clear caches, and otherwise.

Azure: Develop Azure Functions using Visual Studio | MSDEVBUILD explains how to create Azure functions and test them on a local system, without deploying the solution to Azure.

First, I implemented a simple proof of concept that simply logs webhook data. Using Visual Studio 2019 Community Edition, create a new project using the Azure Functions template and select HTTP trigger.

using System.IO;
using Microsoft.AspNetCore.Mvc;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;

namespace azfwhook
{
    public static class LogWebhook
    {
        [FunctionName("LogWebhook")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            log.LogInformation(requestBody);
            return new OkObjectResult("OK");
        }
  }
}

Running the project shows that the function is available at the following URL:

http://localhost:7071/api/LogWebhook

You can test this with a browser or postman, but a headless CMS in the cloud may not have access to your local system. You can use a solution such as ngrok to provide a DNS entry and an HTTP forwarding mechanism. I run ngrok to forward HTTP and HTTPS traffic for jw.ngrok.io to http://localhost:7071 (notice that this is *NOT* HTTPS).

ngrok.exe http -subdomain=jw http://localhost:7071

Here is some of the output, first from calling the localhost URL from a browser (empty body), then after configuring a webhook and publishing an entry from Contentstack (JSON):

Here is the ngrok output showing a single forwarded request:

Intercepting CMS webhooks to invoke search webhooks for each published record is inefficient. See What and How to Index for Search with Headless CMS – Deliverystack.net for additional information. You could use webhooks to put data on a message bus, and use something else to trigger indexing. Maybe you could use webhooks after Contenstack releases complete to trigger indexing or rebuilding of indexes. Note that you should secure anything that accepts webhooks to ensure that only authorized systems invoke them.

In addition to updating existing URLs, all solutions should include processes to rebuild indexes. Additionally, indexing logic should handle deletion as well as addition and updates. Anyway, here is a proof of concept that tries to index new and updated entries. This uses the Algolia SDK for indexing but uses the raw JSON from the webhook rather than any Contentstack SDK.

using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json.Linq;

using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Algolia.Search.Clients;

namespace azfwhook
{
    public static class IndexWebhook
    {
        public class SearchIndexEntry
        {
            public string ObjectID { get; set; }
            public string Url { get; set; }
            public string Json { get; set; }
            public string ContentType { get; set; }
        }

        //TODO: [HttpPost] 
        //TODO: SECURE THIS ENDPOINT!!!!!!!
        [FunctionName("IndexWebhook")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            string body = await new StreamReader(req.Body).ReadToEndAsync();
            log.LogInformation(body);

            if (String.IsNullOrWhiteSpace(body))
            {
                return new StatusCodeResult((int)HttpStatusCode.BadRequest);
            }

            // convert JSON string to object
            JObject payloadObject = JObject.Parse(body);

            // webhook event (action) and status
            string action = payloadObject["event"].ToString();
            string status = payloadObject["data"]["status"].ToString();

            if (status != "success" || action != "publish")
            {
                return new StatusCodeResult((int)HttpStatusCode.ExpectationFailed);
            }

            // JSON representation of content entry extracted from webhook payload 
            JToken entry = payloadObject["data"]["entry"];

            // object to pass to search index
            SearchIndexEntry searchEntry = new SearchIndexEntry()
            {
                ObjectID = entry["uid"].ToString(),
                Url = entry["url"].ToString(),
                Json = entry.ToString(),
                ContentType = payloadObject["data"]["content_type"]["uid"].ToString(),
            };

            SearchClient client = new SearchClient("FKLVBEGHYF", "9e1552b82befd484668ee9999e5c8ded");
            SearchIndex index = client.InitIndex("urls");
            List<SearchIndexEntry> entries = new List<SearchIndexEntry>();
            entries.Add(searchEntry);
            index.SaveObjects(entries);
            return new StatusCodeResult((int)HttpStatusCode.OK);
        }
    }
}

Here is the Contentstack webhook configuration:

You can also launch a container for the Azure function from the command line. Change to the directory that contains the .csproj file and run func.exe with some parameters. In a Windows Subsystem for Linux shell:

/mnt/c/Users/ms/AppData/Local/AzureFunctionsTools/Releases/3.23.0/cli_x64/func.exe host start --port 7071 --pause-on-error