Paging Through HTTP API Results with .NET

This blog post demonstrates a simple C# coding pattern for paging through JSON results from HTTP APIs. For network considerations and to prevent unbounded queries, HTTP APIs often return records in batches of a maximum size. To process additional data, you can request additional pages. You could use this coding pattern on any platform to page through results from any HTTP API that supports paging.

Paging through records requires knowing or specifying the size of pages. The first page of results can contain the total number of records in all pages of that size, which indicates the number of pages to retrieve.

This example retrieves metadata about folders in the Contentstack SaaS headless CMS using this API:

• https://www.contentstack.com/docs/developers/apis/content-management-api/#get-all-assets

URLs, query string parameters and other API semantics vary by provider.

int processed = 0; // count folders processed 
int limit = 100; // return data in pages of 100 folders (Contentstack limit)
int totalFolders = -1; // total number of folders that exist

// page through 100 folders at a time
do
{
    string url = "https://api.contentstack.io/v3/assets?"
        + "include_folders=true&query={\"is_dir\":true}&limit="
        + limit
        + "&skip="
        + processed;

    // get the count of all folders on the first request
    if (totalFolders < 0)
    {
        url += "&include_count=true";
    }

    // get a page of folder data
    JObject queryResult = JObject.Parse(new
        HttpRequestProcessor().ProcessRequest(url).Result);

    // set the count of all folders on the first request
    if (totalFolders < 0)
    {
        totalFolders = Int32.Parse(queryResult["count"].ToString());
    }

//TODO: process queryResult

    processed += limit; // get from queryResult if accurate count is needed
}
while (processed < totalFolders);

This depends on something like:

• Prototype .NET Class to Invoke [Contentstack] [SaaS] [Headless] [Content Management] HTTPS JSON Service APIs – Deliverystack.net

A more complicated example uses threading to improve performance, especially for large data volumes:

• .NET Core Paging and Threading with Headless (CMS) – Deliverystack.net