Paging Through HTTP API Results with .NET

This blog post demonstrates a simple C# coding pattern for paging through JSON results from HTTP APIs. For network considerations and to prevent unbounded queries, HTTP APIs often return records in batches of a maximum size. To process additional data, you can request additional pages. You could use this coding pattern on any platform to page through results from any HTTP API that supports paging.

Paging through records requires knowing or specifying the size of pages. The first page of results can contain the total number of records in all pages of that size, which indicates the number of pages to retrieve.

This example retrieves metadata about folders in the Contentstack SaaS headless CMS using this API:

URLs, query string parameters and other API semantics vary by provider.

int processed = 0; // count folders processed 
int limit = 100; // return data in pages of 100 folders (Contentstack limit)
int totalFolders = -1; // total number of folders that exist

// page through 100 folders at a time
do
{
    string url = "https://api.contentstack.io/v3/assets?"
        + "include_folders=true&query={\"is_dir\":true}&limit="
        + limit
        + "&skip="
        + processed;

    // get the count of all folders on the first request
    if (totalFolders < 0)
    {
        url += "&include_count=true";
    }

    // get a page of folder data
    JObject queryResult = JObject.Parse(new
        HttpRequestProcessor().ProcessRequest(url).Result);

    // set the count of all folders on the first request
    if (totalFolders < 0)
    {
        totalFolders = Int32.Parse(queryResult["count"].ToString());
    }

//TODO: process queryResult

    processed += limit; // get from queryResult if accurate count is needed
}
while (processed < totalFolders);

This depends on something like:

A more complicated example uses threading to improve performance, especially for large data volumes:

2 thoughts on “Paging Through HTTP API Results with .NET

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: