.NET Core Paging and Threading with Headless (CMS)

When invoking RESTful API calls to retrieve data from a Content Management System (CMS), search index, or other system, a common pattern is that the service returns a limited number of records of a maximum fixed size, called a page. If you need additional records, you request the next page.

When the order of data is not important, you can create threads to do the paging, which can increase performance. You may also want to create a thread for each Content Type.

Contentstack APIs use a skip parameter to support paging (https://www.contentstack.com/docs/developers/apis/content-delivery-api/#skip).

The following C# code iterates all Entries for all Content Types in a Contentstack Stack. You should be able to run this against any Stack.

https://pastebin.com/4ah5Tw9v

This code and/or the SDK include one or more inappropriate names that result in  firstPage.Count being the count of Entries for the Content Type when a reader would assume that it is the count of Entries for the page. Sorry about that.

Any CMS vendor or other system with a “MACH” architecture (https://www.computerweekly.com/blog/Open-Source-Insider/MACH-Alliance-decomposing-monoliths-building-headless-open-cloud-microservices) likely provides similar functionality.

In addition to abstracting vendor internals, an implementation could provide options to disable threading, use a thread pool, join the threads to gather all Entries into one or more lists (for example, separated by Content Type), support passing delegates, and include additional features.

Using additional code and the following class to manage threads, my unrepeated, unexpected, and highly subjective experience is that thread pooling and Parallel.ForEach() can be convenient, but creating threads manually is significantly faster, even in an ASP.NET context.

https://pastebin.com/UhrXMj0Y

Unfortunately, I did not have need to implement the Metallica method.

Published by John West

I like Twizzlers and um

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: