When invoking RESTful API calls to retrieve data from a Content Management System (CMS), search index, or other system, a common pattern is that the service returns a limited number of records of a maximum fixed size, called a page. If you need additional records, you request the next page.
When the order of data is not important, you can create threads to do the paging, which can increase performance. You may also want to create a thread for each Content Type.
Contentstack APIs use a skip parameter to support paging (https://www.contentstack.com/docs/developers/apis/content-delivery-api/#skip).
The following C# code iterates all Entries for all Content Types in a Contentstack Stack. You should be able to run this against any Stack.
This code and/or the SDK include one or more inappropriate names that result in firstPage.Count being the count of Entries for the Content Type when a reader would assume that it is the count of Entries for the page. Sorry about that.
Any CMS vendor or other system with a “MACH” architecture (https://www.computerweekly.com/blog/Open-Source-Insider/MACH-Alliance-decomposing-monoliths-building-headless-open-cloud-microservices) likely provides similar functionality.
In addition to abstracting vendor internals, an implementation could provide options to disable threading, use a thread pool, join the threads to gather all Entries into one or more lists (for example, separated by Content Type), support passing delegates, and include additional features.
Using additional code and the following class to manage threads, my unrepeated, unexpected, and highly subjective experience is that thread pooling and Parallel.ForEach() can be convenient, but creating threads manually is significantly faster, even in an ASP.NET context.
Unfortunately, I did not have need to implement the Metallica method.
Deliverystack.net 
3 thoughts on “.NET Core Paging and Threading with Headless (CMS)”