Handling Pagination in Web Queries

API pagination is a way for REST APIs to return large data sets in manageable pieces. When an endpoint has a lot of data, it divides the results into pages and returns one page per request. This article shows how a Web Query's custom response parser can push each page's data and then request the next page.

Handling pagination in API responses

APIs handle pagination in different ways, but the idea is the same: fetch the data in chunks, and keep requesting more while more pages exist. How you tell whether there is another page depends on how the API describes its pages in the response. Whatever the format, the steps are the same:

1. Push the data: use the push() function to add the response's records to your Dataset.
2. Check for additional pages: after pushing, check whether another page of data exists.
3. Make the next request: using whatever the response says about the next page, build a new request config for it.
4. Send the next request: use the next() function to send the new request config. See Using the next() function for more.

The examples below all assume the API endpoint accepts a page parameter that picks which page of results to return.

Basic pagination logic

Most APIs wrap their data in a JSON object, often with a meta property that carries the pagination details. For instance, this example response has a next_page property inside meta that says whether another page is available:

{
    "config": {
        // the current request config
    },
    "data": {
        "data": [
            {...} // the usable data
        ],
        "meta": {
            "next_page": 2
        }
    }
}

The response parser reaches this object through the response variable. The config property holds the settings of the most recent request, and the data property holds what the API returned. Here, the records are in response.data.data, and the next_page property inside response.data.meta says that page 2 is available. For this kind of response, the custom response parser might look like this:

push(response.data.data);

if (response.data.meta.next_page) {
    var nextConfig = _.assign({}, response.config);
    nextConfig.params.page = response.data.meta.next_page;
    next(nextConfig);
}

Push the array of records itself (response.data.data), not the object that wraps it. push() adds one record per array element, but given an object it adds that whole object as a single record.

Alternative pagination approaches

Some APIs don't return a next_page value. Instead they describe the pages in a more traditional way, with the current page, the total number of pages, and the number of results per page:

{
    "config": {
        // the current request config
    },
    "data": {
        "data": [
            {...} // the usable data
        ],
        "meta": {
            "page": 1,
            "per_page": 6,
            "total": 12,
            "total_pages": 2
        }
    }
}

In this case, the parser checks whether the current page is lower than the total number of pages:

push(response.data.data);

if (response.data.meta.page < response.data.meta.total_pages) {
    var nextConfig = _.assign({}, response.config);
    nextConfig.params.page = response.data.meta.page + 1; // Increment page number
    next(nextConfig);
}

Pagination via URL

Some APIs give the complete URL of the next page in the next_page property:

{
    "config": {
        // the current request config
    },
    "data": {
        "data": [
            {...} // the usable data
        ],
        "meta": {
            "next_page": "https://example.url.com/api/v2/example.json?page=2"
        }
    }
}

In that case, instead of reading a page number, take the page parameter out of the URL for the next request:

push(response.data.data);

if (response.data.meta.next_page) {
    var nextConfig = _.assign({}, response.config);
    var nextPageParam = response.data.meta.next_page.split("?")[1]; // Extract the query string from the URL, "page=2"
    var nextPage = nextPageParam.split("=")[1];
    nextConfig.params.page = nextPage;
    next(nextConfig);
}

This simple parse assumes page is the only parameter in the URL. If the URL carries others, find the page= pair among them before reading its value.