Pagination (#158)
APIs often need to provide collections of data. Collections can be arbitrarily sized, and usualy grow over time, increasing lookup time as well as the size of the responses. Therefore, it is important that collections be paginated.
There are two popular pagination styles: offset-limit and cursor pagination. Refer to this article for definitions and comparison.
Guidance
All collection endpoints MUST be paginated from the outset, since adding pagination is a backward-incompatible change.
Collections in Kong APIs generally fall into one of the following categories:
- customer-managed collections like services, versions, runtime groups
- system-generated, unbounded collections like audit log entries
- second-order human-created entities like consumers, developers, and applications
APIs at Kong should prefer cursor pagination. In particular, system-generated, unbounded collections must use cursor pagination. Customer-managed and second-order collections like consumers should use cursor pagination, since the size of these collections will vary widely across Kong customers. An outline of offset-limit pagination is offered here for exceptional cases where cursor pagination doesn't meet API client needs.
Cursor
APIs that implement cursor pagination should follow the guidelines below, which are inspired by the JSON-API Pagination profile.
Sorting Requirement
Pagination only applies to an ordered list of results. This order must not change between requests unless the underlying data changes, to ensure that results don’t arbitrarily move between pages.
If the client’s paginated request includes a ?sort query parameter that only partially orders the results, the server MUST apply additional sorting constraints — consistent with the client-requested ones — to produce a unique ordering, if it wishes to support pagination of that data.
Similarly, when the collection being paginated has no natural or client-requested order, the server MUST assign an order if it wishes to support pagination.
The server MAY reject pagination requests if the client has requested that the results be sorted in a way that server cannot efficiently paginate. In that case, the server MUST reject the request with a 400 response.
Cursor Values
Cursors should be opaque, so that the implementation can change without breaking clients who may write business logic against cursor values.
Guidance
It is RECOMMENDED for servers to calculate cursor values using the following encoding strategy:
cursor = base64url_encode(xor_cipher(raw_cursor_value))
The contents of raw_cursor_value is an implementation detail left up to the API server. An example could be a stringified JSON object with properties that provide a pointer to the recordset, for example:
{
"id": "d8d16afe-082b-4851-a008-a6c0fd928519",
"created_at": "2023-05-18T00:00:00Z",
}
Query Parameters
Endpoints that support cursor pagination accept a page query param. page is an object with the following optional properties:
size - the number of results that the client would like to see in the response. If not provided, the server must choose a default page size.
after and before - the pages after or before a given cursor. Both are optional. They are also mutually exclusive, meaning that only one of them can be used in a single API request. Both parameters cannot be present simultaneously (see Range Pagination).
See https://jsonapi.org/profiles/ethanresnick/cursor-pagination/#auto-id-query-parameters for more details about query parameters.
Range Pagination
A range pagination request is defined by an API request that includes both page[after] and page[before] query parameters.
Kong APIs do not support range pagination requests and must return a 400 response indicating that range pagination is not supported when one is received.
Response Body
The response body of a paginated endpoint should include the following attributes:
data - the subset of records being returned
meta.page - an object that includes page info such as
size - the requested page sizenext - URI to the next page (may be null)previous - URI to the previous page (may be null)first - URI to the first page (optional)last - URI to the last page (optional)
first and last properties are optional properties. Which means if the API does support those fields,
first and last SHOULD always be defined.
For clarity, the size response property is always a copy of the value supplied via query parameter. This still applies when:
size does not equal the length of the data array
It is RECOMMENDED that servers include first and last links when these are inexpensive to compute.
Servers MUST include previous and next links for each instance of paginated data in a response when data for those links is available.
Servers SHOULD set these links to null when it can inexpensively determine that the current response is for the first or last page respectively.
However, if the server can’t easily determine whether there are prior results (when computing the previous link) or subsequent results (when computing the next link), it MAY use a URI in these links that returns an empty array as its paginated data.
URIs MUST include all query parameters that were sent in the API request with the exception of:
page[next]page[previous]page[first]page[last]
since these parameters are calculated per-request for pagination by the server.
Note that the order of the query parameters in the API request and the corresponding response links can vary.
Examples
Suppose we have a collection of critters. Note: the meta attribute of each record is optional.
[
{ "name": "cats", "id": "uuid-1", "meta": {"page": {"cursor" : "dXVpZC0x"} } },
{ "name": "dogs", "id": "uuid-5", "meta": {"page": {"cursor" : "dXVpZC01"} } },
{ "name": "ants", "id": "uuid-7", "meta": {"page": {"cursor" : "dXVpZC03"} } },
{ "name": "emus", "id": "uuid-8", "meta": {"page": {"cursor" : "dXVpZC04"} } },
{ "name": "bats", "id": "uuid-9", "meta": {"page": {"cursor" : "dXVpZC05"} } }
]
... with a default page size of 2 and default sort by id. A call to /critters would yield
{
"data": [
{ "name": "cats", "id": "uuid-1" },
{ "name": "dogs", "id": "uuid-5" }
],
"meta": {
"page": {
"size": 2,
"previous": null,
"next": "/critters?page[after]=dXVpZC01"
}
}
}
To change the page size, pass the size in a page parameter: /critters?page[size]=4:
{
"data": [
{ "name": "cats", "id": "uuid-1" },
{ "name": "dogs", "id": "uuid-5" }
{ "name": "ants", "id": "uuid-7" },
{ "name": "emus", "id": "uuid-8" }
],
"meta": {
"page": {
"size": 4,
"previous": null,
"next": "/critters?page[after]=dXVpZC04&page[size]=4"
}
}
}
To fetch the next page, use the value in page.next, which is /critters?page[after]=dXVpZC04&page[size]=4:
{
"data": [
{ "name": "bats", "id": "uuid-9" },
],
"meta": {
"page": {
"size": 4,
"previous": "/critters?page[before]=dXVpZC05&page[size]=4",
"next": null
}
}
}
Query parameters provided in the request must be persisted in the URI links returned in the response. A call to /critters?foo=1&sort=id will result in:
{
"data": [
{ "name": "cats", "id": "uuid-1" },
{ "name": "dogs", "id": "uuid-5" }
],
"meta": {
"page": {
"size": 2,
"previous": null,
"next": "/critters?page[after]=dXVpZC01&foo=1&sort=id"
}
}
}
Limit-Offset
Collections that have low cardinality may implement limit-offset pagination. When they do, the API should support the following properties in the page object of the request, and echo them back in the page meta of the response:
size - the requested page sizenumber - the current page number (optional, the default number is 1)
For clarity, these response properties are always a copy of the values supplied via query parameter. This still applies when:
size does not equal the length of the data arraynumber is greater than the total number of available pages
Examples
Given the same dataset as above, to fetch the second page of results, use /critters?page[number]=2:
{
"data": [
{ "name": "ants", "id": "uuid-7" },
{ "name": "emus", "id": "uuid-8" },
],
"meta": {
"page": {
"number": 2,
"size": 2,
"total": 5
}
}
}
A request that extends beyond the end of the dataset, such as /critters?page[number]=5&page[size]=10:
{
"data": [
],
"meta": {
"page": {
"number": 5,
"size": 10,
"total": 5
}
}
}
Total and Estimated Total
Regardless of the pagination strategy selected, the response may include either a total or estimated_total in the response.
References