Elastic Search Pagination

Deep pagination is one of the top performance killers for your cluster. Deep pagination means to allow the user access to too many pages.

Use Search_after to get next page:

# Search after

# First request
GET products/_search
{
  "query": {
	"match": {
  	"name": "cars"
	}
  },
  "size": 10,
  "sort": [
	{
  	"_score": "desc"
	},
	{
  	"id.keyword": "asc"
	}
  ]
}

# Next request
GET products/_search
{
  "query": {
	"match": {
  	"name": "cars"
	}
  },
  "size": 10,
  "sort": [
	{
  	"_score": "desc"
	},
	{
  	"id.keyword": "asc"
	}
  ],
  "search_after": [
	0.2876821,
	"1"
  ]
}

Pagination and Search After are stateless. That means there is no guarantee that the order of the search results will be the same when users click back and forth between pages. That would likely be frustrating and confusing for your users.

If you need to make sure the search experience is the same over a certain amount of time, you need a stateful pagination technique.

Point in time API or use scroll api for older elasticsearch

Note that scroll api is high cost.

Stackoverflow: navigate terms aggregation in Elastic with very large number of buckets

Learn how to use scroll Elasticsearch aggregation

Required paging in aggregation

AWS OpenSearch 2.3

Describe the issue:

After filtering the data from index count of doc is 220000. There are distinct 99000 values when I apply aggregation on that. but Query return only 65535 rows as max.

Solution:

You can do this with Composite Aggregation:

  "size": 0,
  "aggs": {
    "my_buckets": {
      "composite": {
        "size": 1000,
        "sources": [
          {
            "data": {
              "terms": { "field": "Hits.Recipient.keyword" }
            }
          }
        ]
      }
    }
  }
You will get non-null “after_key” in response, so you have to use that in “after” param in next call:

  "size": 0,
  "aggs": {
    "my_buckets": {
      "composite": {
        "size": 1000,
        "after": { ... },
        "sources": [
          {
            "data": {
              "terms": { "field": "Hits.Recipient.keyword" }
            }
          }
        ]
      }
    }
  }

More information on composite aggregation