Elasticsearch

01The idea: an inverted index

A book index lists words with the pages they appear on. An inverted index does the same for documents: for every term, it stores the list of document IDs that contain it, plus positions and frequencies. To search “red running shoes”, Elasticsearch looks up three short lists and intersects or merges them, instead of reading every document. Relevance is calculated with BM25 by default: a term counts more if it is rare across the index and frequent in the document, and less in very long fields. Elasticsearch is near real-time: new documents become searchable after a refresh, which happens every second by default.

• Index: a logical collection of documents with one mapping, like a table.
• Document: a JSON object, the unit you index and search.
• Term: a normalised token produced by the analyzer, for example “running” becomes “run”.

02Mappings and analyzers

A mapping defines the type of each field. The most important distinction is text versus keyword. A text field is analyzed: split into tokens, lowercased, maybe stemmed, so it is good for full-text search but not for sorting or exact filters. A keyword field is stored as one exact value, good for filters, sorting and aggregations (brand, status, SKU). Often you map a field both ways with a sub-field. An analyzer is a pipeline: character filters, a tokenizer and token filters (lowercase, stop words, stemming, synonyms). Choose language analyzers (english, german, russian…) per field. Define mappings explicitly; dynamic mapping guesses types and often guesses wrong.

1PUT /products
2{
3  "settings": { "number_of_shards": 1, "number_of_replicas": 1 },
4  "mappings": {
5    "dynamic": "strict",
6    "properties": {
7      "name":       { "type": "text", "analyzer": "english",
8                      "fields": { "raw": { "type": "keyword" } } },
9      "brand":      { "type": "keyword" },
10      "price":      { "type": "scaled_float", "scaling_factor": 100 },
11      "in_stock":   { "type": "boolean" },
12      "created_at": { "type": "date" }
13    }
14  }
15}

03Querying: query context and filter context

Search uses the Query DSL, a JSON language. The bool query combines clauses: must and should contribute to the relevance score, filter and must_not only include or exclude. Filters are cached and do not compute scores, so put every yes/no condition (in stock, brand, price range) in filter and keep must for what the user typed. multi_match searches several fields and lets you boost one, such as name over description. fuzziness: "AUTO" tolerates typos. Aggregations run on the same request to build facets, such as counts per brand, so one call returns both results and filters for the sidebar.

1GET /products/_search
2{
3  "query": {
4    "bool": {
5      "must":   { "multi_match": { "query": "red runing shoes",
6                  "fields": ["name^3", "description"], "fuzziness": "AUTO" } },
7      "filter": [
8        { "term":  { "in_stock": true } },
9        { "range": { "price": { "lte": 120 } } }
10      ]
11    }
12  },
13  "aggs": { "brands": { "terms": { "field": "brand", "size": 10 } } },
14  "size": 20
15}

04Cluster, shards and replicas

An index is split into primary shards, each a self-contained Lucene index, and every primary can have replica copies on other nodes. Shards are how Elasticsearch spreads data and parallelises queries; replicas provide failover and extra read capacity. A query fans out to one copy of every shard, each returns its top hits, and the coordinating node merges them. The number of primary shards is fixed at index creation (changing it needs a split, shrink or reindex), so size it from expected data: Elastic's guidance is roughly 10–50 GB per shard and not more than about 200 million documents. Too many tiny shards waste memory; a few huge ones are slow to recover.

• Cluster health green: all shards assigned; yellow: replicas missing; red: a primary is missing.
• Node roles: master-eligible, data (hot, warm, cold, frozen), ingest, coordinating, ml.
• Use three master-eligible nodes for a production cluster to avoid split decisions.

05Indexing from an application, safely

Your primary database stays the source of truth; Elasticsearch is a derived, searchable copy. Index documents in batches with the _bulk API, not one request per document. Use your database ID as the Elasticsearch _id so re-indexing is idempotent. For changes, either queue an indexing job after each write (an outbox or queue keeps it reliable) or use change data capture. Always read and write through an alias, not the real index name: when the mapping needs to change, build a new index, reindex into it and switch the alias atomically. Security is on by default since 8.0 (TLS and authentication), so use an API key with only the privileges the app needs.

1POST /_bulk
2{ "index": { "_index": "products", "_id": "1001" } }
3{ "name": "Trail running shoe", "brand": "acme", "price": 89.9, "in_stock": true }
4{ "index": { "_index": "products", "_id": "1002" } }
5{ "name": "Road running shoe", "brand": "acme", "price": 99.0, "in_stock": false }
6 
7POST /_aliases
8{ "actions": [
9  { "remove": { "index": "products_v1", "alias": "products_live" } },
10  { "add":    { "index": "products_v2", "alias": "products_live" } }
11] }

06Licences, OpenSearch and managed options

In 2021 Elastic moved from Apache 2.0 to the SSPL and Elastic License, and AWS forked the last open version into OpenSearch, now governed by the OpenSearch Software Foundation under the Linux Foundation. In 2024 Elastic added AGPLv3 as a licence option for the source code. The two projects share roots and much of the Query DSL, but they have diverged in features, plugins and client libraries, so pick one and use its own client. Hosting options include Elastic Cloud (including a serverless offering), Amazon OpenSearch Service, or self-managed clusters on VMs or Kubernetes (with the ECK operator for Elasticsearch).

07Segments, refresh, flush and merges

Inside each shard, Lucene writes data into immutable segments. New documents go to an in-memory buffer and the translog (a write-ahead log). A refresh (every 1 second by default, only on indices that were searched recently) turns the buffer into a new searchable segment; a flush commits segments to disk and trims the translog. Because segments are immutable, updates are really delete plus insert, and deleted documents are only removed when background merges combine small segments into bigger ones. For bulk loads, set refresh_interval to -1 and replicas to 0, load, then restore both; it can multiply indexing speed. Avoid frequent single-document updates on the same IDs, which create many tiny segments and deleted docs.

1PUT /products_v2/_settings
2{ "index": { "refresh_interval": "-1", "number_of_replicas": 0 } }
3 
4// ...run the bulk load, then:
5PUT /products_v2/_settings
6{ "index": { "refresh_interval": "1s", "number_of_replicas": 1 } }
7POST /products_v2/_refresh

08Vector, semantic and hybrid search

Elasticsearch 8 and 9 support dense_vector fields with approximate kNN search using HNSW graphs, with quantization (int8, int4 and BBQ, a binary quantization) to cut memory. The semantic_text field type goes further: you attach an inference endpoint and Elasticsearch chunks the text and creates embeddings at index time. Retrievers let you combine a standard BM25 query with a kNN query using reciprocal rank fusion (RRF), which merges two ranked lists by position instead of raw score. Hybrid search usually beats either method alone: keywords catch exact SKUs and names, vectors catch meaning. Measure with a labelled query set before and after.

1GET /products/_search
2{
3  "retriever": { "rrf": { "retrievers": [
4    { "standard": { "query": { "match": { "name": "waterproof hiking boots" } } } },
5    { "knn": { "field": "name_vector", "k": 20, "num_candidates": 100,
6               "query_vector_builder": { "text_embedding": {
7                 "model_id": "my-embedding-endpoint", "model_text": "waterproof hiking boots" } } } }
8  ] } }
9}

09Deep pagination and consistent scrolling

from + size pagination makes every shard collect from + size hits and the coordinator sort them all, so page 500 is very expensive; the default limit (index.max_result_window) is 10,000 results for that reason. For user-facing infinite scrolling or exports, use search_after with a sort that ends in a unique tiebreaker, combined with a point in time (PIT) so the view does not shift while new documents arrive. The older scroll API is still available but is no longer recommended for deep pagination. For exports of very large result sets, consider sliced PIT searches in parallel.

10Operating a cluster: memory, tiers and snapshots

Give the JVM heap no more than about half the RAM and stay below the compressed-pointer threshold (around 30 GB); the rest is used by the OS file cache, which Lucene relies on heavily. Recent versions size the heap automatically, which is usually right. Use hot, warm, cold and frozen data tiers with ILM for time-based data so old indices move to cheaper storage and are eventually deleted. Snapshot regularly to object storage (S3, GCS, Azure) with snapshot lifecycle management; snapshots are incremental and are your real backup, because replicas do not protect against deleting an index. Upgrade with rolling restarts within a major, and check the deprecation API before major upgrades.