Configure a lifecycle policy

Stack

An index lifecycle management (ILM) policy defines how your indices are managed over time, automating when and how they transition as they age.

Consider these aspects when configuring an ILM policy:

For ILM to manage an index, you need to specify a valid policy in the index.lifecycle.name index setting.
To configure a lifecycle policy for rolling indices or data streams, you create the policy and add it to the index template.
To use a policy to manage a single index, you can specify a lifecycle policy when you create the index, or apply a policy directly to an existing index.
ILM policies are stored in the global cluster state and can be included in snapshots by setting include_global_state to true when you take the snapshot. When the snapshot is restored, all of the policies in the global state are restored and any local policies with the same names are overwritten.

Note

This page is specifically about ILM. If you're looking for a simpler lifecycle management option for data streams, refer to Data stream lifecycle. Check Data lifecycle to compare these lifecycle management options.

Overview

To set up ILM to manage one or more indices, the general procedure is as follows:

Create a lifecycle policy
Create an index template to apply the lifecycle policy

If you're configuring ILM for rolling indices and not using data streams, you additionally need to:

Create an initial managed index

You can perform these actions in Kibana or using the Elasticsearch API.

Create a lifecycle policy

A lifecycle policy defines a set of index lifecycle phases and the actions to perform on the managed indices in each phase.

Kibana

To add an ILM policy to an Elasticsearch cluster:

Go to Stack Management > Index Lifecycle Policies, and select Create policy.
Specify a name for the lifecycle policy. Later on, when you create an index template to define how indices are created, you'll use this name to assign the lifecycle policy to each index.
In the Hot phase, by default an ILM-managed index rolls over when either:
- It reaches 30 days of age.
- One or more primary shards reach 50 GB in size.
Disable Use recommended defaults to adjust these values or to roll over based on the size of the primary shard, the number of documents in the primary shard, or the total number of documents in the index.

Important

The rollover action implicitly rolls over a data stream or alias if one or more shards contain 200,000,000 or more documents. Typically, a shard will reach 50GB before it reaches 200M documents, however, this isn’t the case for space efficient data sets. This built-in limit exists to avoid Search performance loss if a shard contains more than 200M documents.
By default, only the hot index lifecycle phase is enabled. Enable each additional lifecycle phase that you'd like, and for each choose any index lifecycle actions to perform on indices when they enter that phase.

For example, you could choose the action to downsample the index, which aggregates the time series data into statistical summaries, reducing the data storage size.

Note that for each phase after the hot phase, you have the option to move the data into the next phase after a certain duration of time. This duration is calculated from the time of the index rollover and not from the time the index is created.
For the final phase that's enabled, choose to either keep the data in the phase forever or delete the data after a specified period of time.

API

Use the Create or update policy API to add an ILM policy to the Elasticsearch cluster:

		 PUT _ilm/policy/my_policy {
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "25GB"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

	

Roll over the index when one or more primary shards reach 25GB in size.
Delete the index 30 days after rollover

Important

The rollover action implicitly rolls over a data stream or alias if one or more shards contain 200,000,000 or more documents. Typically, a shard will reach 50GB before it reaches 200M documents, however, this isn’t the case for space efficient data sets. This built-in limit exists to avoid Search performance loss if a shard contains more than 200M documents.

Create an index template to apply the lifecycle policy

To use a lifecycle policy that triggers a rollover action, you need to configure the policy in the index template used to create each new index. You specify the name of the policy and the alias used to reference the rolling indices.

Tip

If you already have an index template to which you'd like to add an ILM policy, you can do this from Stack Management > Index Lifecycle Policies. Search for and select the policy you want, and from the Actions menu, select Add to index template.

Kibana

To add an index template to a cluster and apply the lifecycle policy to indices matching the template:

Go to Stack Management > Index Management. In the Index Templates tab, select Create template.
On the Logistics page:
1. Specify a name for the template.
2. Specify a pattern to match the indices you want to manage with the lifecycle policy. For example, my-index-*.
3. If you're storing continuously generated, append-only data, you can opt to create data streams instead of indices for more efficient storage.
  
  Note
  
  When you enable the data stream option, an option to set Data retention also becomes available. Since you're creating an index lifecycle policy to manage indices, the Data retention option must remain disabled. Data retention is applicable only if you're using a data stream lifecycle, which is an alternative to ILM. Refer to the Data stream lifecycle to learn more.
4. Configure any other options you'd like, including:
  - The index mode to use for the created indices.
  - The template priority, version, and any metadata.
  - Whether or not to overwrite the action.auto_create_index cluster setting.
  Refer to the Create or update index template API documentation for details about these options.
On the Component templates page, use the search and filter tools to select any component templates to include in the index template. The index template will inherit the settings, mappings, and aliases defined in the component templates and apply them to indices when they're created.
On the Index settings page:
1. Configure ILM by specifying the ILM settings to apply to the indices:
  - index.lifecycle.name - The lifecycle policy to manage the created indices.
  - index.lifecycle.rollover_alias - The index alias used for querying and managing the set of indices associated with a lifecycle policy that contains a rollover action.
    
    Tip
    
    The index.lifecycle.rollover_alias setting is required only if you're using index lifecycle management with an alias. It is unnecessary when using Data Streams.
2. Optional: Add any additional index settings, that should be applied to the indices as they're created. For example, you can set the number of shards and replicas for each index:
```
{
  "index.lifecycle.name": "my_policy",
  "index.lifecycle.rollover_alias": "test-alias",
  "number_of_shards": 1,
  "number_of_replicas": 1
}
```
Optional: On the Mappings page, customize the fields and data types used when documents are indexed into Elasticsearch. Refer to Mapping for details.
Optional: On the Aliases page, specify an alias for each created index. This isn't required when configuring ILM, which instead uses the index.lifecycle.rollover_alias setting to acceess rolling indices.
On the Review page, confirm your selections. You can check your selected options, as well as both the format of the index template that will be created and the associated API request.

The newly created index template will be used for all new indices with names that match the specified pattern, and for each of these, the specified ILM policy will be applied.

API

Use the Create or update index template API to add an index template to a cluster and apply the lifecycle policy to indices matching the template:

		 PUT _index_template/my_template {
  "index_patterns": ["test-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "my_policy",
      "index.lifecycle.rollover_alias": "test-alias"
    }
  }
}

	

Use this template for all new indices with names that begin with test-.
Apply my_policy to new indices created with this template.
Define an index alias for referencing indices managed by my_policy.

Tip

The index.lifecycle.rollover_alias setting is required only if you're using index lifecycle management with an alias. It is unnecessary when using Data Streams.

Create an initial managed index

When you set up policies for your own rolling indices and are not using the recommended data streams, you must manually create the first index managed by a policy and designate it as the write index.

The name of the index must match the pattern defined in the index template and end with a number. This number is incremented to generate the name of indices created by the rollover action.

Important

When you enable index lifecycle management for Beats, Elastic Agent, or for the Elastic Agent or Logstash Elasticsearch output plugins, the necessary policies and configuration changes are applied automatically. If you'd like to create a specialized ILM policy for any data stream, refer to our tutorial Customize built-in policies.

Kibana

To create the initial managed index:

Go to Stack Management > Index Management. In the Indices tab, select Create index.
Specify a name for the index that matches the index template pattern and that ends with a number. For example, test-000001.
Leave the Index mode set to the default Standard.

Create an alias for the index:

Open Dev tools.
Send the following request:

		 POST /_aliases {
  "actions" : [
    { "add" : { "index" : "my-index", "alias" : "my-alias" } }
  ]
}

	

Replace my-index with the name of the initial managed index that you created previously and set my-alias to the rollover alias specified by index.lifecycle.rollover_alias in the index template.

Now you can start indexing data to the rollover alias specified in the lifecycle policy. With the sample my_policy policy, the rollover action is triggered once the initial index exceeds 50GB. ILM then creates a new index that becomes the write index for the test-alias.

API

Use the Create an index API to create the initial managed index.

The following request creates the test-000001 index. Because it matches the index pattern specified in my_template, Elasticsearch automatically applies the settings from that template.

		 PUT test-000001 {
  "aliases": {
    "test-alias":{
      "is_write_index": true
    }
  }
}

	

Set this initial index to be the write index for this alias.

Now you can start indexing data to the rollover alias specified in the lifecycle policy. With the sample my_policy policy, the rollover action is triggered once the initial index exceeds 25GB. ILM then creates a new index that becomes the write index for the test-alias.