Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 37. Replacing a failed etcd member

If some etcd members fail, but you still have a quorum of etcd members, you can use the remaining etcd members and the data that they contain to add more etcd members without etcd or cluster downtime.

37.1. Removing a failed etcd node
Copy link

Before you add a new etcd node, remove the failed one.

Procedure

  1. From an active etcd host, remove the failed etcd node: 

    # etcdctl -C https://<surviving host IP>:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key cluster-health

# etcdctl -C https://<surviving host IP>:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key member remove <failed member identifier>
    Copy to Clipboard Toggle word wrap

  2. Stop the etcd service on the failed etcd member: 

    # systemctl stop etcd
    Copy to Clipboard Toggle word wrap

37.2. Adding an etcd member
Copy link

You can add an etcd host either by using an Ansible playbook or by manual steps.

37.2.1. Adding a new etcd host using Ansible
Copy link

Procedure

  1. In the Ansible inventory file, create a new group named [new_etcd] and add the new host. Then, add the new_etcd group as a child of the [OSEv3] group: 

    [OSEv3:children]
masters
nodes
etcd
new_etcd
    1 
    

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com

[new_etcd]
    2 
    
etcd0.example.com
    3
    Copy to Clipboard Toggle word wrap
    1 2 3
    Add these lines.

  2. From the host that installed OpenShift Container Platform and hosts the Ansible inventory file, run the etcd scaleup playbook: 

    $ ansible-playbook  /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/scaleup.yml
    Copy to Clipboard Toggle word wrap

  3. After the playbook runs, modify the inventory file to reflect the current status by moving the new etcd host from the [new_etcd] group to the [etcd] group: 

    [OSEv3:children]
masters
nodes
etcd
new_etcd

... [OUTPUT ABBREVIATED] ...

[etcd]
master-0.example.com
master-1.example.com
master-2.example.com
etcd0.example.com
    Copy to Clipboard Toggle word wrap

  4. If you use the service catalog, you must update its list of etcd servers: 

    $ oc edit ds apiserver -n kube-service-catalog
    Copy to Clipboard Toggle word wrap

    Add the FQDN for the new etcd node to the --etcd-servers argument. This argument contains a comma-separated list.

  5. If you use Flannel, modify the flanneld service configuration on every OpenShift Container Platform host, located at /etc/sysconfig/flanneld, to include the new etcd host: 

    FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379
    Copy to Clipboard Toggle word wrap

  6. Restart the flanneld service: 

    # systemctl restart flanneld.service
    Copy to Clipboard Toggle word wrap

37.2.2. Manually adding a new etcd host
Copy link

Procedure
Modify the current etcd cluster

To create the etcd certificates, run the openssl command, replacing the values with those from your environment.

  1. Create some environment variables: 

    export NEW_ETCD_HOSTNAME="*etcd0.example.com*"
export NEW_ETCD_IP="192.168.55.21"

export CN=$NEW_ETCD_HOSTNAME
export SAN="IP:${NEW_ETCD_IP}, DNS:${NEW_ETCD_HOSTNAME}"
export PREFIX="/etc/etcd/generated_certs/etcd-$CN/"
export OPENSSLCFG="/etc/etcd/ca/openssl.cnf"
    Copy to Clipboard Toggle word wrap
    Note

    The custom openssl extensions used as etcd_v3_ca_* include the $SAN environment variable as subjectAltName. See /etc/etcd/ca/openssl.cnf for more information.

  2. Create the directory to store the configuration and certificates: 

    # mkdir -p ${PREFIX}
    Copy to Clipboard Toggle word wrap

  3. Create the server certificate request and sign it: (server.csr and server.crt) 

    # openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}server.key  \
    -out ${PREFIX}server.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN

# openssl ca -name etcd_ca -config ${OPENSSLCFG} \
    -out ${PREFIX}server.crt \
    -in ${PREFIX}server.csr \
    -extensions etcd_v3_ca_server -batch
    Copy to Clipboard Toggle word wrap

  4. Create the peer certificate request and sign it: (peer.csr and peer.crt) 

    # openssl req -new -config ${OPENSSLCFG} \
    -keyout ${PREFIX}peer.key \
    -out ${PREFIX}peer.csr \
    -reqexts etcd_v3_req -batch -nodes \
    -subj /CN=$CN

# openssl ca -name etcd_ca -config ${OPENSSLCFG} \
  -out ${PREFIX}peer.crt \
  -in ${PREFIX}peer.csr \
  -extensions etcd_v3_ca_peer -batch
    Copy to Clipboard Toggle word wrap

  5. Copy the current etcd configuration and ca.crt files from the current node as examples to modify later: 

    # cp /etc/etcd/etcd.conf ${PREFIX}
# cp /etc/etcd/ca.crt ${PREFIX}
    Copy to Clipboard Toggle word wrap

  6. While still on the surviving etcd host, add the new host to the cluster. To add additional etcd members to the cluster, you must first adjust the default localhost peer in the peerURLs value for the first member:

    1. Get the member ID for the first member using the member list command: 

      # etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
      1 
      
    member list
      Copy to Clipboard Toggle word wrap
      1
      Ensure that you specify the URLs of only active etcd members in the --peers parameter value.

    2. Obtain the IP address where etcd listens for cluster peers: 

      $ ss -l4n | grep 2380
      Copy to Clipboard Toggle word wrap

    3. Update the value of peerURLs using the etcdctl member update command by passing the member ID and IP address obtained from the previous steps: 

      # etcdctl --cert-file=/etc/etcd/peer.crt \
    --key-file=/etc/etcd/peer.key \
    --ca-file=/etc/etcd/ca.crt \
    --peers="https://172.18.1.18:2379,https://172.18.9.202:2379,https://172.18.0.75:2379" \
    member update 511b7fb6cc0001 https://172.18.1.18:2380
      Copy to Clipboard Toggle word wrap
    4. Re-run the member list command and ensure the peer URLs no longer include localhost.

  7. Add the new host to the etcd cluster. Note that the new host is not yet configured, so the status stays as unstarted until the you configure the new host.

    Warning

    You must add each member and bring it online one at a time. When you add each additional member to the cluster, you must adjust the peerURLs list for the current peers. The peerURLs list grows by one for each member added. The etcdctl member add command outputs the values that you must set in the etcd.conf file as you add each member, as described in the following instructions. 

    # etcdctl -C https://${CURRENT_ETCD_HOST}:2379 \
  --ca-file=/etc/etcd/ca.crt     \
  --cert-file=/etc/etcd/peer.crt     \
  --key-file=/etc/etcd/peer.key member add ${NEW_ETCD_HOSTNAME} https://${NEW_ETCD_IP}:2380
    1 
    

Added member named 10.3.9.222 with ID 4e1db163a21d7651 to cluster

ETCD_NAME="<NEW_ETCD_HOSTNAME>"
ETCD_INITIAL_CLUSTER="<NEW_ETCD_HOSTNAME>=https://<NEW_HOST_IP>:2380,<CLUSTERMEMBER1_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER2_NAME>=https:/<CLUSTERMEMBER2_IP>:2380,<CLUSTERMEMBER3_NAME>=https:/<CLUSTERMEMBER3_IP>:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
    Copy to Clipboard Toggle word wrap
    1
    In this line, 10.3.9.222 is a label for the etcd member. You can specify the host name, IP address, or a simple name.

  8. Update the sample ${PREFIX}/etcd.conf file.

    1. Replace the following values with the values generated in the previous step:

      • ETCD_NAME
      • ETCD_INITIAL_CLUSTER
      • ETCD_INITIAL_CLUSTER_STATE

    2. Modify the following variables with the new host IP from the output of the previous step. You can use ${NEW_ETCD_IP} as the value. 

      ETCD_LISTEN_PEER_URLS
ETCD_LISTEN_CLIENT_URLS
ETCD_INITIAL_ADVERTISE_PEER_URLS
ETCD_ADVERTISE_CLIENT_URLS
      Copy to Clipboard Toggle word wrap
    3. If you previously used the member system as an etcd node, you must overwrite the current values in the /etc/etcd/etcd.conf file.

    4. Check the file for syntax errors or missing IP addresses, otherwise the etcd service might fail: 

      # vi ${PREFIX}/etcd.conf
      Copy to Clipboard Toggle word wrap
  9. On the node that hosts the installation files, update the [etcd] hosts group in the /etc/ansible/hosts inventory file. Remove the old etcd hosts and add the new ones.

  10. Create a tgz file that contains the certificates, the sample configuration file, and the ca and copy it to the new host: 

    # tar -czvf /etc/etcd/generated_certs/${CN}.tgz -C ${PREFIX} .
# scp /etc/etcd/generated_certs/${CN}.tgz ${CN}:/tmp/
    Copy to Clipboard Toggle word wrap
Modify the new etcd host

  1. Install iptables-services to provide iptables utilities to open the required ports for etcd: 

    # yum install -y iptables-services
    Copy to Clipboard Toggle word wrap

  2. Create the OS_FIREWALL_ALLOW firewall rules to allow etcd to communicate:

    • Port 2379/tcp for clients

    • Port 2380/tcp for peer communication 

      # systemctl enable iptables.service --now
# iptables -N OS_FIREWALL_ALLOW
# iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
# iptables-save | tee /etc/sysconfig/iptables
      Copy to Clipboard Toggle word wrap
      Note

      In this example, a new chain OS_FIREWALL_ALLOW is created, which is the standard naming the OpenShift Container Platform installer uses for firewall rules.

      Warning

      If the environment is hosted in an IaaS environment, modify the security groups for the instance to allow incoming traffic to those ports as well.

  3. Install etcd: 

    # yum install -y etcd
    Copy to Clipboard Toggle word wrap

    Ensure version etcd-2.3.7-4.el7.x86_64 or greater is installed,

  4. Ensure the etcd service is not running: 

    # systemctl disable etcd --now
    Copy to Clipboard Toggle word wrap

  5. Remove any etcd configuration and data: 

    # rm -Rf /etc/etcd/*
# rm -Rf /var/lib/etcd/*
    Copy to Clipboard Toggle word wrap

  6. Extract the certificates and configuration files: 

    # tar xzvf /tmp/etcd0.example.com.tgz -C /etc/etcd/
    Copy to Clipboard Toggle word wrap

  7. Modify the file ownership permissions: 

    # chown -R etcd:etcd /etc/etcd/*
# chown -R etcd:etcd /var/lib/etcd/
    Copy to Clipboard Toggle word wrap

  8. Start etcd on the new host: 

    # systemctl enable etcd --now
    Copy to Clipboard Toggle word wrap

  9. Verify that the host is part of the cluster and the current cluster health:

    • If you use the v2 etcd api, run the following command: 

      # etcdctl --cert-file=/etc/etcd/peer.crt \
          --key-file=/etc/etcd/peer.key \
          --ca-file=/etc/etcd/ca.crt \
          --peers="https://*master-0.example.com*:2379,\
          https://*master-1.example.com*:2379,\
          https://*master-2.example.com*:2379,\
          https://*etcd0.example.com*:2379"\
          cluster-health
member 5ee217d19001 is healthy: got healthy result from https://192.168.55.12:2379
member 2a529ba1840722c0 is healthy: got healthy result from https://192.168.55.8:2379
member 8b8904727bf526a5 is healthy: got healthy result from https://192.168.55.21:2379
member ed4f0efd277d7599 is healthy: got healthy result from https://192.168.55.13:2379
cluster is healthy
      Copy to Clipboard Toggle word wrap

    • If you use the v3 etcd api, run the following command: 

      # ETCDCTL_API=3 etcdctl --cert="/etc/etcd/peer.crt" \
          --key=/etc/etcd/peer.key \
          --cacert="/etc/etcd/ca.crt" \
          --endpoints="https://*master-0.example.com*:2379,\
            https://*master-1.example.com*:2379,\
            https://*master-2.example.com*:2379,\
            https://*etcd0.example.com*:2379"\
            endpoint health
https://master-0.example.com:2379 is healthy: successfully committed proposal: took = 5.011358ms
https://master-1.example.com:2379 is healthy: successfully committed proposal: took = 1.305173ms
https://master-2.example.com:2379 is healthy: successfully committed proposal: took = 1.388772ms
https://etcd0.example.com:2379 is healthy: successfully committed proposal: took = 1.498829ms
      Copy to Clipboard Toggle word wrap
Modify each OpenShift Container Platform master

  1. Modify the master configuration in the etcClientInfo section of the /etc/origin/master/master-config.yaml file on every master. Add the new etcd host to the list of the etcd servers OpenShift Container Platform uses to store the data, and remove any failed etcd hosts: 

    etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
    - https://master-0.example.com:2379
    - https://master-1.example.com:2379
    - https://master-2.example.com:2379
    - https://etcd0.example.com:2379
    Copy to Clipboard Toggle word wrap

  2. Restart the master API service:

    • On every master: 

      # systemctl restart atomic-openshift-master-api
      Copy to Clipboard Toggle word wrap

    • Or, on a single master cluster installation: 

      # systemctl restart atomic-openshift-master
      Copy to Clipboard Toggle word wrap
      Warning

      The number of etcd nodes must be odd, so you must add at least two hosts.

  3. If you use Flannel, modify the flanneld service configuration located at /etc/sysconfig/flanneld on every OpenShift Container Platform host to include the new etcd host: 

    FLANNEL_ETCD_ENDPOINTS=https://master-0.example.com:2379,https://master-1.example.com:2379,https://master-2.example.com:2379,https://etcd0.example.com:2379
    Copy to Clipboard Toggle word wrap

  4. Restart the flanneld service: 

    # systemctl restart flanneld.service
    Copy to Clipboard Toggle word wrap
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat