Last modified: January 24, 2026
This article is written in: 🇺🇸
In large-scale distributed architectures, multiple processes, microservices, or nodes must operate in concert to achieve consistency, fault tolerance, and robust state management. Coordination services address these challenges by offering primitives like distributed locks, leader election, and configuration storage. This document provides an overview of common coordination problems, introduces popular coordination systems, and outlines best practices for integrating them into a distributed environment. ASCII diagrams and command examples highlight practical usage scenarios.
In a distributed setup, systems are physically and/or logically separated across multiple machines or datacenters. They need ways to keep certain states or tasks in sync, manage membership (who is up or down), and resolve conflicts. Coordination services such as Apache ZooKeeper, etcd, and HashiCorp Consul provide a consistent data store and specialized APIs to solve these problems.
Typical challenges they address include:
By offloading these tasks to a trusted coordination service, developers avoid implementing and debugging low-level consensus and concurrency logic themselves.
Selecting a node to act as primary in a cluster ensures that only one node handles tasks like cluster management, partition rebalancing, or job scheduling. Other nodes remain followers or standbys. If the leader fails, a new one is chosen.
| Node A (Leader) |
+-------------------+
^
|
| (Leader election)
|
+-------------------+ +-------------------+
| Node B (Follower) | | Node C (Follower) |
+-------------------+ +-------------------+
When Node A fails, B or C is elected as new leader.A distributed lock ensures that only one instance can perform a specific operation at a time (e.g., updating a shared file). If multiple nodes attempt to grab the same lock, the coordination service grants it to one while making others wait or fail.
Services register themselves (their IPs, ports, health states) with the coordination store, letting clients query or watch for updates. This eliminates hard-coded addresses. For example, a web service can discover available database instances through a registry rather than a static config file.
Storing application settings, feature flags, or runtime parameters in a central key-value store ensures that all instances read a unified configuration. Changes can be watched in real time by applications to adapt on the fly.
One of the earliest widely adopted coordination services, ZooKeeper implements a replicated, in-memory hierarchical data store. Clients create, read, update, or watch “znodes,” which store data and can notify watchers on changes.
Example (zkCli command snippet):
# Connect to a ZooKeeper server
zkCli.sh -server 127.0.0.1:2181
# Create a znode
create /mylock "some-data"
# List children
ls /
# Set data
set /mylock "updated"A key-value store from the CoreOS project, etcd uses the Raft consensus algorithm for strong consistency and fault tolerance. It’s known for powering Kubernetes’ cluster state.
/app/config/db_url). Example (etcdctl snippet):
# Set a key
etcdctl put /myapp/config "config_data"
# Retrieve a key
etcdctl get /myapp/config
# Watch for changes
etcdctl watch /myapp/Consul integrates service discovery, health checks, key-value storage, and DNS-based queries.
Example (Consul CLI snippet):
# Set a KV pair
consul kv put myapp/config "{\"max_retries\": 5}"
# Retrieve a KV pair
consul kv get myapp/config
# Register a service
consul services register -name="web" -port=8080Services that require exactly one active instance use ephemeral znodes (ZooKeeper) or a compare-and-swap key (etcd) to claim leadership. Others watch that key; if the leader node fails, a new node claims leadership.
+--------------+ +---------+ +---------+
| ZK/etcd/etc |---| Node A |--| Node B |
| (KV store) | +---------+ +---------+
^
| ephemeral node or CAS check
v
Leader = Node AAn application obtains a lock by creating a znode or updating an etcd key with a unique token. If the key is already taken, it waits or fails. Upon completion, it deletes the key or ephemeral znode to release the lock.
Store config items under paths (e.g., /app/config/db). Clients watch for changes, so updating /app/config/db triggers watchers to reload new settings without a redeploy.
Key: /app/config/db_url
Value: "jdbc:mysql://dbhost:3306/mydb"Nodes create ephemeral keys signifying they’re alive. On failure or network partition, the ephemeral key goes away (session ends), alerting other components that the node is offline.
Coordination services typically run on multiple nodes (3 or 5 recommended). Each node keeps a copy of the state, and they apply consensus algorithms (like ZooKeeper’s Zab or etcd’s Raft) to maintain consistency.
CODE_BLOCK_PLACEHOLDER An odd number of nodes avoids ties. A majority must be reachable for writes to succeed.
If the current leader fails, the remaining nodes elect a new leader. In a 3-node cluster, any 2 form a quorum to keep operating. Under deeper network splits, partial partitions lose their quorum and become read-only or unavailable.
+----------+ +----------+ +----------+
| Node1 | | Node2 | | Node3 |
| (Leader) | | (Follower| | (Follower|
+----------+ +----------+ +----------+
^^^ Replication/Consensus ^^^import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
CuratorFramework client = ... // connect to ZooKeeper
InterProcessMutex lock = new InterProcessMutex(client, "/mylock");
try {
lock.acquire();
// Perform critical operation
} finally {
lock.release();
}# Create a lease for 10 seconds
LEASE_ID=$(etcdctl lease grant 10 | grep ID | awk '{print $3}')
# Lock key with that lease
etcdctl lock --lease=$LEASE_ID mylock "Performing critical section"
# On success, it prints that we hold the lock. The lock is released after 10s or when we exit.