Replication: Maintaining Copies of Database Changes
Replication keeps database data on more than one node by transferring changes from one database instance to another.
A common topology is:
writes
Application ---------> Primary
|
| replication stream
v
Replica 1
|
+--------> Replica 2
The copies are related, but this does not mean they are identical at every instant.
A replica may be:
milliseconds behind
seconds behind
minutes behind
completely disconnected
while the primary continues accepting transactions.
This immediately creates the central question in replication:
When the database tells the client that
COMMITsucceeded, how much of that transaction has reached the other copies?
That question determines:
durability after primary failure
replication lag
read consistency
failover data loss
write latency
Why replicate a database?
A single database server creates a simple failure mode:
┌───────────────┐
Application ---> │ Database │
│ only copy │
└───────────────┘
X
If that server becomes unavailable, the application may become unavailable too.
Replication introduces additional copies:
Primary
/ \
/ \
Replica 1 Replica 2
Those copies can serve different purposes.
| Goal | How replication can help |
|---|---|
| High availability | Promote another node when the primary fails |
| Disaster recovery | Maintain a copy in another location |
| Read scaling | Send reporting/read traffic to replicas |
| Backup offloading | Run some backup work on secondary systems |
| Analytics | Feed another database or analytical system |
| Geographic reads | Place readable copies closer to users |
| Upgrade/migration | Logical replication can move data between environments |
But replication is not automatically all of these things.
For example:
a reporting replica
and:
a failover replica
may have very different requirements.
A concrete example: create order 104
Suppose an application creates:
order_id = 104
customer_id = 27
amount = €89
The application sends:
BEGIN;
INSERT INTO orders(order_id, customer_id, amount)
VALUES (104, 27, 89);
COMMIT;
Assume a primary and one replica.
In asynchronous replication, the timeline might be:
Primary Replica
INSERT order 104
COMMIT
|
+---- durable locally
return "success"
to application
replication data ---------->
receive change
persist/log change
apply change
order 104 becomes
visible to queries
The key observation is:
client hears SUCCESS
before:
replica can necessarily return order 104
That interval is replication lag.
Replication usually transfers a change stream, not repeated full database copies
A naive replication design might sound like:
database changed
→ copy entire database again
That would obviously be far too expensive.
Instead, database systems normally maintain a stream describing changes.
Examples include:
PostgreSQL
WAL — Write-Ahead Log
MySQL
binary log
SQL Server Availability Groups
transaction log records
Conceptually:
Database state
+
|
v
ordered change stream
change 101
change 102
change 103
change 104
...
The replica starts from some known database state and then follows that change stream.
PostgreSQL example: WAL streaming
PostgreSQL physical streaming replication transfers WAL records.
The primary generates WAL as transactions modify the database.
A standby connects to the primary and streams WAL records as they are generated:
PostgreSQL primary
database changes
|
v
WAL
|
| walsender
v
================ network ================
|
| walreceiver
v
PostgreSQL standby
receive WAL
|
write/flush WAL
|
replay WAL
|
updated standby database
PostgreSQL documents that streaming replication sends WAL records incrementally rather than waiting for a whole WAL segment to fill, and that streaming replication is asynchronous by default.
This is a good example of an important pattern:
Replication often means shipping the database's recovery/change log to another machine and replaying it there.
The most important replication pipeline
A useful generic model is:
PRIMARY
transaction
|
v
commit/change log
|
v
SEND
|
|
| network
v
REPLICA
RECEIVE
|
v
PERSIST / HARDEN
|
v
APPLY / REPLAY
|
v
VISIBLE TO READERS
These are not interchangeable milestones.
The replica can have received a transaction without having applied it.
It can have persisted the transaction to its own log without having made the changed rows visible to queries.
That distinction explains a large fraction of replication behavior.
Receive, persist, apply, and visibility
Consider order 104.
Stage 1 — Receive
The replica has obtained the replication message:
Replica memory/network buffers:
"insert order 104"
This says little about crash durability.
The data may not yet be safely stored.
Stage 2 — Persist / harden
The replica has recorded the change in durable or recovery storage according to the engine's durability rules.
Conceptually:
Replica disk:
replication log contains order 104
If the replica process restarts, it may be able to recover the transaction from that log.
Stage 3 — Apply / replay
The replica's apply mechanism has executed or replayed the change against the database state:
orders table now contains order 104
Stage 4 — Visible to a read
The database's consistency rules allow a query to observe the applied transaction. Once order 104 has been applied on the replica, its orders table contains:
| order_id | customer_id | amount |
|---|---|---|
| 104 | 27 | 89.00 |
The read returns that row:
SELECT *
FROM orders
WHERE order_id = 104;
| order_id | customer_id | amount |
|---|---|---|
| 104 | 27 | 89.00 |
The important principle is:
Persisted on a replica is not necessarily the same thing as applied on the replica.
Why this distinction matters
Suppose:
Primary transaction commits.
Replica has already persisted the change.
Replica has not applied it yet.
Then two different statements can simultaneously be true:
The transaction is protected against loss
if the primary disappears.
and:
A query against the replica still cannot see it.
This surprises people because durability and read freshness are often incorrectly treated as the same property.
They are different.
SQL Server demonstrates this especially clearly
With SQL Server Always On Availability Groups, changes are sent from the primary to secondary replicas as transaction log records.
On the secondary:
receive log records
|
v
harden transaction log
|
v
redo
|
v
changes visible in secondary database
SQL Server synchronous-commit mode waits for the synchronous secondary to harden the log before the primary completes the relevant commit protocol.
But hardening does not mean redo is finished.
Microsoft explicitly documents that a redo queue can contain log records that have already been hardened on the secondary but have not yet been applied to its database pages. Reads on that secondary can therefore still return stale data.
So:
synchronous replication
does not automatically mean:
every read on the replica immediately sees the transaction
Replication lag is not one single delay
When somebody says:
replica lag = 5 seconds
ask:
Lag between which stages?
There can be multiple kinds of lag:
primary generated
↓
SEND
↓ ← send lag
replica received
↓
PERSIST
↓ ← flush/harden lag
replica durable
↓
APPLY
↓ ← replay/apply lag
replica visible
A replica might have excellent network throughput but slow apply performance.
Or it might apply quickly once data arrives but have a slow network.
Those are different problems.
PostgreSQL exposes these stages directly
PostgreSQL's replication monitoring includes multiple WAL positions.
Conceptually:
sent_lsn
↓
write_lsn
↓
flush_lsn
↓
replay_lsn
These correspond roughly to:
sent
received/written
durably flushed
replayed
PostgreSQL documentation recommends comparing WAL positions to identify whether lag is primarily:
primary sending slowly
network / receiver delay
or
standby replay delay
and notes that a large difference between received/flushed WAL and replay position means the standby is receiving WAL faster than it can replay it.
This is much more useful than treating "replica lag" as one mysterious number.
Asynchronous replication
With asynchronous replication, the primary does not wait for the replica before acknowledging the transaction to the client.
Conceptually:
Client Primary Replica
| | |
|---- write ---->| |
| | |
| | COMMIT |
| | |
|<--- success ---| |
| | |
| |---- replicate ---->|
| | |
The advantage is lower commit latency.
The client does not wait for:
network round trip
replica storage
replica apply
The cost is a failure window.
The asynchronous data-loss window
Suppose the primary tells the client:
Order 104 committed successfully.
but has not yet transmitted order 104 to the replica.
Then:
Primary dies permanently.
The replica may contain only:
orders 1 ... 103
If that replica is promoted, order 104 disappears from the surviving database state even though the application previously received success.
This is the fundamental durability tradeoff of asynchronous replication.
PostgreSQL explicitly notes that asynchronous streaming replication can lose transactions that committed on the primary but had not yet reached the standby when the primary failed.
SQL Server similarly documents possible data loss when failing over from asynchronous-commit replicas that have not caught up.
MySQL is asynchronous by default
MySQL source-to-replica replication is asynchronous by default.
A common pipeline is:
MySQL source
transaction
|
v
binary log
|
| replication connection
v
MySQL replica
receiver / I/O thread
|
v
relay log
|
v
applier / SQL thread(s)
|
v
replica database
MySQL's documentation explicitly describes normal replication as asynchronous and supports tracking either binary-log positions or GTIDs to identify replication progress.
This pipeline again separates:
received
from:
applied
Why an immediate replica read may fail
Suppose an API endpoint does:
POST /orders
which writes order 104 to the primary.
The next browser request performs:
GET /orders/104
but the application's load balancer sends reads to a replica.
The timing might be:
Primary:
order 104 committed
Client:
POST succeeded
Replica:
still on order 103
Client:
GET /orders/104
Replica:
404 / not found
Nothing has necessarily malfunctioned.
The read simply arrived before replication had applied the transaction.
This is a read-after-write consistency problem
The user expects:
I just created it
therefore I should be able to read it
That property is commonly called read-your-writes or read-after-write consistency.
Basic asynchronous read replicas do not automatically provide it.
Several application designs are possible.
For example:
write → primary
immediate confirmation read → primary
later ordinary reads → replicas
Another design carries a replication position/token and waits until the selected replica reaches that point.
Another design avoids the confirmation read entirely by returning the newly created object from the write operation.
The correct approach depends on how much staleness the application can tolerate.
Different reads have different freshness requirements
Consider three queries.
Product catalog
"What products are available?"
Being 500 ms behind may be acceptable.
Analytics dashboard
"How many orders did we receive today?"
Several seconds of lag may be acceptable.
Payment confirmation
"Did my €10,000 transfer succeed?"
A stale read may be unacceptable.
The mistake is designing one global rule:
all reads must use replicas
or:
all reads must use primary
without considering the semantics of individual requests.
Synchronous replication
Synchronous replication changes the commit rule.
Instead of:
primary commits
→ tell client success
→ replicate later
the protocol becomes something like:
primary prepares commit information
|
v
send to replica
|
v
wait for required acknowledgement
|
v
return success
The cost is latency.
The benefit is stronger durability.
But the phrase:
synchronous replication
is incomplete unless you know:
which replicas must acknowledge?
what exactly must they acknowledge?
receive?
memory write?
filesystem write?
durable flush?
application/replay?
PostgreSQL makes the acknowledgement level explicit
PostgreSQL's synchronous_commit setting provides an excellent concrete example.
With synchronous standbys configured, PostgreSQL supports several important acknowledgement levels:
| PostgreSQL setting | Commit waits for |
|---|---|
remote_write |
Standby received the WAL and wrote it to its operating system |
on |
Standby flushed the WAL to durable storage |
remote_apply |
Standby replayed the transaction so it is visible to standby queries |
PostgreSQL documents those differences directly. remote_write is weaker than durable standby flush, on waits for durable storage, and remote_apply additionally waits for replay and visibility.
This is exactly why:
receive
persist
apply
must be understood separately.
PostgreSQL remote_apply
Suppose an application requires:
COMMIT returned
therefore
a query sent to the synchronous standby
must be able to see the transaction
PostgreSQL can provide that stronger behavior with:
synchronous_commit = remote_apply
The commit waits until the synchronous standby reports that the commit record has been replayed and has become visible there.
Compare that with:
synchronous_commit = on
which waits for durable WAL on the synchronous standby but not necessarily replay.
That distinction is extremely important for applications performing synchronous replica reads.
Synchronous replication adds network latency to commits
Consider two servers in the same datacenter:
Primary ← 0.3 ms → Standby
The network penalty may be manageable.
Now place the synchronous standby across an ocean:
Frankfurt ←────────→ Sydney
Every synchronous transaction may now need to incorporate a large network round trip into its commit latency.
PostgreSQL explicitly notes that synchronous replication necessarily increases response time and that the minimum extra wait includes the primary-to-standby round trip.
That produces a common architecture:
nearby synchronous replica
+
distant asynchronous disaster-recovery replica
rather than making every geographically distant copy synchronous.
PostgreSQL can require several replicas
Suppose:
Primary
|
+-- Replica A
+-- Replica B
+-- Replica C
PostgreSQL can use priority-based or quorum-based synchronous replication.
For example:
ANY 2 (A, B, C)
means the commit can wait for any two of the listed synchronous candidates rather than requiring one specific machine.
PostgreSQL's current documentation supports both FIRST priority-based and ANY quorum-style synchronous standby selection.
This illustrates another important principle:
"Synchronous" also requires defining how many replicas constitute enough acknowledgement.
MySQL semisynchronous replication
MySQL provides semisynchronous replication in addition to its default asynchronous replication.
Conceptually:
Source
|
| transaction
v
binary log
|
+----------> replica
|
v
relay log
|
|
acknowledgement
|
v
Source returns to client
With semisynchronous replication enabled, the source waits for the required replica acknowledgement that the transaction has been received and logged before returning according to the configured wait point.
Importantly, this is not the same as:
replica has already applied the transaction
The replica's apply step remains separate.
MySQL can wait for more than one replica
MySQL exposes:
rpl_semi_sync_source_wait_for_replica_count
which controls how many replicas must acknowledge a transaction.
The default is one.
For example:
wait_for_replica_count = 2
requires two qualifying acknowledgements while semisynchronous mode remains active.
Again, the useful question is not:
"Is replication synchronous?"
It is:
Which nodes participate in the commit acknowledgement rule?
MySQL semisynchronous replication can fall back
Another operational detail matters.
MySQL's semisynchronous source has a timeout:
rpl_semi_sync_source_timeout
If sufficient acknowledgements do not arrive within that interval, MySQL can revert to asynchronous replication depending on configuration.
Therefore an architecture review should not stop at:
"semisync is enabled"
It should also ask:
What happens if all semisynchronous replicas disappear?
Do writes stop?
Do they continue asynchronously?
What durability guarantee does the application then have?
SQL Server synchronous commit
SQL Server Always On Availability Groups have:
asynchronous-commit mode
and:
synchronous-commit mode
In synchronous-commit mode, the primary waits for the synchronous secondary to acknowledge that the relevant transaction log records have been hardened to disk.
This protects committed transactions against loss on primary failure when the secondary is properly synchronized.
But remember the earlier pipeline:
harden
↓
redo
↓
visible
Synchronous commit does not collapse those into one operation.
SQL Server synchronous commit can still have stale readable secondary data
Suppose:
transaction log is already hardened
but the secondary's redo process is overloaded.
Then:
redo queue grows
and the transaction may not yet be reflected in queries against a readable secondary.
Microsoft explicitly documents this scenario: log records can be safely hardened yet remain unreadable until redo applies them.
This is one of the clearest examples of:
durability lag and query-visibility lag being different concepts.
Technology comparison: commit acknowledgement
| Technology | Default/common asynchronous behavior | Stronger acknowledgement option | What stronger acknowledgement means |
|---|---|---|---|
| PostgreSQL streaming replication | Asynchronous by default | Synchronous replication | Can wait for remote write, durable flush, or even replay depending on synchronous_commit |
| MySQL source/replica | Asynchronous by default | Semisynchronous replication | Source waits for configured replica receipt/log acknowledgement; apply remains separate |
| SQL Server Availability Groups | Async-commit available | Synchronous-commit | Primary waits for synchronized secondary log hardening |
PostgreSQL remote_apply |
— | Stronger than normal synchronous flush | Commit can wait until transaction is replayed and query-visible |
These distinctions are documented by PostgreSQL's WAL and synchronous replication configuration, MySQL's replication and semisynchronous replication documentation, and SQL Server's Availability Group availability-mode documentation.
Starting a new replica requires a consistent starting point
A new replica cannot normally begin from:
an empty database
+
whatever replication messages happen to arrive now
Suppose the primary already contains:
orders 1 ... 10,000,000
and the replica begins listening at:
order 10,000,001
The previous data is missing.
So replication setup normally requires:
1. create a consistent base copy
2. record the change-stream position corresponding to that copy
3. continue replication from that position
This is often called:
seeding
bootstrap
base backup
initial snapshot
depending on the technology.
Why copying live database files casually is unsafe
Suppose a database contains:
customers.dat
orders.dat
indexes.dat
and you manually copy them while transactions are running.
The copy process might capture:
customers.dat at 10:00:01
orders.dat at 10:00:08
indexes.dat at 10:00:15
Those files may represent different logical moments.
You can therefore create an inconsistent database image.
Database engines provide supported backup/snapshot procedures specifically to produce a recoverable starting state.
PostgreSQL, for example, requires a suitable base backup to bootstrap a physical standby.
MySQL likewise documents taking a source snapshot before starting replication from the corresponding position.
Replication positions
The replica needs to answer:
How far through the source's change history have I processed?
Systems therefore maintain replication positions.
Examples include:
PostgreSQL
LSN — Log Sequence Number
MySQL
binary-log file + position
or
GTID set
SQL Server
LSN-based transaction-log positions internally
Conceptually:
Source stream:
100 ─ 101 ─ 102 ─ 103 ─ 104 ─ 105
Replica received through: 105
Replica persisted through: 104
Replica applied through: 102
That is much more informative than simply saying:
replication is running
MySQL GTIDs
MySQL can identify transactions using Global Transaction Identifiers (GTIDs).
Instead of saying:
"continue at binary log mysql-bin.008214 byte 93482"
a replica can reason in terms of which transactions it has already executed.
MySQL supports GTID auto-positioning so a replica can determine which transactions it still needs from the source.
This is especially useful when topology changes make one fixed log filename/offset inconvenient.
PostgreSQL replication slots
A different problem occurs when the replica falls behind.
Suppose the primary generates WAL quickly.
Without protection, it may eventually recycle old WAL that the disconnected replica still needs.
PostgreSQL replication slots tell the primary:
this consumer has not yet passed LSN X
do not discard the WAL it still requires
PostgreSQL documents replication slots specifically as a mechanism to retain WAL needed by standbys.
But this creates another operational risk:
replica offline
↓
slot cannot advance
↓
primary retains more WAL
↓
disk usage grows
PostgreSQL explicitly warns that replication slots can retain enough WAL to fill pg_wal if not managed.
So solving:
replica missed required history
can create:
primary disk exhaustion
if monitoring is absent.
Physical replication
Physical replication works at a level closely tied to the storage/recovery format of the database engine.
PostgreSQL physical streaming replication is the clearest example:
primary WAL
|
v
standby WAL replay
The standby reconstructs the physical database state by replaying WAL.
Advantages include:
very complete copy of the database cluster state
efficient failover-style replication
close integration with crash recovery
The tradeoff is tighter engine/version/platform coupling.
PostgreSQL notes that physical log shipping is tied closely enough to server storage formats that major PostgreSQL versions generally cannot be mixed in a normal physical primary/standby configuration.
Logical replication
Logical replication works with logical data changes rather than reproducing the source's physical storage layout.
Conceptually:
INSERT customer 7
UPDATE order 104
DELETE item 8
rather than:
modify physical page X at byte offsets Y...
PostgreSQL logical replication uses a publication/subscription model and tracks rows through a replication identity, usually a primary key.
This allows much more selective replication.
For example:
Database A
customers
orders
internal_audit
temporary_processing
might publish only:
customers
orders
to another system.
PostgreSQL logical replication
A PostgreSQL logical replication topology might look like:
Publisher
customers
orders
payments
internal_logs
|
| publication:
| customers, orders
v
Subscriber
customers
orders
PostgreSQL can perform an initial table synchronization and then continually send changes made after that snapshot.
Logical replication can support scenarios such as:
replicating only selected data
replicating between PostgreSQL major versions
replicating between different platforms
feeding analytical databases
which PostgreSQL explicitly lists among its logical-replication use cases.
Physical versus logical replication
| Property | Physical | Logical |
|---|---|---|
| Replicates | Storage/recovery-level changes | Logical row/data changes |
| Typical use | HA standby | Selective replication, migrations, integrations |
| Scope | Usually broad database/cluster copy | Can often choose tables/rows/columns |
| Engine coupling | High | Usually lower |
| Cross-version flexibility | Limited | Often better |
| Schema handling | Physical copy naturally follows storage state | Schema compatibility must be considered |
| Example | PostgreSQL WAL streaming | PostgreSQL publications/subscriptions |
Neither is universally better.
They solve different problems.
Logical replication does not mean "everything is copied"
Suppose the publisher adds:
ALTER TABLE customers
ADD COLUMN loyalty_level integer;
Do not assume every logical replication technology automatically reproduces every schema operation.
Logical replication frequently requires the target schema to be compatible with the replicated data.
Likewise:
sequences
large objects
DDL
permissions
extensions
triggers
may have technology-specific behavior.
Therefore:
"The tables are logically replicated"
does not imply:
"Every database object and every schema operation is automatically synchronized. "
Replication lag can come from several places
A replica can fall behind because:
primary generates changes faster than network can send them
network is slow or congested
replica disk cannot persist logs fast enough
replica CPU cannot apply changes fast enough
long queries interfere with apply/redo
replication process is paused
replica is disconnected
These produce different diagnostic signatures.
For example, SQL Server explicitly distinguishes log hardening from redo: a secondary can receive and harden changes faster than redo can apply them, creating a recovery/redo queue.
PostgreSQL similarly exposes separate received/flushed/replayed WAL positions, making it possible to distinguish network/receive delay from replay delay.
Read replicas can make themselves slower
Suppose a replica is intended for large reporting queries. For example, the replica may have these orders:
| order_id | customer_id | amount |
|---|---|---|
| 104 | 27 | 89.00 |
| 105 | 27 | 35.00 |
| 106 | 28 | 120.00 |
A report can aggregate them by customer:
SELECT
customer_id,
COUNT(*) AS order_count,
SUM(amount) AS total_amount
FROM orders
GROUP BY customer_id
ORDER BY customer_id;
| customer_id | order_count | total_amount |
|---|---|---|
| 27 | 2 | 124.00 |
| 28 | 1 | 120.00 |
Those queries consume:
CPU
memory
I/O bandwidth
But the same replica also needs CPU and I/O to apply incoming replication changes.
So:
more reporting work
↓
less resource available to replication apply
↓
replica falls further behind
↓
reports become even more stale
SQL Server explicitly documents that heavy reporting workloads on readable secondary replicas can compete with redo and increase replication latency.
This is why:
"we'll move reads to replicas"
is not automatically free scalability.
Failover
Failover changes which node accepts primary/write responsibility.
Before:
Primary A
|
v
Replica B
A fails.
After failover:
X Primary A
Primary B
This sounds simple, but several things must happen correctly:
select a suitable replacement
ensure it has an acceptable data position
promote it
route application traffic to it
prevent the old primary from continuing to accept writes
Replication by itself does not perform all of those steps safely.
Promotion
A replica normally behaves differently from a primary.
For example, a PostgreSQL physical standby continuously replays WAL.
Promotion tells it:
stop behaving as a recovery standby
become a normal read/write primary
PostgreSQL supports promotion through pg_ctl promote or pg_promote().
After promotion, applications must also know where to connect.
That may require:
load balancer
proxy
VIP
DNS
cluster manager
service discovery
connection-string listener
depending on the platform.
SQL Server Availability Group listener
SQL Server Availability Groups provide the concept of an availability group listener.
Applications connect to the listener rather than hard-coding the current primary machine.
After failover, the listener allows client connections to be routed toward the replica currently holding the primary role. SQL Server can also use read-only routing for appropriate read-intent connections.
This solves an important layer beyond merely copying data:
"Which server should my application connect to now?"
RPO: how much data can we lose?
The Recovery Point Objective (RPO) describes the tolerated data-loss window.
Suppose asynchronous replication normally lags by:
2 seconds
If the primary dies permanently, the surviving replica may be approximately two seconds behind.
The business might therefore lose recently acknowledged transactions.
A system requirement might say:
RPO = 0
meaning acknowledged committed transactions must not be lost under the failures covered by the design.
Another system may accept:
RPO ≤ 60 seconds
for a low-value reporting database.
The replication architecture should follow the business requirement, not the reverse.
RTO: how long can the system remain unavailable?
The Recovery Time Objective (RTO) concerns time rather than data:
failure
|
v
detect failure
choose new primary
promote/recover
redirect clients
application becomes usable
|
v
elapsed time = recovery time
A company might require:
RTO < 30 seconds
while another system might tolerate:
RTO = 4 hours
Those lead to very different architectures.
Replication improves the potential RTO because a mostly up-to-date database already exists.
But automatic detection, promotion, routing, replay backlog, and application reconnection all affect actual recovery time.
Replica apply lag can hurt RTO too
Suppose a failover replica has safely received large amounts of transaction log but has not replayed all of it.
The data may be durable:
good RPO
but recovery may still need to process the backlog before the database becomes fully useful:
worse RTO
SQL Server specifically documents that a large redo queue on a secondary can extend failover time even when those log records have already been hardened there.
Again:
durability
and:
readiness
are different properties.
Split brain
One of the most dangerous failover failures is split brain.
Suppose network communication between two sites fails:
Site A Site B
Primary A X Replica B
A can no longer see B.
B can no longer see A.
If both sides independently decide:
"The other side is dead.
I should be primary."
you can get:
Primary A Primary B
accepts write X accepts write Y
Now the system has two conflicting histories.
That is much more difficult than ordinary replica lag.
Fencing
Safe failover therefore requires more than promotion.
The old primary must be prevented from continuing to process writes if it is no longer authoritative.
That exclusion is commonly called fencing.
Conceptually:
1. Decide B will become primary.
2. Ensure A cannot continue writing.
3. Promote B.
4. Route clients to B.
Possible fencing mechanisms depend on the environment and can involve:
cluster quorum
storage fencing
node power control
leases
STONITH-style mechanisms
cloud control planes
consensus-based ownership
The architectural principle is:
Never rely solely on "we think the old primary is dead. "
You need a mechanism ensuring only the authorized writer remains active.
SQL Server shows why quorum matters
SQL Server synchronous Availability Group automatic failover is not based only on:
secondary can no longer ping primary
Microsoft documents additional requirements including synchronous state, configured automatic failover, WSFC quorum, and failover-policy conditions.
This is an example of a general distributed-systems principle:
Failure detection alone is not enough to safely assign write ownership.
You also need coordination about who has authority.
Asynchronous failover can lose acknowledged writes
Suppose:
T1 committed on Primary A
Primary A told application:
SUCCESS
T1 had not reached Replica B
Primary A permanently fails
Replica B promoted
The new primary never contained T1.
This is why forced failover from an unsynchronized asynchronous SQL Server secondary explicitly permits data loss.
The same fundamental risk exists in asynchronous PostgreSQL and MySQL replication.
Asynchrony trades some durability guarantee for lower commit latency and increased geographic flexibility.
Synchronous replication changes the failure tradeoff
Suppose the commit rule instead requires:
Primary local durable write
AND
Replica durable write
before
SUCCESS
Then loss of only the primary does not lose the acknowledged transaction.
But now consider:
replica unavailable
What should happen?
Possibilities include:
stop accepting commits
wait indefinitely
wait for another synchronous replica
fall back to asynchronous operation
Different systems/configurations choose differently.
That is why synchronous replication also creates an availability tradeoff.
Durability versus availability during network partitions
Consider:
Primary A --------X-------- Synchronous replica B
The network link fails.
A now has to choose between two desirable properties:
keep accepting writes
and:
guarantee every acknowledged write exists on B
It cannot do both using B while communication is impossible.
A strict design may stop commits.
A looser design may continue asynchronously.
The important point is:
Replication configuration is partly a business decision about what to sacrifice during failures: latency, availability, or risk of data loss.
Replication is not backup
Suppose an administrator accidentally runs:
DELETE FROM customers;
Replication faithfully copies the change.
Soon:
Primary:
customers deleted
Replica A:
customers deleted
Replica B:
customers deleted
Replication succeeded perfectly.
Yet the data is gone everywhere.
That is why:
multiple current copies
is not the same as:
historical recoverability
Corruption and bad application writes can replicate too
Other examples:
UPDATE accounts
SET balance = 0;
without a WHERE clause.
Or an application bug writes:
customer_email = NULL
for millions of customers.
Or malicious credentials delete records.
Replication may propagate those changes to every healthy replica.
A replica protects against some infrastructure failures.
It does not automatically protect against:
logical corruption
operator mistakes
application bugs
malicious valid writes
Backup solves a different problem
A backup preserves an earlier recoverable state.
For example:
Monday backup
Tuesday backup
Wednesday backup
+
transaction/WAL/binlog archive
may allow restoration to:
Wednesday 14:37:12
before an accidental deletion at:
14:37:15
This is point-in-time recovery.
Replication provides:
another current copy
while backups and retained logs provide:
historical recovery points
A resilient system often needs both.
Replica, backup, and failover are different concepts
| Mechanism | Main question |
|---|---|
| Replica | Do I have another copy receiving current changes? |
| Backup | Can I restore an earlier state? |
| Failover | Can another node safely become the writer? |
| Point-in-time recovery | Can I restore to a chosen moment? |
| Read replica | Can I offload some read workload? |
Having one does not prove that the others are properly implemented.
For example:
three replicas
does not answer:
Can I recover data deleted last Tuesday?
Replication is not sharding
These concepts are also frequently confused.
Replication
Node A:
customers 1-1,000,000
Node B:
customers 1-1,000,000
The nodes contain overlapping copies.
Sharding
Shard A:
customers 1-500,000
Shard B:
customers 500,001-1,000,000
Different nodes own different subsets.
Replication solves:
copying / availability / read scaling
Sharding solves:
partitioning data or write load
They can be combined:
Shard 1
primary
replica
Shard 2
primary
replica
Shard 3
primary
replica
Read scaling does not scale writes
Suppose:
Primary
|
+--- Replica A
+--- Replica B
+--- Replica C
You can distribute:
SELECT
traffic across several replicas.
But if all writes still go through one primary:
INSERT
UPDATE
DELETE
write throughput is still constrained by the single-writer path.
Adding read replicas does not magically partition write load.
That is one reason multi-primary systems and sharding exist—but they introduce much harder conflict and coordination problems.
Cascading replication
Replicas do not always connect directly to the primary.
A topology can be:
Primary
|
v
Replica A
|
+------> Replica B
|
+------> Replica C
This is cascading replication.
It can reduce:
connections to primary
cross-site bandwidth
but adds another dependency.
If A falls behind, B and C cannot be more current than the data they receive through A.
PostgreSQL supports cascading physical streaming replication, where one standby can stream WAL to downstream standbys. PostgreSQL currently documents that cascading replication itself is asynchronous.
Monitoring replication
A production replication system should answer at least:
| Question | Why it matters |
|---|---|
| Is the replica connected? | Disconnected replicas stop advancing |
| What position has the source produced? | Defines the latest available state |
| What position has the replica received? | Measures transport lag |
| What has the replica persisted? | Measures durability progress |
| What has the replica applied/replayed? | Measures query freshness |
| Is lag growing or shrinking? | Distinguishes transient delay from overload |
| How much log is retained for the replica? | Prevents source disk exhaustion |
| Is the replica eligible for failover? | Avoids promoting a bad candidate |
| How large is the redo/apply queue? | Affects freshness and potentially RTO |
Monitoring only:
"replication process = running"
is not enough.
Concrete monitoring concepts by engine
| PostgreSQL | MySQL | SQL Server |
|---|---|---|
pg_stat_replication |
Replication connection/status metadata | Availability Group dashboard / DMVs |
| WAL LSN positions | GTID / binary-log positions | received/hardened/redone LSNs |
| receive/flush/replay progress | receiver vs applier progress | send queue / redo queue |
| replication slot state | relay logs | synchronization state |
pg_stat_wal_receiver |
replica I/O/applier state | sys. dm_hadr_database_replica_states |
PostgreSQL documents WAL-position monitoring and replication sender/receiver views. MySQL tracks source and relay-log/applier positions and supports GTID-based progress tracking. SQL Server exposes redo and synchronization information through Availability Group monitoring and sys. dm_hadr_database_replica_states.
Technology summary
| Feature | PostgreSQL physical streaming | MySQL source/replica | SQL Server Availability Groups |
|---|---|---|---|
| Change source | WAL | Binary log | Transaction log |
| Replica transport | walsender → walreceiver |
Source → replica receiver | Primary → secondary |
| Apply mechanism | WAL replay | Replica applier thread(s) | Redo |
| Default/basic mode | Asynchronous streaming | Asynchronous | Async or sync configured per replica |
| Stronger durability mode | Synchronous standbys | Semisynchronous plugin | Synchronous commit |
| Can acknowledgement precede apply? | Yes, except remote_apply |
Yes | Yes |
| Readable replica | Hot standby | Replica can serve reads | Readable secondary |
| Progress identifier | LSN | Binlog position / GTID | Log/LSN state |
| Typical HA use | Physical standby + external orchestration | Source/replica + HA tooling | Built-in Availability Group roles/failover integration |
| Selective logical replication | Built in publication/subscription | Binlog replication can filter; separate group/topology features exist | SQL Server has separate replication technologies besides AGs |
PostgreSQL's current documentation describes asynchronous physical streaming, synchronous standbys, WAL positions, hot standbys, and logical publication/subscription replication. MySQL documents source-to-replica asynchronous replication, semisynchronous replication, relay logs, and GTID positioning. SQL Server documents asynchronous/synchronous Availability Group replicas, readable secondaries, log hardening, and failover modes.
Choosing what each replica is for
A replica should have an explicit job.
For example:
Replica A
synchronous
same datacenter
failover candidate
Replica B
asynchronous
another geographic region
disaster recovery
Replica C
asynchronous
large machine
analytical/reporting reads
Trying to make every replica satisfy every goal can create bad tradeoffs.
A reporting replica may tolerate lag but need substantial CPU.
A failover replica may prioritize:
low lag
durable replication
fast promotion
strict monitoring
over analytical workloads.
A useful design exercise
For each replica, answer:
What is this replica for?
May applications read from it?
How stale may those reads be?
Can it be promoted?
How much acknowledged data may it be missing?
What acknowledgement does the primary wait for?
Does the acknowledgement mean receive, persist, or apply?
How do we know the replica is healthy?
Who decides failover?
How is the old primary fenced?
How do clients discover the new primary?
Where are the historical backups?
If those questions do not have clear answers, merely knowing:
"we have replication"
does not tell you much about the system's actual reliability.
Final mental model
Do not memorize replication as:
Primary copies data to replica.
Use this model:
WRITE ON PRIMARY
|
v
PRIMARY CHANGE LOG
|
v
SEND
|
v
RECEIVE ON REPLICA
|
v
PERSIST / HARDEN
|
v
APPLY / REPLAY
|
v
VISIBLE TO REPLICA READS
Then ask where COMMIT SUCCESS occurs.
Asynchronous
PRIMARY COMMIT
|
SUCCESS TO CLIENT
|
v
SEND → RECEIVE → PERSIST → APPLY
Fast commits, but a primary failure can lose acknowledged transactions that have not reached the surviving replica.
Synchronous durability
PRIMARY
|
SEND
|
REPLICA PERSIST
|
ACK
|
SUCCESS TO CLIENT
Stronger protection, but higher latency.
Synchronous visibility
Some technologies can go further:
PRIMARY
|
SEND
|
REPLICA PERSIST
|
REPLICA APPLY
|
ACK
|
SUCCESS TO CLIENT
PostgreSQL remote_apply is a concrete example.
Finally, keep these four concepts separate:
REPLICATION
another current copy
FAILOVER
make another copy authoritative
BACKUP
retain recoverable historical state
SHARDING
divide different data among nodes
The central replication question is therefore not:
"Do we have a replica? "
It is:
"When we tell the application that a transaction succeeded, where else does that transaction exist, how durable is it there, has it been applied there, and what happens if the current primary disappears at that exact moment? "
That question connects replication lag, synchronous versus asynchronous replication, read consistency, RPO, RTO, and failover into one coherent model.
References
PostgreSQL's current high-availability documentation describes physical WAL streaming, asynchronous and synchronous standbys, hot standby reads, replication slots, cascading replication, promotion, WAL positions, and the different synchronous acknowledgement levels.
PostgreSQL's logical replication documentation describes publications/subscriptions, initial synchronization, logical replication identities, selective replication, and cross-version/platform use cases.
MySQL's current replication documentation describes asynchronous source-to-replica replication, binary/relay logs, GTID positioning, replica initialization, and semisynchronous acknowledgement behavior.
Microsoft's SQL Server Availability Group documentation describes asynchronous and synchronous commit, secondary log hardening, readable secondaries, redo lag, failover modes, listeners, and RPO/RTO-related monitoring.