.avif)
.avif)
TiDB horizontal scaling relies on automatic sharding, but it only works when you add capacity to the saturated layer. Use this 2026 runbook to separate TiKV write pressure from TiDB SQL pressure, a hot Region, or an HTAP workload before changing cluster topology.
TL;DR
- How to scale TiDB horizontally starts with classifying the bottleneck; adding nodes is not the first decision.
- Add TiKV capacity for storage or write pressure; add TiDB capacity for SQL compute and connection pressure.
- A hot Region needs key-distribution or workload work, not another generic scale-out step.
- Use TiFlash only after identifying analytical reads that compete with transactional traffic.
Why a scale-out can fail while the cluster gets bigger
TiDB is a MySQL-compatible distributed SQL database that separates SQL processing, distributed storage, and analytical replicas. That separation is useful only when the operating team identifies which layer is constrained. A larger cluster with the same overloaded TiKV leader, skewed key range, or expensive query plan is still a slower production system.
For revenue-critical systems, Mydbops TiDB Remote DBA services can protect the write path, preserve replica placement, and measure the result against a baseline.
The four bottlenecks that look like a capacity problem
Before opening a topology file or a cloud console, classify the incident. The same symptom, such as rising latency, can come from four different causes.
1. TiKV saturation: storage or write throughput is full
Choose this path when TiKV CPU, disk latency, write stalls, or store capacity are the limiting signals. TiKV stores the data and serves distributed transaction work, so adding TiDB SQL nodes does not relieve a TiKV store that is already saturated.
2. TiDB saturation: the SQL or connection tier is full
Choose this path when TiKV is healthy but TiDB servers show CPU pressure, connection queueing, slow parsing, or expensive query execution. TiDB nodes are stateless SQL servers, which makes this the cleanest horizontal change when the storage tier has headroom.
Adding TiDB nodes behind the existing load balancer increases SQL-serving capacity, but it does not repair a bad execution plan or a connection storm. Confirm that the load balancer can discover the new nodes and that application connection behavior is safe before routing production traffic.
3. A hot Region: one key range owns the pain
A hot Region requires both hot-region scheduling and a data-distribution response. Automatic sharding does not remove a hot-key access pattern. It often appears when sequential keys, a single tenant, a popular account, or a narrow time range concentrates reads or writes on one Region leader. The cluster can have spare capacity while that leader remains the ceiling.
Treat this as a key-design and access-pattern investigation first. Review hot read and hot write information, identify the table and key range, then decide whether the fix is a Region split, application-side write distribution, query change, or a planned data model change. Adding nodes without changing the skew leaves the workload concentrated.
4. Analytical reads are stealing the OLTP budget
A dashboard, report, or ad-hoc scan can compete with checkout, payments, and other transactional work when both run against TiKV. This is a workload-isolation problem, not proof that the transaction cluster needs more hardware.
TiFlash is the right tool when analytical reads are identifiable, read-only, and costly enough to interfere with the OLTP path. TiFlash replicas are created per table; a successful DDL statement starts replication but does not prove the replica is ready or selected by the optimizer.
What you need before changing topology
Prepare a short change record with the following inputs:
- A stable baseline for application throughput, error rate, and P99 latency under a known transaction mix
- Store-level TiKV CPU, disk, capacity, leader count, Region count, and network activity
- TiDB CPU, active connections, query latency, and slow-query evidence
- PD scheduler and hot Region information
- Load balancer health-check behavior for any new TiDB node
- A current topology map showing zone, rack, and host labels
- A rollback owner and a fixed validation window after the change
Do not use a traffic spike as the baseline. A scale-out should be judged against a comparable workload period, not against a different promotion, batch job, or customer event.
The scale-out runbook
1. Freeze the question you are trying to answer
Write one sentence before the change: "TiKV write capacity is limiting transaction throughput," "TiDB SQL compute is saturated," or "a hot Region is concentrating writes." That sentence determines the change and the pass condition.
Expected outcome: every graph and test result has a defined owner and purpose.
Common mistake: using average latency as the only pass metric. Average latency can improve while P99 latency still damages the transaction path.
2. Expand TiKV only when the data layer needs it
Add TiKV nodes when capacity, write throughput, disk behavior, or broad store-level load indicates a storage-layer limit. PD scheduling moves and rebalances work over time; the goal is not merely to see a new node join, but to confirm that useful Regions, leaders, and workload have moved onto it.
Keep topology labels current as nodes are introduced. Replica placement across zone, rack, and host failure domains matters as much as raw node count. A scale-out that places critical replicas into the same failure domain improves neither resilience nor the operational recovery position.
Check migration activity against application latency while PD rebalances. Aggressive movement can create network and disk pressure during a busy period; leaving it too constrained can keep the new capacity idle. Use the observed migration impact to decide whether to continue at the current rate.
Expected outcome: the pressured TiKV stores lose meaningful load and the new stores carry an increasing share of Regions and leaders.
Common mistake: declaring success when the node appears in the inventory, before its workload distribution changes.
3. Add TiDB nodes when SQL service is the limit
Add TiDB SQL nodes when the storage tier is healthy and the pressure is clearly in connections, SQL execution, or TiDB CPU. Put the nodes behind the production load balancer only after health checks, routing, TLS, and connection behavior are tested.
Mydbops remote DBA teams should record the application pool configuration before adding TiDB nodes. A pool that opens too many connections per application instance can consume the new headroom immediately and obscure the original diagnosis.
Expected outcome: request queueing and SQL-tier saturation fall without a matching rise in TiKV stress.
Common mistake: scaling the SQL tier because user-facing latency is high, without checking whether the expensive work is actually below the SQL layer.
4. Isolate analytical work with TiFlash deliberately
Create TiFlash replicas for the specific tables behind heavy analytical reads, not for an entire schema by default. The documented pattern is ALTER TABLE table_name SET TIFLASH REPLICA count; REPLICA 2 creates two replicas when the topology can support them.
TiFlash replication begins after the table-level change, but the initial copy consumes cluster resources. Track replica progress before deciding that analytics has been isolated. Then use query plans to verify whether the optimizer is selecting TiFlash for the intended read-only workload.
Do not describe TiFlash as a write-scaling solution. It isolates supported analytical reads from the transaction path; it does not replace TiKV capacity planning.
Expected outcome: the targeted analytical query stops competing for TiKV resources while transactional P99 latency remains stable or improves.
Common mistake: creating replicas broadly, then discovering that initial synchronization creates pressure during an already busy period.
5. Resolve hot Regions before adding generic capacity
When hot-read or hot-write evidence points to a narrow key range, stop treating the issue as a cluster-average capacity problem. Identify the workload shape: monotonically increasing keys, a single account, a timestamp range, or a large scan can each create a different concentration pattern.
The repair must match that shape. A Region split can create more distribution boundaries; a write-key redesign can distribute future traffic; a query rewrite can remove the expensive access path. Validate the intended result by checking whether the hot signal moves or disappears, not merely whether the original server becomes quieter.
Expected outcome: the concentrated read or write load spreads across multiple Regions or the triggering query cost falls.
Common mistake: moving workload manually without fixing the access pattern that made the Region hot.
6. Prove the new topology under representative traffic
Repeat the same workload used for the baseline: read-heavy, write-heavy, or mixed. Compare transaction throughput, error rate, P99 latency, TiKV load, TiDB load, and hot Region activity. Keep the comparison scoped to an equivalent traffic pattern.
For a controlled pre-production test, use the production schema shape and a realistic transaction mix. For a production change, use a limited validation period with rollback criteria agreed before routing full traffic. The final pass condition is an improved transaction path, not a higher node count.
Expected outcome: the targeted constraint improves without creating a new bottleneck elsewhere in the stack.
Common mistake: testing only throughput. A higher throughput result with worse P99 latency is not a clean win for interactive transactions.
Troubleshooting by symptom
- A new TiKV node remains lightly loaded: inspect PD scheduling, migration activity, store labels, and whether existing Regions have actually moved. A joined node is not automatically a balanced node.
- P99 latency stays high after TiKV scale-out: inspect hot Region evidence and disk behavior on the former hot stores. This points to skew or I/O rather than a lack of raw cluster capacity.
- TiDB nodes are healthy but requests still queue: review connection-pool behavior, long-running SQL, and whether the downstream TiKV path is now limiting.
- TiFlash replicas exist but the report still reads TiKV: inspect the query plan and table replica status. TiFlash requires a ready replica and a plan that selects it.
- Replica movement hurts production latency: reduce concurrent movement pressure and extend the change window. Protecting the transaction path matters more than completing redistribution quickly.
- A zone-level change creates uneven placement: recheck topology labels and failure-domain policy before adding more capacity. Node count does not guarantee durable replica separation.
Tools that belong in the operating loop
- PD Dashboard: store distribution, scheduler activity, and hot Region investigation
- Grafana and Prometheus: the time-series baseline for TiKV, TiDB, PD, and TiFlash
- Slow-query and execution-plan analysis: proof that SQL is the constraint before adding TiDB nodes
- Load testing: repeatable evidence for throughput and P99 latency before and after a topology change
- TiUP or the approved orchestration path: controlled node lifecycle management for self-managed TiDB
What to do next
If the bottleneck cannot be classified from the current monitoring data, do not start a generic scale-out. Run a focused TiDB consulting review that maps each application symptom to TiKV, TiDB, PD, hot Region, or TiFlash evidence, then choose one reversible capacity change.
FAQ
How do I scale TiDB horizontally without downtime?
Scale TiDB horizontally by adding capacity to the constrained layer while the cluster remains online, then validating workload distribution and transaction latency. The correct layer is TiKV for storage or write pressure, TiDB for SQL and connection pressure, and TiFlash for analytical reads.
Should I add TiKV or TiDB nodes first?
Add TiKV nodes first only when storage, write throughput, disk behavior, or TiKV load is constrained. Add TiDB nodes when TiKV has headroom and the SQL or connection tier is the bottleneck.
Why did TiDB latency stay high after adding nodes?
TiDB latency can remain high after a scale-out when a hot Region, disk bottleneck, expensive query, or connection surge remains unchanged. Check P99 latency and hot Region evidence instead of judging the change by node count.
Does TiFlash improve transaction write throughput?
TiFlash does not increase transaction write throughput directly. TiFlash isolates supported analytical reads so they stop competing with transactional work on the TiKV path.
How do I know whether a TiDB Region is hot?
A TiDB hot Region appears in hot-read or hot-write monitoring as concentrated load tied to a narrow key range or query pattern. Identify the table and access pattern before choosing a split, key-design, or query fix.
How many TiFlash replicas should a table have?
The replica count depends on availability requirements and topology capacity. TiDB supports table-level replica counts such as `REPLICA 2`, but replica readiness and placement must be validated before analytics is moved.
What should I measure after a TiDB scale-out?
Measure the targeted constraint, transaction throughput, error rate, P99 latency, workload distribution, and the next limiting layer. Compare against an equivalent baseline instead of a different traffic period.
Scale capacity only after proving the constraint
The strongest 2026 TiDB scaling decision is often a decision not to add nodes yet. If the evidence points to one hot key range or one analytical query, fix that concentration first; otherwise new capacity simply waits around the same bottleneck.
Get a TiDB scaling review
A cluster health check identifies the active bottleneck before a scale-out change.

.avif)
.avif)

.avif)

.avif)