CPU Utilization Spike Due to I/O Wait During MySQL Backup
The Deceptive Alert of CPU Saturation
For Database Administrators (DBAs) and Site Reliability Engineers (SREs), a high CPU utilization alert is one of the most common production pages. The immediate instinct when seeing a CPU graph pegged at near-maximum capacity is to investigate application queries, look for missing indexes, scale up compute resources, or check for transaction lock waits.
However, high CPU utilization metrics can be highly deceptive. An engine that appears to be consuming 100% of its compute resources might not actually be performing active calculations. Instead, the processor might spend the majority of its cycles completely idle, blocked while waiting for storage subsystems to retrieve data.
In database environments, this phenomenon is particularly common during heavy administrative tasks, such as scheduled daily physical backups. When application queries compete with the massive physical disk read operations generated by backup tools, storage bottlenecks can easily masquerade as compute bottlenecks.
This case study reviews a real-world incident where a MySQL replica server generated a critical CPU utilization alert, spiking to 95% during its scheduled backup window. We will walk through the underlying metrics, system performance baselines, and step-by-step diagnostic workflows that revealed the true root cause: extreme storage contention and high CPU I/O wait rather than actual CPU saturation.
Incident Overview: Environment & Timeline
To understand the context of the incident, we must first examine the operational environment and the exact sequence of events.
System Environment
- Database Engine: MySQL 8.0
- Server Role: Replica
- Storage Medium: Solid State Drive (SSD)
- Backup Utility: Percona XtraBackup
- Backup Method: Full Physical Backup
- Backup Schedule: 02:00 AM Daily
Incident Timeline
The following sequence of events occurred during the early morning backup window:
Initial Symptoms
At first this looked like CPU saturation.
After investigation:
CPU wasn't busy.
CPU was waiting.
3. The Core Metric: What is CPU I/O Wait?
I/O Wait (%wa) is the percentage of time that the CPU remains idle while there are outstanding disk or storage read and write operations. It is a subset of overall idle time, indicating that the processor has tasks ready to execute but is blocked because the required data has not yet been fetched from the storage device into memory.
A high CPU utilization percentage driven primarily by I/O Wait is a classic signature of a storage bottleneck, not compute saturation. If application queries require data that resides on disk, and the disk is fully occupied with other tasks (such as a massive backup job), those queries wait, and the CPU spends its cycles in an idle waiting state.
Under the Hood: How Physical Backups Cause Storage Contention
Physical backup tools like Percona XtraBackup create database copies by reading raw data files directly from disk. To achieve a consistent snapshot, the tool sequentially scans all allocated tablespace files (such as ibdata1, individual .ibd tablespace files like orders.ibd or users.ibd, and active redo logs).
For larger datasets—for instance, an 800 GB database—the backup utility reads nearly the entire volume of data from the underlying storage. This process generates a massive, sustained sequence of read requests. Because the backup is highly parallelized (configured with --parallel=8 in this incident), it consumes the majority of the available Input/Output Operations Per Second (IOPS) and disk read bandwidth.
At the same time, the replica server continues to process application read queries. If a query requests a data page that is not currently cached in the MySQL InnoDB Buffer Pool, MySQL must issue a read request to the same physical disk. Because the storage queue is already saturated with backup reads, the query wait times increase dramatically.
When the backup process shares the same storage as the database, it generates heavy disk reads that compete with application queries for I/O resources. As a result, database requests spend more time waiting for disk access, increasing I/O Wait (%wa). Although overall CPU utilization may appear high (90–95%), the CPU is primarily idle while waiting for storage operations to complete rather than actively processing queries. This storage contention leads to higher disk latency, larger I/O queues, increased active threads, slower query response times, and reduced overall database performance.
MySQL Issues a Disk I/O Request
Application Query :
SELECT * FROM orders WHERE customer_id=100;Required page :
orders.ibdBut storage is already busy serving backup reads. So MySQL waits.
Observed System Metrics
CPU Utilization
Value: 95%
Description: Overall CPU usage reached 95%, but most of it was due to I/O Wait, indicating the CPU was waiting for disk operations rather than actively processing MySQL workloads.
I/O Wait
Value: 73%
Description: Around 73% of CPU time was spent waiting for storage I/O to complete, confirming that the storage subsystem had become the primary bottleneck during the backup.
Disk Read Throughput
Normal: 80 MB/s
During Backup: 620 MB/s
Description: Disk read throughput increased nearly 8× as the backup tool continuously scanned MySQL data files, consuming a significant portion of the available storage bandwidth.
Disk Latency
Before Backup: 4 ms
During Backup: 55 ms
Description: Disk latency increased from 4 ms to 55 ms, meaning each I/O request took significantly longer to complete, resulting in slower query execution and higher I/O Wait.
Overall Observation
The combined metrics clearly indicate that the backup process saturated the storage subsystem, leading to increased disk latency, higher I/O queues, and elevated CPU utilization driven primarily by I/O Wait, rather than actual CPU processing.
MySQL Metrics
Threads Running
Normal: 15
Peak: 76
Description: The number of active MySQL threads increased from 15 to 76, as queries waited longer for disk I/O to complete during the backup.
Slow Queries
Normal: 0
Peak: 145
Description: Slow queries increased from 0 to 145 because disk read latency increased, causing many queries to exceed the configured long_query_time.
Buffer Pool Hit Ratio
Value: 99%
Description: The Buffer Pool Hit Ratio remained at 99%, indicating that most data requests were served from memory and the issue was not caused by poor cache efficiency, but by storage contention for pages that required disk access.
Lock Waits
Value: None Observed
Description: No lock waits were detected, confirming that query delays were not caused by row-level or table-level locking.
Replication Status
Value: Healthy
Description: Replication remained healthy throughout the backup, with no significant replication lag or SQL thread delays observed.
Deadlocks
Value: None Observed
Description: No deadlocks occurred during the incident, indicating that transaction conflicts were not a contributing factor.
Overall Observation
The MySQL metrics indicate that the database engine itself was functioning normally. The increase in Threads Running and Slow Queries was a direct consequence of storage I/O delays, while the absence of lock waits, deadlocks, and replication issues confirms that the root cause was disk contention introduced by the backup activity, rather than an internal MySQL locking or replication problem.
Sample Linux Evidence
CPU Utilization (top)
Description: The top output shows that only 14% of CPU time was spent executing user processes and 8% on system tasks, while 73% was consumed by I/O Wait, confirming that the CPU was primarily waiting for storage operations rather than processing MySQL workloads.
I/O Statistics (iostat -x 1)
Read Requests per Second (r/s)
Value: 950
Description: The storage device processed approximately 950 read requests per second, indicating an intensive read workload generated by the backup process.
Read Throughput (rkB/s)
Value: 620,000 kB/s (~620 MB/s)
Description: Disk read throughput reached approximately 620 MB/s, showing that the backup continuously scanned MySQL data files and heavily utilized the available storage bandwidth.
Average Wait Time (await)
Value: 48 ms
Description: Each I/O request waited an average of 48 ms before completion, indicating increased storage latency due to heavy disk activity.
Service Time (svctm)
Value: 3 ms
Description: The storage device serviced each individual I/O request in approximately 3 ms, suggesting that the increased response time was mainly caused by queued requests rather than slow hardware.
Disk Utilization (%util)
Value: 100%
Description: The storage device reached 100% utilization, confirming that it was fully occupied processing backup and database read requests with no spare I/O capacity available.
Backup Process
ps -ef | grep xtrabackupxtrabackup
--backup
--parallel=8Description: The process list confirmed that Percona XtraBackup was running with 8 parallel threads, generating sustained disk read operations that saturated the storage subsystem during the backup window.
Overall Observation
The Linux system metrics clearly indicate that the server was not CPU-bound. Instead, the backup process generated a high volume of disk reads, driving storage utilization to 100%, increasing I/O wait and disk latency, and ultimately causing the observed CPU spike due to storage contention.
MySQL Processlist
The SHOW FULL PROCESSLIST command was used during the incident to identify the state of active MySQL sessions while the backup was running.
SHOW FULL PROCESSLIST;Id Command Time State
235 Query 38 Sending data
241 Query 42 Sending data
248 Query 30 Waiting for diskQuery State
State: Sending data
Description: Multiple sessions were in the Sending data state, indicating that MySQL was actively retrieving rows for client queries. Although the name suggests data was being sent to clients, these queries were also waiting for data pages to be read from disk before they could continue processing.
Query Execution Time
Observed: 30–42 seconds
Description: Several queries had been running for 30–42 seconds, significantly longer than their normal execution time, indicating delays caused by slow disk I/O rather than CPU or lock contention.
Waiting for Disk
State: Waiting for disk
Description: One or more sessions entered the Waiting for disk state because the required data pages were not immediately available in memory and the storage subsystem was already busy servicing backup read requests.
Processlist Observation
Description: The process list showed an increasing number of active sessions waiting for data pages to be fetched from disk. Since the backup process was generating continuous sequential reads, application queries experienced longer wait times before their I/O requests could be serviced.
Why "Sending data" Doesn't Always Mean Network Activity
Description: Despite its name, the Sending data state does not necessarily mean MySQL is transmitting results to the client. It often indicates that MySQL is scanning tables, reading index pages, fetching rows from storage, or processing result sets. During the backup window, many queries remained in this state because they were waiting for disk reads to complete.
Overall Observation
The process list confirmed that MySQL itself was healthy and actively processing client requests. However, many sessions remained in Sending data or Waiting for disk for an extended period because the backup process had saturated the storage subsystem. This resulted in delayed query execution, increased Active Threads, and higher CPU I/O Wait, even though there were no lock waits, deadlocks, or replication issues.
Root Cause
- The backup process generated sustained sequential disk reads.
- Storage bandwidth became saturated.
- Application queries had to wait for disk access.
- CPU accumulated I/O Wait time.
- Monitoring therefore reported high CPU utilization, although the processor itself was not executing significant work.
Investigation Using PMM (Percona Monitoring and Management)
The following steps were performed in PMM to identify the root cause of the CPU utilization spike during the backup window.
Step 1: Verify CPU Utilization
PMM Dashboard:
Home
└── Nodes Overview
└── CPU UsageObserve the following metrics:
- CPU Utilization (%)
- User CPU
- System CPU
- I/O Wait
Finding:
CPU Utilization : 95%
I/O Wait : 73%Observation:
Although CPU utilization was high, most of the CPU time was spent in I/O Wait, indicating that the processor was waiting for storage operations rather than executing MySQL queries.
Step 2: Check Disk Performance
PMM Dashboard:
Node Summary
↓
Disk PerformanceObserve the following metrics:
- Disk Read Throughput
- Disk Write Throughput
- Disk Utilization
- Disk Latency
Finding:
Disk Reads : 620 MB/s
Disk Utilization : 100%
Latency : 55 msObservation:
Disk read throughput increased significantly after the backup started, saturating the storage subsystem.
Step 3: Analyze Disk Queue
PMM Dashboard
Node Summary
↓
Disk I/OObserve the following metrics:
- Queue Depth
- Read IOPS
- Read Requests
Finding
Queue Depth
Normal : 1
Peak : 27Observation
The storage device accumulated pending I/O requests because it could not process incoming read operations quickly enough.
Step 4: Verify MySQL Activity
PMM Dashboard
MySQL Overview
↓
MySQL ActivityObserve the following metrics:
- Threads Running
- Questions
- Queries
- Connections
Finding
Threads Running
Normal : 15
Peak : 76Observation
The increase in running threads indicated that application queries were waiting longer for storage I/O.
Step 5: Review Slow Queries
PMM Dashboard
MySQL
↓
Query Analytics (QAN)Observe the following metrics:
- Slow Queries
- Query Response Time
- Top Queries
Finding
Slow Queries
Peak : 145Observation
The increase in slow queries coincided with the backup window, confirming that higher disk latency affected query execution.
Step 6: Verify InnoDB Metrics
PMM Dashboard
MySQL InnoDB Details
Observe the following metrics:
- Buffer Pool Hit Ratio
- Buffer Pool Reads
- Buffer Pool Size
Finding
Buffer Pool Hit Ratio
99%Observation
The Buffer Pool remained highly efficient, indicating that the issue was not caused by poor cache performance but by the small percentage of requests that required disk access.
Step 7: Correlate with Backup Window
PMM Dashboard
Annotations / Timeline / compare the graphs around the backup schedule.
Observation
At approximately 02:00 AM, the backup process started. Immediately after:
- Disk Reads increased
- Disk Utilization reached 100%
- Disk Latency increased
- I/O Wait increased
- Threads Running increased
- Slow Queries increased
All metrics returned to normal shortly after the backup completed.
Root Cause Identification Workflow
Resolution
- Backup completed successfully.
- Disk utilization returned to normal.
- I/O Wait decreased.
- Active Threads reduced.
- Query response times normalized.
No MySQL restart was required.
Recommendations
Short Term
- Schedule backups during periods of lower application traffic.
- Reduce backup parallelism if storage becomes saturated.
- Continuously monitor iostat, disk latency, and CPU I/O Wait.
Long Term
- Use faster storage (NVMe/Provisioned IOPS) for backup-intensive workloads.
- Offload backups to a dedicated replica to isolate production traffic.
- Consider incremental backups where appropriate to reduce read volume.
- Review backup scheduling to avoid overlap with maintenance or reporting jobs.
Final Observations
- The incident was a classic example of storage contention during backup activity, not CPU saturation.
- High CPU utilization alone is insufficient to determine the root cause of a performance issue.
- Correlating CPU metrics with storage metrics such as I/O Wait, disk latency, queue depth, and disk utilization provides a more accurate diagnosis.
- Regular monitoring of system and MySQL performance metrics enables faster root cause identification and reduces troubleshooting time.
- Proper backup scheduling and storage capacity planning are essential to minimize performance impact in production environments.
Are you struggling with performance slowdowns, high I/O wait, or replica lag during your backup windows? Our MySQL database engineering specialists are here to assist. At Mydbops, we provide proactive MySQL Managed Services and performance tuning to analyze your query workloads, optimize your backup parameters, and ensure your system maintains reliable recovery paths without impacting production traffic.
.avif)
.avif)
.avif)
.avif)
.svg)
