When a production MySQL instance abruptly terminates, the standard response is to search inside the database: did a runaway analytical query consume all available memory? Did an unindexed schema migration trigger table lock contention? Did a client application flood the connection pool?

Recently, one of our managed production clusters monitored by our DBA Managed Services team, faced an unexpected outage where the database was entirely blameless. Queries were running normally, traffic was standard, and CPU and memory usage showed no spikes. Yet, InnoDB’s internal watchdog decided the server was hung and intentionally forced an abort.

The actual culprit was one layer down in the operating system: fstrim, a routine Linux disk maintenance job. A burst of block discard requests saturated the disk controller long enough to trip InnoDB's safety limits, as outlined in the MySQL 8.0 Reference Manual: InnoDB Troubleshooting.

Here is the step-by-step investigation of how this happened, why thick-provisioned storage made it worse, and how to prevent it in your environment.

The Incident: A Fatal Semaphore Abort

Early on a Monday morning, a production MySQL server running Percona Server 8.0.33-25 on VirtIO block storage stopped serving traffic. The error log did not point to an Out-Of-Memory (OOM) killer event or a host-level hypervisor crash. As documented in MySQL Error Handling and Fatal Aborts, InnoDB had chosen to terminate itself:

[ERROR] [MY-012872] [InnoDB] [FATAL] Semaphore wait has lasted > 600 seconds.
We intentionally crash the server because it appears to be hung.
mysqld got signal 6 ;

The stack trace confirmed that srv_error_monitor_thread called ib::fatal.

MySQL has an internal watchdog thread that periodically inspects sync primitives (mutexes and read-write locks). If a thread waits on an internal semaphore for longer than 600 seconds (10 minutes), InnoDB assumes that code execution is deadlocked or broken. Rather than risking data corruption or hanging indefinitely in an undefined state, it raises SIGABRT to trigger a crash recovery on restart.

The question was: What kept an InnoDB thread blocked for 10 full minutes?

The Investigation: Working Down the Stack

Step 1: Rule Out Database-Level Issues

We started by reviewing activity immediately preceding the crash:

• Running queries showed normal execution plans and execution times.
• Connection counts were well within configured thresholds.
• InnoDB row lock waits, transaction lists, and throughput metrics were clean.
• No Data Definition Language (ALTER TABLE) statements were running.

Result: nothing stood out. No runaway query, no unusual load, no obvious lock contention. The database had no visible reason to be stuck, so the cause had to be outside it.

Step 2: Check Host and Scheduled Jobs

With the database ruled out, we moved to the OS-level metrics and to anything else that ran on the host around thesame time. The systemd journal showed a job that lined up exactly with the incident: fstrim.service, started bythe weekly fstrim.timer. It was sending discard requests for several TiB of free blocks to the very volumes InnoDB reads from and writes to:

00:54:17 systemd: Starting Discard unused blocks on filesystems from /etc/fstab...
01:59:25 fstrim: /logs: 457.6 GiB trimmed on /dev/vdd
01:59:25 fstrim: /data_mysql02: 1.6 TiB trimmed on /dev/vdc
01:59:25 fstrim: /data_mysql01: 3.2 TiB trimmed on /dev/vdb
01:59:25 systemd: fstrim.service: Deactivated successfully. (UTC, abridged)

Step 3: Line up the timeline

fstrim started at 06:24 IST (00:54 UTC). InnoDB's 600 s counter shows the stall began no later than 01:31:34 UTC,about 37 minutes into the job, and the abort came 47 minutes after it started, squarely inside fstrim's active window. In other words, InnoDB threads were waiting on disk I/O that fstrim's discard burst had stalled. The job finished at 01:59:25, while the server was still starting back up.

That was the lead. The next question was what fstrim actually does, and how it could stall disk I/O enough to hurt adatabase.

Digging Deeper: What fstrim Actually Does

fstrim is a Linux utility that tells the underlying storage which filesystem blocks are no longer in use. When a file is deleted, the filesystem only marks its blocks free; the storage below has no idea. fstrim sends discard (TRIM/UNMAP)requests for those free ranges so the backend can reclaim them.

sudo fstrim -v /data

When it is useful: Only when the storage can act on it: thin-provisioned volumes, thin LVM pools and flash media. On thin storage, physical capacity is allocated on demand, so reclaiming deleted blocks frees real pool space. On thick(fixed) provisioning the capacity is already allocated, so there is nothing to give back.

How it works: DISC-GRAN and DISC-MAX

You can verify whether a Linux system accepts discard requests by inspecting the parameters detailed in the Linux fstrim(8) Manual.

fstrim asks the filesystem for its free extents, and the block layer cuts them into discard requests using the limits thedevice advertises. lsblk -D shows those limits:

lsblk -D /dev/vdb
NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
vdb 0 4K 2G 0

Root Cause: Why a Disk Cleanup Can Crash MySQL

Think of MySQL's internals as a busy single-lane bridge. Many workers (InnoDB threads) need to cross it to readtable information, update bookkeeping or fetch data pages. To avoid chaos only one worker is allowed on the bridge ata time. That one-at-a-time turnstile is what a mutex or latch is.

Normally each worker crosses in a split second. But if the road surface under the bridge, the disk, suddenly gets busywith a completely different task such as a huge cleanup sending thousands of 'these blocks are free' instructions, the worker on the bridge gets stuck mid-crossing. Everyone else queues behind it.

MySQL watches the clock. If a worker is stuck for more than 600 seconds, InnoDB's internal monitor assumes the server is hung and deliberately aborts instead of continuing in an unknown state.

Inside InnoDB, threads share a few core structures, each guarded by a latch that is meant to be held formicroseconds:

• DICT_SYS: the data dictionary (table and index metadata)• FIL_SHARD: tablespace file bookkeeping (opening, extending, flushing files)
• Buffer-pool latches: guard pages while they are read from or flushed to disk
• Redo log and binlog writes: must be fsynced before a commit can finish

The danger is a thread that holds one of these latches while it waits for disk. When disk latency goes from millisecondsto seconds, that thread cannot release the latch, every other thread needing it must wait on a semaphore, and the queue grows for as long as the disk stays busy. Warnings about long semaphore waits typically start appearing afterabout four minutes; srv_error_monitor_thread pulls the plug at 600 seconds (ib::fatal, signal 6).

Note: the log excerpt we captured shows the abort but not which latch was involved. The names above are the typical candidates for this pattern; the SEMAPHORES section of SHOW ENGINE INNODB STATUS or the long-wait warnings just before a crash would name the exact one.

The trigger: how fstrim stalls disk I/O

In that single run fstrim walked the data and log volumes in turn and discarded about 5.2 TiB of free blocks in 65 minutes, most of it on the MySQL data volumes:

Discards are real I/O. They travel through the same virtual block queue and storage backend as InnoDB's redo-log writes, page flushes and binlog fsyncs, and depending on the filesystem (ext4 or XFS) the trim also works through free-space structures that file extension and log rotation may need.

Even when a discard ends up as a no-op on thick storage, the backend still has to process every request. A multi-TiB burst therefore stalls disk I/O: reads, writes and fsyncs queue behind the discards, and fsync latency goes from milliseconds to seconds. That I/O stall is exactly what jams the bridge

Even when running resilient high-availability setups, such as those detailed in How to Set Up InnoDB Cluster Using MySQL Shell and our guide to Deploying and Managing MySQL InnoDB Cluster, an underlying OS storage freeze will take down a primary node if the maintenance jobs run concurrently.

The Fix: Disable fstrim on Thick-Provisioned Storage

The question was simple: what does fstrim give us on this storage? Our MySQL volumes are thick (fixed)provisioned: the full capacity is allocated up front. Deleting data returns space to the filesystem, but there is no thin pool underneath to hand it back to. The cost-benefit looked like this:

Is it on by default? It depends on the distribution:

What we did. We disabled the timer on the MySQL host; no restart is needed:

sudo systemctl disable --now fstrim.timer # stop and disable the weekly job
sudo systemctl mask fstrim.timer fstrim.service # optional: prevent re-enabling
systemctl list-timers --all | grep fstrim # verify nothing is scheduled
grep -r fstrim /etc/cron* 2>/dev/null # make sure no cron job runs it

Disk capacity is now managed through filesystem housekeeping, log and backup cleanup, utilization monitoring, andstorage expansion when required.

When to keep fstrim: on thin-provisioned or flash-backed storage where reclaim matters, run it in an off-peakwindow, test on a staging server first, consider throttling (fstrim -m / -l, a lower discard_max_bytes), andalert on I/O latency while it runs.

The Result

Since the timer was disabled we have not seen another InnoDB semaphore-wait abort. Nothing was lost by turning it off, because on fixed-provisioned storage fstrim was never reclaiming anything. Throttling it instead was not worth theeffort: with nothing to reclaim, even a throttled run is pure overhead.

Key Takeaways

• Don't stop at the database: when queries and database metrics look clean, move to OS-level metrics andscheduled jobs. The cause may be one layer down.
• Check the schedule: an event that lines up with an early-morning window deserves a look at systemctllist-timers and the systemd journal.
• Maintenance jobs can stall I/O: a multi-TiB discard burst from fstrim competes with InnoDB's own reads andwrites. Watch disk I/O latency during scheduled maintenance windows.
• Discard support is not usefulness: non-zero DISC-GRAN / DISC-MAX means fstrim can run, not that it should. Only thin or flash-backed storage benefits.
• Match maintenance to storage type: on thick-provisioned database volumes, disable the weekly fstrim and manage capacity directly.

Final Thoughts

When a database shuts itself down, the instinct is to look inside the database. Here the answer was one layer down: a housekeeping job that is enabled by default on many Linux distributions and is perfectly sensible on thin storage, but only stalled disk I/O on ours. Rule out the workload first, then check what else is scheduled on the host and whether it actually benefits your storage.

Prevent Infrastructure-Level Outages with Mydbops

Unplanned database downtime often stems from the interaction between Linux kernel defaults and storage architecture. Mydbops provides full-stack database consulting, infrastructure audits, and 24/7 managed services to ensure your production MySQL clusters remain resilient under any workload.