"

Someone awake when your
InnoDB Cluster loses a node

Mydbops operates MySQL InnoDB Cluster environments end to end: Group Replication, Router, Shell, and ProxySQL. A P1 incident reaches an on-call MySQL specialist inside 15 minutes under a formal SLA. Failover is tested on a schedule rather than discovered during an outage.

Scope a support retainer →
REVIEWED ON Clutch
4.9 · Verified reviews
<15 minP1 response SLA
24×7On-call specialists
n = 2f + 1Fault tolerance
ISO 27001 & 9001Certified delivery
Certified delivery ISO 27001 ISO 9001 AWS Advanced Tier Partner Certified MySQL DBAs
Brands that trust Mydbops
Severity & escalation, published

What happens, in what order,
when your cluster pages us

Support is only as good as the sentence you can hold us to. These are the response commitments in the Mydbops support agreement. Response time is time to a qualified MySQL specialist actively working the incident, not time to an acknowledgement email.

SeverityWhat it means for an InnoDB ClusterResponseCover
P1 · Critical Cluster has lost quorum, no primary is writable, Router is refusing connections, or data integrity is in question 15 minutes 24/7/365
P2 · High A group member is offline or in RECOVERING and the cluster is running without fault tolerance, or replication lag is breaching the agreed threshold Pending sign-off 24/7/365
P3 · Moderate Degraded performance, flow control engaging under write load, a failed backup, or a node needing rejoin with the cluster still fault tolerant Pending sign-off Business hours
P4 · Planned Version upgrades, patching, topology change, capacity work, advisory questions Scheduled with your team Business hours
Note

The 15-minute P1 commitment is published and governed by a formal SLA. P2 and P3 response targets are agreed with your team and finalised in the support agreement during onboarding, so this page states definitions and coverage rather than a number delivery has not yet committed to hold.

T+0 Monitoring fires Group member leaves the group T+15 min Specialist working it Quorum, GTID and Router state checked During You are in the channel Actions narrated as they are taken Service restored Cause identified Rejoin or rebuild, not a restart and hope After Written record Runbook updated for your topology

Diagram 1. The escalation path for a P1 InnoDB Cluster incident. Green marks the steps inside the 15-minute P1 commitment; amber marks the outcome that commitment exists to produce.

What we operate, and where the boundary sits

The whole stack, not just the database nodes

A healthy InnoDB Cluster is four moving parts, and three of them are not the database. Most cluster incidents we are called into are not MySQL faults at all: they are a Router that was never made highly available, a group that was left running on two members, or a setting that was applied at runtime and never persisted to disk. Support that stops at the database nodes stops short of where clusters actually break.

MYDBOPS SUPPORT BOUNDARY Application servers Router co-located here MySQL Router 6446 read/write · 6447 read-only metadata cache, transparent failover ProxySQL, where used query rules, connection pooling Group Replication, single primary fault tolerance n = 2f + 1, so three members tolerate one failure PRIMARY reads and writes ONLINE SECONDARY read-only ONLINE SECONDARY read-only the one that goes OFFLINE Watched continuously member state, primary election, quorum margin GTID consistency, applier queue, flow control events Router metadata refresh and connection routing certification conflicts, XCom message delay backup success, restore rehearsal, disk headroom

Diagram 2. Reference architecture for a single-primary InnoDB Cluster with the support boundary drawn on it. Every box, including Router high availability and the proxy layer, plus the runbook that says what to do with each of them at 3am, sits inside the boundary.

MYDBOPS SUPPORT BOUNDARY

Application servers

Router co-located here

↓

MySQL Router

6446 read/write · 6447 read-only

metadata cache, transparent failover

ProxySQL, where used

query rules, connection pooling

↓

Group Replication, single primary

fault tolerance n = 2f + 1, so three members tolerate one failure

PRIMARY reads and writes
ONLINE
SECONDARY read-only
ONLINE
SECONDARY read-only
the one that goes OFFLINE
Watched continuously
  • member state, primary election, quorum margin
  • GTID consistency, applier queue, flow control events
  • Router metadata refresh and connection routing
  • certification conflicts, XCom message delay
  • backup success, restore rehearsal, disk headroom

Diagram 2. Reference architecture for a single-primary InnoDB Cluster with the support boundary drawn on it. Every box, including Router high availability and the proxy layer, plus the runbook that says what to do with each of them at 3am, sits inside the boundary.

Group Replication

Member states, primary election, quorum margin, GTID consistency, applier queue depth and flow control. We alert on a cluster that has stopped being fault tolerant, not only on one that has stopped serving. A three-member group tolerates exactly one failure, so the second failure is not an emergency to be discovered.

MySQL Router

Configuration, version, metadata refresh, and its own availability. Oracle's MySQL Community Manager has written plainly that Router high availability is the customer's responsibility. On a Mydbops retainer it is ours, and it is inside the boundary drawn above.

MySQL Shell and AdminAPI

All cluster administration through the supported path, plus Mydbops platform tooling built on Shell for backup, recovery and data streaming. No manual surgery where an API exists.

The proxy layer

ProxySQL query rules, connection pooling and read splitting where you run it, and an honest answer about whether you should, in the routing section below.

Anatomy of a 3am incident

What a bad night actually looks like

An engineer who has lived through this described the moment better than any vendor page has: "No reads. No writes. Just silence where your production data used to be." Underneath that silence, the causes repeat. Group Replication settings applied at runtime and never persisted, so a restart brings a node back outside the group. GTID sets that no longer agree. A duplicate server UUID from a cloned VM. A stalled instance whose volumes flipped read-only, taking a second member with it and leaving the group with no fault tolerance at all.

Oracle's own MySQL Community Manager has written that in a split-brain "the MySQL service won't work properly until a human fixes the situation". That sentence is the product. The question a support contract answers is not whether a human will be needed. It is which human, how quickly they arrive, and whether they have seen this failure before.

What we do not do during an incident

Restart nodes to see what happens
Force a quorum without first establishing which member holds the most complete GTID set
Change group_replication_unreachable_majority_timeout during an incident
Case · Ecosmob · InnoDB engine recovery

A catastrophic InnoDB engine corruption, with the billing pipeline still running

Ecosmob's telecom gateway bills from real-time Call Detail Records. A severe InnoDB corruption threatened to halt billing. Rather than taking the system down for a long repair, Mydbops built a parallel path to keep capturing new billing records while the corrupted engine was rebuilt underneath it.

Zero dropped transactions 6.56 TB restoredprov. pending 1200x load reduction on runaway recovery Zero duplicates or data loss on final merge

Ashish Pandya, Associate Manager, Service DevOps, Ecosmob Technologies.quote provenance pending

Router or ProxySQL, answered honestly

Two ways to route traffic, and when each one is right

Percona's Marco Tusa tested MySQL Router 8.2 and concluded it "is still far, far away from being competitive with ProxySQL". That assessment deserves a straight answer from anyone selling support for both.

MySQL Router

The supported path, fewest moving parts

Ships with the cluster, understands cluster metadata natively, and is the right choice when you want the fewest moving parts and your routing needs are read/write splitting on ports 6446 and 6447.

Its weakness is its own availability: run it on the application servers, not as a shared tier, or you have converted a highly available cluster into a single point of failure with extra steps.

ProxySQL

When the workload needs more

The right choice when you need query rules, connection multiplexing at scale, fine-grained read weighting, or query rewriting during a migration.

It costs you a component to operate, monitor and upgrade, and that cost belongs inside the support contract rather than on your team.

We support both. We will tell you which one your workload actually needs, including when the answer is the one that makes our job easier rather than harder.

Case · Fast-scaling fintech on MySQL RDS · Routing

Single-node limits and read-heavy traffic choking the primary, rebuilt with ProxySQL in front of Multi-AZ

Near-zero downtime with automatic failover 3x query speed with optimised routing 60% CPU reduction on the primary after read offloading 80% less operations effort
What round-the-clock coverage actually costs

The arithmetic of covering 3am with employees

A week has 168 hours. One DBA covers about 40 of them. Everyone selling support knows this and almost nobody writes it down, so here it is with sources.

The UK consultancy DSP publishes an itemised cost for one in-house DBA reaching £77,180 a year for business-hours administration only, and its own conclusion is that true 24/7 cover "would require at least three DBAs" [3]. In the United States, Solvaria cites a median database administrator wage of $104,620 as of May 2024, and a fully loaded cost of $145,000 to $208,000 once benefits are added [4].

Then there is the part no budget line captures. As Command Prompt observes of database on-call, "Burnout doesn't come from handling emergencies well. It comes from handling the same emergencies repeatedly."

Hours of the week covered, out of 168 One DBA 40 hrs, business hours only, no cover for leave Two DBAs 80 hrs, still nobody awake at 3am on a Sunday Three DBAs 120 hrs, thin rota, no depth Mydbops retainer 168 hrs, on-call rota, escalation path, MySQL specialists only 0 hrs 168 hrs

Diagram 3. Coverage arithmetic. Amber is the coverage a full-time hire provides; the dark bar is the coverage a retainer provides. Hiring figures published by DSP (UK) and Solvaria (US). The chart states hours, not prices, so it stays true regardless of the price-band decision.

A retainer is not always cheaper than hiring. If your cluster is stable, your team already has MySQL depth, and your incidents happen during office hours, one good hire may serve you better. A retainer wins when the failure you are insuring against is rare, expensive, and arrives at an hour when your rota has nobody with InnoDB Cluster experience in it.price band pending

Taking over a cluster that is already unwell

Inherited a cluster nobody fully understands?

Most clusters we take over were built once, by someone who has since left, and have been running on assumptions ever since. Onboarding starts by finding out what is actually true.

First

The facts

Member states and history, quorum configuration, GTID consistency across members, Router configuration and version, whether persisted settings match running settings, backup validity tested by restore rather than by log file, and whether the group is genuinely fault tolerant today.

Then

The fixes, in priority order

Split-brain exposure and misconfigured quorum first. Flow control tuning where write stalls are being caused by it. Read/write splitting corrected at the Router or proxy layer. Version floor and patch gaps closed on a plan agreed with your team.

Then

The handover that makes us replaceable

A runbook written for your topology, monitoring you can see, and alert thresholds you agreed to rather than inherited.30-day plan durations pending

Planned change, and the upgrade you keep postponing

Version upgrades and patching, without the maintenance window you fear

Security patches and minor version upgrades are core scope, not a project. Major versions are planned with your team, rehearsed in staging, and executed as a rolling update through the cluster during a window you choose. On a healthy three-member group, that is what the topology is for.

Case · Paystack · Planned change at scale

Eleven production databases modernised without a single incident

When Aurora MySQL 2 reached end of life, Paystack, a Stripe-acquired platform processing payments for over 200,000 businesses, faced a hard deadline and an unacceptable risk. Mydbops co-engineered the migration across all eleven production databases.

100% uptime across 11 clusters Zero dropped queries during the entire window 50 TB of storage reclaimed $540K annual cost avoidedprov. pending
Proof layer

InnoDB Cluster work, named and measured

Published Mydbops engagements on MySQL, not borrowed proof from another engine. Every figure is either published by Mydbops or held pending provenance and marked as such.

Telecom · InnoDB engine recovery

Ecosmob keeps billing live through a catastrophic InnoDB corruption

Zero
Dropped transactions while the corrupted engine was rebuilt underneath a running billing pipeline

A parallel capture path kept new Call Detail Records flowing while the engine was restored, with no duplicates or data loss on the final merge.

Read full case study →
Digital platform · RDS MySQL · Tested HA

Optimising RDS for high availability with Route53 across Multi-AZ

100%
Availability maintained even during failover testing

The read path was rebuilt with Route53 endpoints across Multi-AZ, then the failure case was tested rather than assumed. Query execution 3x faster, CPU from over 60% to below 20%.

Read full case study →
Fintech · RDS MySQL · ProxySQL routing

A fast-scaling fintech offloads read traffic with ProxySQL on Multi-AZ

60%
CPU reduction on the primary after read offloading, with near-zero downtime

Single-node limits and read-heavy traffic were choking the primary. A ProxySQL layer in front of Multi-AZ delivered 3x query speed and 80% less operations effort.

Read full case study →
Payments · Aurora MySQL · Planned change

Paystack modernises eleven production databases without a single incident

11
Production clusters migrated at 100% uptime, zero dropped queries during the window

With Aurora MySQL 2 at end of life and a hard deadline, Mydbops co-engineered the migration across every production database with no incident.

Read full case study →

The stability of our platform and our disaster recovery posture improved measurably once Mydbops took operational ownership of the databases behind it.

Teresa
Teresa CDMON

What stands out is the ongoing support cadence. It is not a ticket you file and wait on; it is a team that stays close to how our systems actually run.

Henry Suryawirawan
Henry Suryawirawan Flip

Mydbops has been a dependable partner in keeping our database clusters healthy and performing as we scale.

Shital Mehta
Shital Mehta Ola

We have worked with Mydbops over a long relationship, and that continuity is exactly what you want from the team responsible for your databases.

Sunil Kumar
Sunil Kumar Shiprocket
Common questions

InnoDB Cluster support, answered

Our support is governed by a formal Service Level Agreement. When your MySQL InnoDB Cluster has a critical issue, an on-call specialist is alerted immediately and begins active resolution, day or night. Our guaranteed response time for P1 (critical) incidents is 15 minutes, where response means a qualified MySQL specialist working the incident, not an acknowledgement email.
P1 critical incidents carry a published 15-minute response commitment under 24/7/365 cover. P2 and P3 targets, and their coverage windows, are agreed with your team and written into the support agreement during onboarding; P4 planned work is scheduled with you. We publish severity definitions rather than a single headline number so you know exactly which failures map to which commitment.
We commit to measuring failover on your topology and publishing the result, rather than quoting an unmeasured number. For context on managed platforms, AWS's own documentation states that RDS Multi-AZ DB instances fail over in 60 to 120 seconds, while Multi-AZ DB clusters fail over in under 35 seconds [1, 2]. A well-tuned InnoDB Cluster with Router in front typically detects and re-routes faster than that, but the honest answer is the figure we measure on your cluster during a controlled failover test.
Group Replication tolerates failures by majority, so n = 2f + 1. Three members tolerate exactly one failure; two members tolerate none, which is the single most common architectural misunderstanding we are called into. For cross-region disaster recovery, InnoDB ClusterSet links clusters across data centres so that a whole-region loss is survivable.
Yes, if you deploy it as a shared tier. Oracle's MySQL Community Manager has stated that "the best place to install the router is the application server", co-located with each app instance so no single Router is a bottleneck. However you run it, Router configuration, version and availability are inside the Mydbops support boundary, not left to your team.
MySQL Router is the supported, lowest-footprint path for read/write splitting. ProxySQL is the right choice when you need query rules, connection multiplexing at scale, read weighting, or query rewriting during a migration. Percona's Marco Tusa found Router 8.2 "still far, far away from being competitive with ProxySQL" for advanced routing. We support both and will tell you which your workload actually needs, including when the simpler answer is the better one.
A week is 168 hours and one DBA covers about 40. DSP (UK) itemises one in-house DBA at £77,180 a year for business hours only and concludes true 24/7 cover needs at least three DBAs; Solvaria (US) cites a fully loaded cost of $145,000 to $208,000. A retainer wins when the failure you are insuring against is rare, expensive, and arrives when your rota has nobody with InnoDB Cluster experience awake. If your cluster is stable and your incidents are in office hours, one good hire may serve you better, and we will say so.
Yes. InnoDB ClusterSet links a primary InnoDB Cluster to one or more replica clusters in other regions for disaster recovery. It is worth correcting a widely repeated claim that asynchronous replication between InnoDB Clusters is unsupported: that statement predates ClusterSet, which exists to do exactly this. We design, operate and test ClusterSet failover as part of the retainer.
Yes, with the boundary stated clearly for each platform. It is worth noting that Oracle's own MySQL HeatWave high availability uses Group Replication under the hood, so the question is never InnoDB Cluster versus managed cloud, it is who operates Group Replication for you. Several of our published engagements run on RDS and Aurora Multi-AZ with Router or ProxySQL in front.
Yes, and this is the most common way engagements start. Onboarding begins with the facts: member states and history, quorum configuration, GTID consistency, whether persisted settings match running settings, and whether the group is genuinely fault tolerant today. We fix split-brain exposure and quorum first, then hand you a runbook written for your topology, monitoring you can see, and alert thresholds you agreed to rather than inherited.
Yes. After a critical incident you receive a documented account of what happened, why, how it was resolved, and the preventive actions taken, with the runbook updated for your topology. Failover tests produce a written result too: the measured time to a writable primary, the client-visible error window, and any configuration drift the test exposed.
Yes. We support MySQL Community and Enterprise, Percona Server for MySQL, and clusters built on Group Replication across legacy versions up to the current release. Security patches and minor upgrades are core scope; major-version upgrades are planned with your team, rehearsed in staging, and executed as a rolling update during a window you choose.
Let's talk

Get a MySQL specialist on your next 3am node failure

Tell us about your cluster: the topology, the versions, and where it hurts. We will map your severity profile to a support model with a published SLA, and be ready before the next incident.

Scope a support retainer → Cluster down now? Start an escalation →
1Share your cluster topology & versions
2Scoping call with a MySQL specialist
3SLA agreed & cover live in days
Certified MySQL DBAs · under-15-minute P1 response SLA · ISO 27001 & ISO 9001 certified delivery
Services
Open Source Database Management
DBA Managed Services
Database Support Services
Database Consulting Services
Performance & Security Audit
Remote DBA Services
Cloud Cost Optimisation
RDS MySQL Upgrade/Migration Services
RDS MySQL 8.0 EOL Surcharge Calculator
MSSQL DBA Services
DocumentDB Migration Services
Oracle & MSSQL to PostgreSQL Migration
Oracle Services
Valkey & Redis Services
Data Engineering & BI
Managed Data Pipelines
Resources
Blogs
Webinars
Case Studies
Podcasts
Meetups
Solutions
Black Friday Database Support
PACMAN – DB Archival Tool
Cyber 5 Database Support
Industries
Fintech
Saas
eCommerce
OTT & Media
Healthcare
Company
About Us
Contact Us
Career
Made By
© All Rights Reserved
Privacy Policy
Terms & Conditions