Magento order emails: the stuck queue_message_status row

The Loom shows up at 23:41 on a Tuesday. A Dutch agency we work with is on speaker with their client, a small Magento 2 store doing maybe 80 orders a day. The client noticed it because a customer called: where is my confirmation? Then she checked her own test order from that morning. Nothing in her inbox either. Then she pulled three days of orders from the admin and started cross-referencing the mail logs. Three days of orders. Zero emails sent.

The store had not changed. No deploy, no module update, no infrastructure work. Orders were going through Stripe fine, inventory was decrementing, sales_order rows were being written, and the customer-facing pages looked completely normal. The only symptom was the absence of an email. Marketing was sending campaigns out of the same SMTP relay without trouble, so the network path was demonstrably healthy. The transactional channel was the silent one.

This post is the walkthrough of how we traced the Magento order email outage to a single row in queue_message_status, and why fixing it took ninety seconds once we knew where to look. If you run Magento 2 in production, this is a failure mode you will see eventually, and there is no admin screen that warns you about it.

The first signal in the database

Order confirmations in Magento 2 do not get sent inline. They get queued. Since 2.3 the default is asynchronous email, which means when an order is placed, a message goes onto the queue and a consumer process picks it up later. That decoupling is good for checkout latency and bad for visibility. A stuck consumer fails silently. The order is fine. The queue is the broken piece. The admin still ticks through to "Complete" and the merchant has no reason to suspect anything.

First thing we did was open the database and ask the most obvious question.

SELECT topic_name, COUNT(*) AS pending
FROM queue_message qm
JOIN queue_message_status qms ON qms.message_id = qm.id
WHERE qms.status IN (2, 4, 7)
GROUP BY topic_name
ORDER BY pending DESC;

Status 7 is NEW, 2 is IN_PROGRESS, 4 is RETRY_REQUIRED. The top row came back with 2,341 pending messages for topic_name = sales.email.order. Three days of order confirmations sitting in a table, waiting for a consumer that never came. The runner-up topics (invoice, shipment, creditmemo) were stacked behind it for the same reason.

Walking the cron chain

In Magento 2 the consumer for that topic gets started by the cron job consumers_runner inside the consumers group. The chain looks like this:

• System cron fires every minute and calls bin/magento cron:run.
• That dispatches the consumers group.
• consumers_runner starts the configured consumers, one of which is sales.email.order.
• The consumer reads from queue_message_status, marks rows as IN_PROGRESS, sends the email, marks them COMPLETE.

We checked the system crontab. Fine. We checked cron_schedule.

SELECT job_code, status, scheduled_at, executed_at, finished_at
FROM cron_schedule
WHERE job_code = 'consumers_runner'
ORDER BY scheduled_at DESC
LIMIT 10;

Every row showed status = success. The cron was running. The consumer was being started. Yet nothing was being processed. That is the classic shape of this incident: every health check higher up the stack says green, because every health check higher up the stack is asking the wrong question.

Then we looked at the process list.

ps -ef | grep queue:consumers | grep -v grep

Three sales.email.order processes were running. According to ps they had been running for between four and seventy hours. None of them were doing anything. They were not crashed. They were not consuming CPU. They were sitting there, blocked on something invisible.

Inside queue_message_status

This is where it got interesting. We asked the table to show its oldest in-progress rows.

SELECT id, message_id, status, updated_at, number_of_trials
FROM queue_message_status
WHERE status = 2
ORDER BY updated_at ASC
LIMIT 5;

The oldest row was three days, four hours old. Status 2 (IN_PROGRESS), number_of_trials = 0. It had been picked up by a consumer that had then disappeared without ever marking it COMPLETE or ERROR. The next consumer instance found it already in IN_PROGRESS, respected that, and walked past it. So did the one after. And the one after that. The queue runs FIFO inside a topic, so every later confirmation piled up behind one ghost row.

Magento's database queue driver uses SELECT ... FOR UPDATE to claim a message and write IN_PROGRESS. If the worker dies between claim and complete, the row sits there until something resets it. There is no built-in reaper. The framework assumes consumers either finish their work or crash cleanly, neither of which is guaranteed when the underlying problem is, say, an OOM kill from the host or a SIGTERM from a deploy script.

In this client's case the host had OOM-killed PHP-FPM children three nights earlier during a backup window that briefly pushed the box past its memory ceiling. The cron-spawned consumer happened to be among the casualties. Its claimed row in queue_message_status was never released. From that moment on, every confirmation email queued up behind it. The Adobe Commerce docs on managing message queues describe the consumer lifecycle, but they do not really cover this failure mode. You learn it by running into it.

Unsticking the row without sending three days of duplicates

The fix is two SQL statements and a process kill, in order.

pkill -f 'queue:consumers:start sales.email.order'

UPDATE queue_message_status
SET status = 7
WHERE status = 2
  AND updated_at < NOW() - INTERVAL 30 MINUTE;

Status 7 puts the row back in the NEW pool for any consumer to claim. The 30-minute floor protects you from clobbering a row that a healthy consumer is mid-way through, which matters if the cron has restarted a consumer in the gap between your pkill and your UPDATE.

The duplicate-email worry was real on the other side too. Three days of unsent confirmations means 2,341 emails about to land at once, some on inboxes that have since rage-quit and moved on. The agency's call here was correct: send them. A confirmation arriving late is annoying. A confirmation that never arrives looks like fraud. We did flag two customers who had since opened a chargeback, deleted those specific queue_message rows, then let the rest go.

DELETE qms FROM queue_message_status qms
JOIN queue_message qm ON qm.id = qms.message_id
WHERE qm.topic_name = 'sales.email.order'
  AND qm.body LIKE '%"order_id":12847%';

Then we restarted the consumer cleanly with a max-messages ceiling, so the catchup wave could not itself wedge another row.

bin/magento queue:consumers:start sales.email.order --max-messages=500

Six minutes later, the table was empty and the mail logs were busy. The agency sent a short note to the customer explaining the delay. The store owner sent six refunds for shipping. Cheaper than the alternative.

The blast radius of one stuck row

The interesting part is not that this happened. The interesting part is that nothing alerted on it. Magento's own admin shows the order as complete. The store's monitoring was watching HTTP 200s and database CPU. Sentry was watching PHP exceptions. None of those layers can see "an email that should exist does not exist." Order confirmations are an absence, and absences do not throw.

The defence is one cron job, running every fifteen minutes, that asks the question we asked at 23:41 and shouts when the answer is wrong:

SELECT COUNT(*) AS stuck
FROM queue_message_status
WHERE status = 2
  AND updated_at < NOW() - INTERVAL 30 MINUTE;

If that count is non-zero for two consecutive checks, something is wedged. Pipe it to Slack, Pushover, whatever you read. A second query worth running is the backlog of NEW messages older than five minutes per topic, since that catches the inverse case: a consumer that never starts at all, usually because cron_consumers_runner is missing from app/etc/env.php after a config rebuild.

Hardening the consumer for next time

Three things worth doing before you close the ticket.

First, set max_messages and the consumer list explicitly in app/etc/env.php. A consumer that exits cleanly every N messages is a consumer that releases its lock on the next bad row instead of holding it forever. The consumers reference covers the keys; the config that worked here was:

'queue' => [
    'consumers_wait_for_messages' => 0,
    'only_spawn_when_message_available' => 1,
],
'cron_consumers_runner' => [
    'cron_run' => true,
    'max_messages' => 200,
    'consumers' => [
        'sales.email.order',
        'sales.email.order.invoice',
        'sales.email.order.creditmemo',
        'sales.email.order.shipment',
    ],
],

Second, add the stuck-row query above to whatever cron monitor you already use. If you have none, a five-line PHP script in /var/scripts that posts to a webhook is enough. The query is cheap, the table is small, the alert is unambiguous.

Third, and this is the one most teams skip, write down what the recovery looks like. The agency we worked with had no runbook for "stuck queue". When it happened, they spent forty minutes guessing at MTA logs before opening the database. A four-line README in the repo is the difference between a six-minute fix and a four-hour incident.

When we built Pier we kept running into exactly this shape of problem on every legacy site we touched. The failure is in a database row, the diagnosis needs to happen at 23:41, and the engineer who actually knows the codebase is asleep. The chat-first MySQL editor we ship is so "show me stuck rows in queue_message_status older than thirty minutes" is a sentence you can type instead of a query you have to write, and the version history means the UPDATE that unsticks them is one click to undo if you misjudge the floor.

The smallest thing you can do today: open a database console on every Magento 2 store you operate, run that one COUNT query against queue_message_status, and write the result down somewhere. If it is anything other than zero, you have found your next incident before it found you.