Magento 2 cache warmer: a bash playbook that survives reboots

It is 21:14 on a Friday. The deploy script just ran bin/magento cache:flush, and there is no cache warmer running behind it. The next visitor to land on a product page hits a cold full page cache and waits 4.2 seconds for TTFB. The second visitor waits 3.9. By the time the cache is warm enough to be useful, three people in the Slack channel have asked whether the site is down.

A Dutch agency we work with had this exact loop running every Friday afternoon for almost a year. The fix that finally stuck was not a Magento 2 cache warmer extension. It was a two-hour bash script, a curated text file of URLs, and a sleep that respects the origin. The whole thing fits on one terminal screen and recovers cleanly from a reboot.

This post is the whole script. Copy it, tune the throttle, and put it behind a systemd unit so it does not die the next time someone in the data centre patches the kernel.

The URL list nobody curates

A cache warmer is only as good as the URLs you feed it. Most Magento 2 stores ship a sitemap.xml that excludes the URLs that actually matter: filtered category pages, sorted listings, popular product variants with stock attributes in the URL, the cart and checkout views you actually want pre-rendered. Warming the sitemap alone leaves you with cold caches on the exact pages the homepage links to.

Build the URL list in two passes. First, pull the sitemap as a baseline:

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+</loc>' \
  | sed -E 's|</?loc>||g' > /var/lib/magento-warmer/urls.txt

Second, add the top 200 product URLs by recent sales. Run this against the Magento database directly. It is faster than crawling and it gives you the URLs that move revenue:

SELECT CONCAT('https://example.com/', ur.request_path) AS url
FROM url_rewrite ur
JOIN sales_order_item soi ON soi.product_id = ur.entity_id
WHERE ur.entity_type = 'product'
  AND ur.store_id = 1
  AND ur.redirect_type = 0
  AND soi.created_at > NOW() - INTERVAL 30 DAY
GROUP BY ur.request_path
ORDER BY SUM(soi.qty_ordered) DESC
LIMIT 200;

Append the result to the same urls.txt, then dedupe with sort -u. For a 5000-SKU catalogue the file lands around 1.2 MB and parses in under a second. Re-run the SQL once a week from cron so the warmer keeps tracking what your customers actually buy, not what your category tree happened to look like in 2024.

Filtered category pages

If you run faceted navigation (price slider, brand checkbox, colour swatch), the cache key changes per filter combination. You cannot warm all of them, and you should not try. Pick the three filters per category that your analytics show actually get clicked, build the URLs by hand, and add them to the list. Twelve filter URLs per category is plenty. More than that and you are burning warmer time on pages no customer reads.

A bash loop that respects the origin

Here is the warmer. It is around thirty lines of POSIX-friendly bash, uses xargs for concurrency, and writes a tab-separated log you can grep:

#!/usr/bin/env bash
set -euo pipefail

URL_FILE="${URL_FILE:-/var/lib/magento-warmer/urls.txt}"
LOG_FILE="${LOG_FILE:-/var/log/magento-warmer.log}"
CONCURRENCY="${CONCURRENCY:-4}"
SLEEP_MS="${SLEEP_MS:-250}"
UA="MagentoWarmer/1.0 (+ops@example.com)"

warm_one() {
  local url="$1"
  local http
  http=$(curl -s -o /dev/null \
    -w '%{http_code} %{time_starttransfer} %{size_download}' \
    -A "$UA" -H 'X-Cache-Warm: 1' \
    --max-time 30 "$url" || echo '000 0 0')
  printf '%s\t%s\t%s\n' "$(date -Iseconds)" "$url" "$http" \
    >> "$LOG_FILE"
}

export -f warm_one
export LOG_FILE UA

SLEEP_SEC=$(awk "BEGIN{print ${SLEEP_MS}/1000}")
xargs -a "$URL_FILE" -n 1 -P "$CONCURRENCY" -I{} \
  bash -c "warm_one '{}'; sleep $SLEEP_SEC"

Two values matter: CONCURRENCY and SLEEP_MS. The defaults (four parallel requests, 250 ms gap) work for a single-node Magento on a modest VPS. If you sit behind Varnish on a four-core origin, you can usually push to eight and 100 ms. Watch the origin during the first run with htop: if PHP-FPM workers stay above 70% sustained, dial it back. The warmer should be invisible to real traffic, not a self-inflicted denial of service. The xargs manpage covers the exact semantics of -P and -I if you want to swap to GNU parallel or change the substitution style.

The X-Cache-Warm: 1 header is for your own logs and for Varnish if you want to track warm versus cold hits separately. It must not change the cache hash. In your VCL, do not add it to vcl_hash and do not strip it before vcl_backend_fetch or the warmer will pollute the cache with a separate variant per URL. Use it for telemetry, not for routing.

Surviving a reboot

A bash script in cron dies silently when the mail spool fills. A bash script in a tmux session dies on reboot. The point of this playbook is that the next time someone reboots the host (kernel patch, security advisory, cloud provider maintenance window), the warmer comes back without you logging in.

Drop the script at /usr/local/bin/magento-warmer.sh, mark it executable, and write a one-shot unit:

# /etc/systemd/system/magento-warmer.service
[Unit]
Description=Magento 2 full page cache warmer
After=network-online.target varnish.service
Wants=network-online.target

[Service]
Type=oneshot
User=www-data
Group=www-data
Environment=URL_FILE=/var/lib/magento-warmer/urls.txt
Environment=CONCURRENCY=4
Environment=SLEEP_MS=250
ExecStart=/usr/local/bin/magento-warmer.sh
Nice=10
IOSchedulingClass=idle
TimeoutStartSec=2h

[Install]
WantedBy=multi-user.target

The interesting flags are Nice=10 and IOSchedulingClass=idle. They tell the kernel that warmer traffic should yield to anything a real customer is doing. TimeoutStartSec=2h matches the two-hour ceiling in the headline: if the warmer is still running after two hours, something is wrong and systemd should kill it rather than let it overlap with the next run.

Then pair it with a timer:

# /etc/systemd/system/magento-warmer.timer
[Unit]
Description=Run Magento warmer at boot and every six hours

[Timer]
OnBootSec=90s
OnUnitActiveSec=6h
Persistent=true

[Install]
WantedBy=timers.target

Enable both with:

systemctl daemon-reload
systemctl enable --now magento-warmer.timer
systemctl list-timers magento-warmer.timer

Persistent=true is the line that matters. If the host was off when a scheduled run was due, systemd fires the warmer once on the next boot to catch up. OnBootSec=90s gives Varnish and PHP-FPM time to settle before the warmer hits them. The full systemd.timer documentation covers the edge cases around drift, accuracy windows and calendar events if you want a cron-style schedule instead.

Verifying the cache actually warmed

A warmer that runs without errors is not the same as a warmer that warms. Magento 2 exposes a debug header on the full page cache you can read with curl. Add this to your post-warm check:

curl -sI -H 'X-Cache-Warm: 1' \
  https://example.com/some-product.html \
  | grep -iE 'x-magento-cache-debug|age|x-cache'

You want X-Magento-Cache-Debug: HIT on the second request, and on Varnish you want Age to be greater than zero. If the first request is HIT and the second is MISS, your cache is being invalidated mid-warm. Likely culprits: a stock update cron flushing tagged entries, or an admin user editing products live during the run. Schedule the warmer for a window when the admin is asleep and the indexers are not running, and the HIT rate climbs.

Sample log line from a healthy run:

2026-06-09T03:14:22+02:00	https://example.com/red-leather-chair.html	200 0.184 41327

That is HTTP 200, 184 ms to first byte, 41 KB transferred. Anything over 1.5 seconds on a warmed page means the warmer is not reaching the cache, and you should investigate before adding more URLs to the list. A simple awk one-liner over the log tells you the p95 in about a second:

awk -F'\t' '{split($3,a," "); print a[2]}' /var/log/magento-warmer.log \
  | sort -n | awk 'BEGIN{c=0}{v[c++]=$1}END{print v[int(c*0.95)]}'

When the warmer is not the bottleneck

If you are warming 5000 URLs and the warm pages still serve at 1.8 seconds TTFB, the warmer is not your problem. The cache is filling correctly. The bottleneck is in the rendering layer: an unindexed query in a layout block, an Elasticsearch suggester running synchronously, or a third-party module hooking collection_load_after and calling an external API per product. No amount of pre-warming will fix that, because every customer who lands on a fresh URL still pays the cold cost.

This is the part where the two-hour script stops being enough. You need to go into the codebase, read the DI graph, and find the offender. That is the work we do when an agency hands us a slow legacy site and asks us to fix the rendering rather than throw more cache at it. When we built Pier we ran into this exact thing on a Magento store: the slow query was obvious in the log but the layout XML responsible for it was three modules deep. The way we ended up handling it was wiring the MySQL editor next to the file tree so you can read core_config_data, the layout XML and the slow-query log in one window, with one version history the moment you save.

The smallest thing to do today: ls -la /var/log/magento-warmer.log. If the file does not exist, you do not have a warmer. Write the thirty lines above, drop the script in /usr/local/bin, and run it once by hand against the sitemap before you wire up the systemd unit. The rest is paperwork.