PHP session race: the case of the wrong invoice PDF

The Loom came in at 23:41. A Dutch agency we work with had spent the evening trying to explain to their customer why their billing portal had emailed an invoice PDF addressed to a different company entirely. Same template, wrong logo, wrong VAT number, wrong line items. The customer had screenshotted the email and asked, politely, whether they should be worried about their data.

They were running a custom-PHP billing portal that had been in production since 2014. PHP 7.4, plain MySQL, no framework, three include files at the top of every script. The kind of legacy site that runs perfectly for years and then, one Tuesday evening, does something that makes you re-read the OWASP top ten before bed.

We spent the next six hours blaming session_start(). The bug was somewhere else.

The three suspect includes

Every page in the portal opened with the same boilerplate:

<?php
require __DIR__ . '/inc/config.php';
require __DIR__ . '/inc/session.php';
require __DIR__ . '/inc/auth.php';

config.php set up the PDO connection and a few constants. session.php called session_name('PORTAL') and then session_start() with session.use_strict_mode on. auth.php read $_SESSION['uid'], looked up the user, and bailed to the login page if anything looked wrong. Nothing exotic. The same shape of bootstrap we have read in roughly two hundred legacy PHP projects.

The download endpoint, /invoice.php?id=123, then did this:

$id  = (int) $_GET['id'];
$inv = $db->invoice($id, $_SESSION['uid']);
if (!$inv) { http_response_code(404); exit; }

header('Content-Type: application/pdf');
header('Content-Disposition: attachment; filename="' . $inv['number'] . '.pdf"');
echo render_pdf($inv);

That query was scoped to the session user. We re-ran it by hand against the database with the affected invoice ID and the affected user ID and it correctly returned NULL. So the PHP code refused to serve that invoice to that user. And yet the user had received it. Twice.

Six hours inside session_start()

The first hypothesis was the obvious one: a session_start() race. Someone had wired up a "resend invoice" cron job earlier in the year and we had a vague memory of CLI code touching $_SESSION in ways no one was proud of. We checked. The cron job did not touch sessions. It built its own auth context from a service-account token.

The next hypothesis was session locking. PHP serialises requests on the same session file by default. Two AJAX calls fired from the same browser will queue, one behind the other, because session_start() takes an exclusive lock on the session file and holds it until session_write_close() or end-of-script. The php.net note on session_write_close spells out the lock semantics in three lines if you have not read it in a while. We checked the access logs for overlapping requests on the same session. Nothing within five seconds of the wrong PDF.

We then turned over the custom session handler. The portal stored sessions in MySQL instead of on disk, which made us briefly hopeful that a bug in the read or write callback could explain the swap. We dumped the sessions table at the moment of an affected download and inspected the row by hand. The session_data blob deserialised cleanly. Every field matched the audit log of recent customer activity. The handler was not lying.

Then we got serious. We added a logging line to session.php directly after session_start():

error_log(sprintf(
    '[sess] pid=%d sid=%s uid=%s uri=%s',
    getmypid(),
    session_id(),
    $_SESSION['uid'] ?? 'none',
    $_SERVER['REQUEST_URI']
));

The logs were boring. Every request had a session ID. Every session ID mapped to one and only one user. We tailed the log on the production box and asked the affected customer to redownload an invoice. The line we saw was correct. Their session ID. Their user ID. The right invoice number in the URL. And then the PDF that came back belonged to someone else.

By hour four we reached the stage of debugging where everyone in the room agrees to try the thing they had been avoiding. We ran strace -f on the PHP-FPM worker pool during a reproduction attempt. The worker that handled the affected request never opened the invoice PDF file. It never touched the database. As far as the syscall log was concerned, that PHP process did no work at all for the request that returned a PDF.

That was the moment the floor moved.

Where the swap actually happened

The portal sat behind nginx. The agency's previous ops contractor had turned on fastcgi_cache sometime in 2022 to take load off PHP-FPM during end-of-month invoice runs. The config looked, to a tired eye, like this:

fastcgi_cache_path /var/cache/nginx/portal levels=1:2
    keys_zone=PORTAL:50m max_size=2g inactive=30m;

location ~ \.php$ {
    fastcgi_cache       PORTAL;
    fastcgi_cache_key   "$scheme$request_method$host$request_uri";
    fastcgi_cache_valid 200 10m;
    fastcgi_pass        unix:/run/php/php7.4-fpm.sock;
    include             fastcgi_params;
}

Read the fastcgi_cache_key line slowly. It is composed of scheme, method, host and URI. It does not include the PORTAL session cookie. It does not include the Authorization header. It does not include anything at all that identifies the requesting user.

The download endpoint was /invoice.php?id=123. The first user to hit that URL after the cache was cold had their PDF rendered, returned, and stored in /var/cache/nginx/portal under a key derived purely from the URI. Every other authenticated user who downloaded an invoice with the same numeric ID for the next ten minutes got that first user's PDF, byte for byte, with no PHP execution at all. The session log line we added never fired for those requests. nginx served the cached response and the request never reached PHP-FPM.

The reason it took us six hours: invoice IDs are usually sequential and customer-scoped, so collisions on the same ID across customers are rare. The reason it broke this week: the agency had migrated three customers from a legacy table into the portal earlier that day, and the imported invoices kept their old IDs. Two of the imported IDs collided with existing IDs from a different customer. That was the trigger. Everything else was nginx doing exactly what we had told it to do.

We confirmed it with one extra line in the nginx config. add_header X-Cache-Status $upstream_cache_status always;, reload, ask the customer to download an invoice again, watch the response. The header came back as HIT. That was the receipt we needed. The PHP code had not malfunctioned. It had simply not been asked. nginx had answered the question itself, from a stale entry, using a key that did not know who was asking.

The fix, and the things we found while fixing it

The minimum patch was three lines. Tell nginx not to cache anything from authenticated users, and tell PHP to send the headers that would make any future cache layer respect that.

location ~ \.php$ {
    fastcgi_cache       PORTAL;
    fastcgi_cache_key   "$scheme$request_method$host$request_uri$http_cookie";
    fastcgi_cache_bypass $cookie_PORTAL;
    fastcgi_no_cache    $cookie_PORTAL;
    fastcgi_pass        unix:/run/php/php7.4-fpm.sock;
    include             fastcgi_params;
}

And in invoice.php, the headers we should have sent from day one:

header('Cache-Control: private, no-store, max-age=0');
header('Pragma: no-cache');

The MDN reference on Cache-Control is worth re-reading every time you wire up a download endpoint. private alone is not enough if your reverse proxy is configured to ignore Cache-Control entirely, which fastcgi_cache is by default unless you opt in with fastcgi_ignore_headers. Both layers have to agree. The nginx fastcgi_cache documentation says this in roughly seven scattered paragraphs.

While we were in there, we also added a hard ownership check on the invoice query, not just a session-scoped fetch:

$inv = $db->invoice($id);
if (!$inv || $inv['user_id'] !== $_SESSION['uid']) {
    http_response_code(404);
    exit;
}

The original query had been "correct" in that it filtered by user in the WHERE clause, but defensive ownership checks at the application layer make the next bug like this one self-disclosing in code review. A query that fetches by ID and then asserts ownership is easier to audit than a query that smuggles ownership into a JOIN.

Verification was straightforward and slow. We hit /invoice.php from two browsers logged in as two different users, ten seconds apart, with the same invoice ID in the URL. Both responses came back with X-Cache-Status: BYPASS. We logged out, hit a non-authenticated marketing page on the same vhost, and watched the cache work normally on a public asset. We left X-Cache-Status in the response headers permanently. It is a one-line addition and it answers a question we had spent five hours not being able to answer.

What we left running afterwards

Once the immediate fire was out, we added three small pieces of monitoring that the portal had been missing for six years.

The first was a synthetic check. A dedicated test user logs in every fifteen minutes, downloads a known invoice, and the script verifies the SHA-256 of the response matches a recorded baseline. If a second test user under a different agency tenant ever receives the same SHA for a different invoice ID, the alert fires before a real customer notices.

The second was a daily grep over the nginx access log for any HIT on a path that begins with /invoice or /account. There should never be a cache hit on those paths. If one shows up, something in the cache config has regressed and we want to know on Monday morning, not on a Friday evening when the customer is screenshotting.

The third was a yearly diary entry on the agency's shared calendar to re-read the nginx config from top to bottom. Caching rules drift. Comments rot. The next contractor will not remember why fastcgi_no_cache is set on the session cookie, and they will be tempted to tidy it up.

The audit no one wanted to write

The hardest part of the incident was not the fix. The fix was three lines of nginx and two lines of PHP. The hard part was answering the question that came at 09:00 the next morning: which invoices, exactly, went to which customers? The agency had to tell three of its own customers whether their VAT numbers, bank details, and line items had been seen by another company.

We had nginx access logs going back ninety days. We had the cache directory on disk. We did not have a record of which response body had been served from cache versus rendered live, and we did not have a record of which file in the cache directory had been keyed to which URI at the time of each hit. nginx does not write that down. You can reconstruct most of it from log timestamps and the cache file's stored KEY: header, but "most of it" is not the word a compliance officer wants to hear.

We ended up writing the disclosure to the three affected customers on Wednesday afternoon. Two replied within an hour, asked one or two clarifying questions about which fields had been visible, accepted the explanation. The third wanted a written incident report on letterhead, which is the version of this conversation that takes a week to close. None of them left. The agency owner told us afterwards that the apology email took longer to draft than the patch took to write, and that felt about right.

When we built Pier we ran into this exact shape of problem from the other side. Pier writes version history for every file touched through the app and every row written through the MySQL editor, so when the question is "what did this server actually serve, and who edited what before it did", the answer is in the log instead of in your memory.

The single smallest thing worth doing today: open the nginx config for the oldest PHP site you maintain, grep it for fastcgi_cache_key, and check whether the key includes anything that identifies the requesting user. If it does not, and the site authenticates anyone, you have the same bug we did. You just have not been billed for it yet.