One of the Shopify apps we build collects storefront events through a web pixel: a cart is added, a checkout starts, a checkout completes. The first time we installed it on a store, it collected nothing. The dashboard stayed empty, and almost everything we had built to tell us whether it worked said that it did.
It took one day and four separate faults to get the first event through. None of them was exotic. What they had in common is that each time, the check we trusted stood on the same side as the bug. Here they are, with the identifiers removed and the dates moved.
1. "No pixel yet" came back as an error, not as an empty answer
A pixel extension only declares the pixel. Nothing runs on a storefront until the app creates a pixel for that shop through the Admin API. So our code first asked whether the shop already had one, and if not, created it.
For a shop that had never had a pixel, we expected an empty answer. In the API version we were on, the answer was the pixel field set to null, with an error next to it: no web pixel was found for this app. The GraphQL client we use turns a response that carries errors into an exception. Our catch block read the exception as a failed activation, logged it, and stopped. The create step never ran.
The one state the code existed to handle, "there is no pixel yet", was the one state it could never get past. On a new install, that is every shop.
It hid well, too. An earlier probe of the same query printed data ?? errors, so it showed the null and dropped the error sitting right beside it. When you inspect a response, print all of it.
2. The preflight went to a handler we had not written for it
Once the pixel existed, it fired correctly on add-to-cart. The event still never arrived.
The pixel sends a JSON POST to our server, which lives on a different origin from the storefront. For that kind of request the browser first sends an OPTIONS request, the CORS preflight, and only sends the POST if the preflight is answered with a 2xx status.
We had written the OPTIONS handling in the route's action. In our React Router app the OPTIONS request went to the route's loader instead, and the loader answered "POST only" with a 405. That 405 even carried the correct CORS headers, which is why it was easy to miss: everything about the response looked right except its status. The browser rejects a non-2xx preflight whatever headers it carries, so the POST was never sent. The server log showed two OPTIONS requests, two 405s, and no POST after either.
The fix was a few lines in the loader. The lesson came with the test for it: when we reverted the fix, the assertion on the headers still passed, because the broken response had the same headers. Only the assertion on the status failed. The status was the part that carried the weight.
3. Every curl check said the endpoint was alive
Before we found the preflight, we had checked the endpoint by hand, more than once, and it answered every time.
That is because curl is not a browser. It sends no preflight. It sends the POST directly, the server accepts it, and the channel looks healthy. The thing that was actually broken, the browser asking for permission and being refused, simply does not exist in that test.
What can see it is a browser on a different origin making the same request the pixel makes. We now have exactly that as an automated test, plus a small script, part of our deploy check, that asks the deployed server the preflight question directly. Both were checked against a server that reproduces the 405-with-correct-headers fault, and both fail on it.
4. The unit tests mocked the answer we expected
The unit tests were green through all of this. For the "no pixel yet" case they mocked a response with the pixel set to null and nothing else, the clean shape we had assumed. The real API never sent that shape for this case. The tests were not testing the API. They were testing our assumption about it, and our assumption was the bug.
The fixed tests use the shape the API actually sent, in both forms: the exception our client raises, and the raw response with the error beside the null. A control run confirms that they hold: bring the old behaviour back and exactly those tests fail.
The rule behind all four
Each check answered a slightly different question from the one we needed answered. "Did the query return?" instead of "did it return what a new shop returns?" "Does the endpoint answer?" instead of "does it answer a browser?" "Do the tests pass?" instead of "would they fail if the API behaved the way it really does?"
The question we ask of a check now is simple: what would this check show if the thing were broken? If the answer is "the same as now", it is not a check yet.
Two more things came out of that day that were not bugs at all, and that a merchant will meet on day one. Browsing product pages produces no event: in our app the first event is adding to the cart. And the figures are summarised once a day, so a cart added a minute ago is collected but not yet counted. An app that is measuring nothing looks exactly like a shop with no traffic, so the screen now says both of these things out loud.