Data
Shopify webhooks to polling, Christmas week
Shopify warned our merchants that our webhooks would stop working, and its support gave conflicting answers. Another engineer and I moved 130+ stores to polling without waiting for an answer. No data lost.
130+ Stores moved to polling
- Company
- ReturnLogic
- Role
- Built with one other engineer: half of engineering
- Period
- 2022
- Stack
- AWS Lambda · SQS · Shopify REST API · TypeScript · Node.js · PostgreSQL
“Add to cart” takes you to the store with this project in your cart.
What happened
ReturnLogic ran as a private Shopify app and took in each merchant’s data through webhooks. We had registered them without a version, which Shopify’s docs said meant “latest.”
In Christmas week 2022, right as peak returns season began, Shopify flagged that version as deprecated.
What we built
Lambda functions on a 30-minute schedule pulled products, orders, customers, collections, and inventory from Shopify’s REST API and fed them through SQS into our existing ETL, so nothing downstream changed.
Each poll asked for records updated since the last one for that store and resource, with a small overlap, throttled and backing off on Shopify’s 429 responses. Records were upserted by Shopify ID, so the overlap, and data arriving through both webhooks and polling, merged instead of duplicating. Thirty minutes was fresh enough: returns start days after an order.
Another engineer and I split it by layer: I built the Lambda pollers, and the other engineer handled the path through SQS into the ETL. We switched every store over at once.
Outcome
No outage, and record counts matched Shopify’s through the peak returns week. The deprecation turned out to be a non-issue.
- <1wkFrom Shopify’s email to all stores polling