Syncing inventory: where the time goes

Syncing inventory is cheap when you compare two lists with the right structure: 50,000 SKUs in 14.94 ms with a Map, against 11.87 seconds with linear search. The time you actually lose sits elsewhere: in the 3.88 MB payload, in dropped webhooks and in deciding which system is right.

The comparison is not the bottleneck

The gap between 14.94 ms and 11.87 s teaches one thing: never compare two lists with nested loops. With a Map built once, the comparison is linear and stays under 15 ms at 50,000 SKUs. If your import takes minutes, the problem is not the product count: it is the database queried once per row, or the file parsed on every pass.

Where the time goes: the payload

3,877,891 bytes for 50,000 SKUs is 77 bytes per product, and 248,356 bytes compressed is 5 bytes per product: gzip removes 94% because the fields repeat. Sync every five minutes and that is 288 transfers a day, 1.1 GB of JSON or 71 MB compressed. The same information as a list of differences, only changed SKUs and quantities, is 5,000 rows rather than 50,000: the first optimisation is not sending what did not change.

One single source of truth

The shop and the ERP cannot both be right. Pick the system that owns the physical quantity, usually the ERP, and treat the availability you display as a dated copy. If the shop sells and the ERP does not know, the order gets rejected: that is overselling, and it costs twice, because the customer has already paid. If the ERP deducts and the shop does not know, you show sold-out products and lose sales. The first is a legal risk, the second a commercial cost, and the choice belongs in the design.

The diff in one line

The function below takes the warehouse list and the shop list and returns only the differences, in one pass. It is the minimum version of what you need: in practice add handling for new and vanished rows, and the last-update timestamp per product, which is the most useful field when a customer disputes an order.

node -e "const a=[{sku:'A',q:1},{sku:'B',q:2}],b=[{sku:'A',q:1},{sku:'B',q:0}];const m=new Map(a.map(p=>[p.sku,p.q]));console.log(b.filter(p=>m.get(p.sku)!==p.q).map(p=>p.sku+' '+m.get(p.sku)+'->'+p.q).join(', '))"

Webhooks, polling and event order

What to show when the data is old

No sync is instant, so the page should say when it last ran: "stock updated at 14:32" is honest information and costs one line of code. When the quantity is uncertain, "last few units, shipping in 2-3 days" beats "in stock": declaring availability you do not have is a misleading practice under Articles 20-23 of the Consumer Code, and in distance sales the consumer can withdraw anyway, so a cancelled order costs you twice. If a product runs out while the order is in flight, the only correct move is to tell the customer at once and offer to wait or to refund.

The limitation

The numbers measure an in-memory comparison on lists already loaded: in production the clock is set by the ERP API latency and the database queries, which on 50,000 rows can cost seconds. The test does not cover concurrency: two processes writing the same quantity at the same moment need a lock or a conditional write, and no single-process benchmark shows that. And 5% of changes is a test value: in your shop the real percentage changes the payload cost, not the comparison cost.

Where to start