What separates an import that holds from one that zeroes every price overnight: which system owns each field, how you publish without anyone seeing a half-built catalogue, and the four checks that stop a disaster before the customers do.
by Nadia Maar ·
The nightly import runs from two to twenty past. It went fine for eight months. Then one night the supplier exports the price list with a different decimal separator, and at four minutes past two thirty thousand products are worth a few cents each.
The supplier's mistake is not the problem: it will happen again, and it is not yours to prevent. The problem is that between the bad file and the public catalogue there was nothing — no check asking whether a price list in which 96% of prices collapse is a plausible price list, and nowhere to hold it while somebody looks.
This piece is how that «nothing» gets built. Four things: which system owns each field, what happens to every incoming row, how you publish without taking the site down, and the thresholds that stop a bad night before customers see it.
The import that gets built in half a day does three things: read a file, delete what was there, write everything back. It works beautifully as long as the file is right, the connection holds, and nobody edits anything by hand on the site. Those are three conditions that hold for months, and then stop holding all at once.
«What happens if this file is wrong?» A well-built import has that answer in writing, and the answer is never «we find out from the complaints». The rest of this article is that answer, taken apart into its four pieces.
The first document is not technical: it is a table saying, for each field, which system is right when the two disagree. Without it, conflicts are settled by whoever wrote last — which is to say, by chance.
| Field | Owner | What happens if it changes on the other side |
|---|---|---|
| List price | Back office | An edit made on the site is overwritten the same night. That is deliberate: prices have one source. |
| Stock | Back office | Overwritten on every run, several times a day. It is the field that goes stale fastest of all. |
| Name and description | Site | The back office proposes the text only when the product is first created. After that, what you wrote stays. |
| Images | Site | The back office never touches them. Good photos cost more than the catalogue row they illustrate. |
| Category and filters | Site | Commercial classification is not accounting classification, and forcing them to match loses both. |
| Code and unit of measure | Back office | They are the row's identity: if they change from the site, the link between the two systems breaks silently. |
Six rows, half an hour in a meeting, and they remove the longest argument of the whole project. The rule of thumb: the back office owns anything that is a number, the site owns anything that is a commercial decision.
Every row in the file goes through a series of checks, and at each check some rows drop out. The rows that drop out are not waste: they are the value of the import. An import that writes everything it receives has no checks, only trust. Tap a step for the rule.
30,412 rows in tonight's file
30,108 continue; 304 quarantined.
Rows with no code, with a duplicate code, or with a code matching nothing in the product master. You do not invent products: if the code really is new, creating it has to be somebody's decision, not a file's side effect. The three hundred and four rows land in a list someone looks at in the morning.
30,086 continue; 22 quarantined.
A price that is not a number, a negative quantity, an unknown unit of measure, an impossible date. It is the dumbest check and the one that saves the most nights: most import disasters are not logical, they are a format that changed without notice.
1,240 continue; 28,846 identical, skipped.
Comparing against the current state is the difference between a twenty-minute import and a forty-second one. Twenty-eight thousand identical rows should not be rewritten: writing them anyway burns time, fills the change history with noise, and makes it impossible to answer «what changed last night».
1,202 continue; 38 held for confirmation.
This is the check that is almost always missing. A price moving 3% goes through; one dropping 90% stops and waits for a human eye. It applies in aggregate too: if more than 20% of prices move the same way in a single run, that is not an updated price list, it is a broken file — and the whole import stops.
Eleven hundred and two rows out of thirty thousand, in under a minute. And above all: written into a catalogue no customer is looking at, which is what the next section is about.
The classic import writes into the catalogue customers are looking at. The right one always writes into the idle side, and when that side is complete and has passed the checks, the pointer is swapped. The operation takes about as long as one database row. Press the button.
v. 2026-08-29 · 30,108 products
v. 2026-08-30 · 30,108 products
The real advantage is not the speed: it is that going back is the exact same operation, run the other way. No restore, no file to put back, no decision to take at three in the morning with somebody on the phone.
It is worth saying what it costs: two copies of the catalogue take twice the space, and space is cheap. The extra complexity is entirely in keeping orders and live baskets working during the swap, and it is solved once, at the start.
Thresholds are not there to prevent mistakes: they are there so that a mistake stops before it becomes public. You write them once, and they are numbers, not impressions.
A notification to a person, containing the number that tripped the check and the comparison with the average — not a generic message saying «import failed». The difference is that the first is resolved in two minutes from a phone, the second requires opening a laptop and going to look for what happened. And in the meantime the public catalogue is yesterday's, intact: that is the point of the whole architecture.
In the audit we look at three concrete things: what happens today if the file arrives wrong, how long the window is in which the public catalogue is half-built, and which fields overwrite each other between the back office and the site. From there it is clear whether your flow is at risk or already sound — that happens, and in that case I say so.
If you have already had a bad night, the same half hour reconstructs it: the cause is usually in that run's file, and the useful part is not working out whose fault it was but where the check should have been.
You keep two copies of the catalogue: one answers customers, the other receives the changes. When the second is complete and has passed its checks, you swap the pointer — an operation taking less than a tenth of a second. Anyone browsing notices nothing, and nobody ever sees a half-built catalogue. The biggest advantage is not the speed: it is that rolling back is the same operation run the other way, with no restore.
It depends on what an order for a product you do not have costs you. With slow-moving stock a nightly run is enough; where the same item sells to several customers in a day you need more frequent runs on stock alone, which is a small, fast file. The rule of thumb: sync often what changes often — stock and prices — and everything else once a day.
Almost never through a calculation error: usually the file arrived truncated, with a different decimal separator or one column short, and the import wrote it without asking whether it was plausible. The check that prevents this is aggregate, not per row: if more than a fifth of the catalogue moves the same way in one run, or an unusual share of products drops to zero, the import stops and alerts a person instead of publishing.
Neither, for everything. You need a table assigning each field to exactly one system: the back office owns anything that is a number — list price, stock, code, unit of measure — and the site owns anything that is a commercial decision — name, description, images, categories and filters. Without that table conflicts are settled by whoever wrote last, and every night the manual work gets overwritten.