From 1,000 rows to 100,000
Sellers onboard their catalogs by uploading a spreadsheet and the product images that go with it. When I took the flow over it had a hard ceiling of a thousand rows per file — and the ceiling was in the browser.
Where the limit actually was
The original upload ran entirely in the front end. React parsed the file and ran every validation rule in the UI before anything was sent. That design has a real advantage — the seller gets errors immediately, without a round trip — and it works fine at a few hundred rows.
It does not scale, because all of that happens on the browser's main thread. Parsing a large spreadsheet and validating every row blocks rendering, so the tab freezes and then the page dies. A thousand rows was not a business rule. It was the point past which the browser stopped coping, written down as a limit.
So the real problem was never "make validation faster". It was that one thread was doing parsing, validation and rendering, and the request path was carrying work that did not belong in it.
The redesign
I rebuilt the flow so every piece of work happens somewhere it can actually be afforded.
A seller selects a folder holding one spreadsheet and its images. A web worker parses and pre-validates it off the main thread, then streams rows to the API in batches of 100. The API returns signed upload URLs and publishes each batch to a Pub/Sub topic. The browser uploads image files straight to object storage using those URLs, so image bytes never pass through the API. Validation workers consume the topic across instances, keyed by a correlation id, writing failures to the database and finally producing an error file and a success file. Accepted products appear in Product Management.
Parsing moved off the main thread
The seller now selects a folder containing the spreadsheet and its images. Parsing happens in a web worker, so the UI stays responsive no matter how large the file is. The worker still runs validations — but only the cheap structural ones the backend needs in order to accept a row at all, not the full rule set.
That split is the important one. Client-side validation stopped being the authority and became a fast first filter.
I built both halves of this — the React upload client and the worker that parses inside it, as well as the services on the other side of the request. Being on both ends is what made the boundary movable: the fix here was deciding which side each job belonged on, which is hard to do when you only own one of them.
Rows stream to the backend in batches of 100
Parsed rows are streamed up in batches rather than posted as one document. Neither side ever holds the whole catalog: the browser sends a batch at a time, and the API handles a batch at a time. Peak memory is a function of batch size — a number we chose — instead of file size, which the seller chooses.
Image bytes never touch the API
For each batch the API returns signed upload URLs, streamed back for that same batch. The browser then uploads the image files directly to object storage. If the seller supplies image URLs instead of files, the backend verifies those instead.
This is the change that takes the most load off the platform. A catalog upload is mostly image bytes, and none of them are now proxied through application servers — those servers only ever handle metadata and hand out permission to write.
Validation happens after the response, not during it
In the same step, the API publishes the parsed batch to Pub/Sub. That is where the request path ends: the browser is told the batch was accepted long before the batch has been fully validated.
Consumers pick the work up from the topic and validate across multiple server instances in parallel, each row carried by a unique identifier that ties every result back to the upload it came from. Throughput becomes a matter of how many consumers are running, and a large upload no longer occupies a request thread while it is checked.
The results come back as artifacts
Because validation is asynchronous, there is no response to put failures in. Errors are written to the database as they are found, and when every batch of an upload has been processed the system generates an error file and a success file. The seller gets a precise account of what went in and what did not, and accepted products appear in Product Management.
Result
- Rows per upload
- 1,000 (hard cap)
- 100,000, load-validated
- Parsing
- Main thread, UI froze
- Web worker, UI stays live
- Image bytes
- Proxied through the API
- Direct to object storage
- Validation
- Inside the request
- Consumers, off the topic
- Scaling knob
- Make the handler faster
- Add more consumers
The ceiling moved by a factor of a hundred, and it did so under concurrent multi-user submission rather than in a quiet benchmark.
What I would call the lesson
Nothing here made any individual operation faster. Parsing a row costs what it always cost. What changed is where each kind of work happens: parsing to a worker, image bytes to storage, validation to consumers, and only coordination left on the request path.
I had met the same constraint before in the open — mongodb-export-large-reports streams millions of records out of MongoDB at constant memory. Different direction, same rule: never hold the whole set, and never let one thread own work that can be handed off.