neerajkumar.me

Reading a file you can't fit in memory

Almost every file-upload feature starts the same way:

const rows = parse(await readFile(path));
const valid = rows.filter(isValid);
await db.insertMany(valid);

Three lines, obviously correct, passes review. It also has a failure mode that will not show up in any test you are likely to write: peak memory scales with the size of the input.

Why the test suite misses it

You test with a small fixture. Of course you do — a 200-row CSV makes the test fast and the assertions readable. At 200 rows the code above uses a few hundred kilobytes and returns instantly.

Production hands it a 100,000-row file. Now you are holding the raw buffer, the parsed array, and the filtered array simultaneously — three copies of the same data, at a size you did not choose. Then a second user uploads at the same time.

The thing that makes this bug nasty is that it is not deterministic. The service survives one large file. It falls over when two arrive together, which means it falls over in production, under load, and not on your machine.

Treating the file as a stream

The fix is a change of mental model rather than a clever trick. The file is not a value your program holds. It is a sequence your program passes through.

const stream = createReadStream(path).pipe(parse({ columns: true }));

for await (const batch of chunk(stream, BATCH_SIZE)) {
  const valid = batch.filter(isValid);
  if (valid.length) await db.insertMany(valid);
}

The shape barely changed. The memory profile changed completely: peak usage is now a function of BATCH_SIZE, a number you picked, rather than of the file size, a number a stranger picked.

The part people skip

Here is where it usually goes wrong. This looks equivalent:

const writes = [];
for await (const batch of chunk(stream, BATCH_SIZE)) {
  writes.push(db.insertMany(batch.filter(isValid)));  // no await
}
await Promise.all(writes);

It is faster on a small file and it reintroduces the exact bug you just fixed. Without awaiting, the reader races ahead of the database and every un-settled promise holds its batch alive. You have not removed the unbounded memory, you have moved it from a file buffer into a queue of pending writes — where it is harder to see.

Awaiting the write inside the loop is what applies backpressure. The writes pace the read. It feels like you are giving up concurrency, and you are giving up a little, in exchange for a memory ceiling you can actually reason about.

If you do want overlap, bound it explicitly — a small pool of concurrent writes rather than an unbounded array of them. The point is that the bound exists and you chose it.

Where this shows up

Anywhere the size of the input is not yours to decide: catalog uploads, report exports, migrations, log processing. The tell is a variable holding “all the records” — const all = await …. That line is where the ceiling disappears.

I have now built this shape twice, in both directions: streaming rows into a catalog service, and streaming millions of records back out of MongoDB. Different problem, same sentence — never hold the whole set.

← All writing