Split a large file

When you need it

You don’t need this to upload. Uploads have no size limit, and Dimension Labs splits a file for you if it is too large.

Use it when your own side can’t handle the whole file: a spreadsheet or script that runs out of memory, a mail or file-share limit when you send a sample to support, or a review you want to do one part at a time.

Splitting by line count with a text tool breaks two things. A quoted field that contains a line break is cut in half, and a conversation can land half in one file and half in the next. This recipe avoids both.

Input

Example input: the combined.csv written by Combine several exports, split into parts of at most 3 rows so the example stays short. For real files, pick a size your tools handle; the script’s default of 50,000 rows is an example, not a limit.

combined.csv

sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,channel,store,account_id
chat-1001,cus_51,true,Is the Pearl store open late?,1771340400000,chat-1001,chat,Portland Pearl,
chat-1001,cus_51,false,It's open until 8 PM.,1771340401000,chat-1001,chat,Portland Pearl,
email-1001,cus_51,true,My wholesale invoice is wrong.,1771344000000,email-1001,email,,ACCT-204
email-1001,cus_51,false,Fixed. A new invoice is attached.,1771347600000,email-1001,email,,ACCT-204
email-1002,cus_90,true,Please pause my subscription.,1772812800000,email-1002,email,,ACCT-311

Output

combined_part1.csv

sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,channel,store,account_id
chat-1001,cus_51,true,Is the Pearl store open late?,1771340400000,chat-1001,chat,Portland Pearl,
chat-1001,cus_51,false,It's open until 8 PM.,1771340401000,chat-1001,chat,Portland Pearl,

combined_part2.csv

sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,channel,store,account_id
email-1001,cus_51,true,My wholesale invoice is wrong.,1771344000000,email-1001,email,,ACCT-204
email-1001,cus_51,false,Fixed. A new invoice is attached.,1771347600000,email-1001,email,,ACCT-204
email-1002,cus_90,true,Please pause my subscription.,1772812800000,email-1002,email,,ACCT-311

Part 1 holds only 2 rows: adding the next session, email-1001 (2 rows), would have made it 4, over the target of 3, and a session is never divided.

Rules

• Parse, don’t count lines. A CSV parser knows that a line break inside quotes is part of a field.
• Repeat the header in every part. Each part is a file of its own and needs a header row.
• Never split a session. All rows of a sessionId go in the same part. A part can be larger than the target when one session is very long.
• Strip a byte order mark from the input so it doesn’t end up glued to the first column name.
• Count before and after. The parts’ rows should add up to the input’s rows.
• If you upload the parts, upload them one at a time into the same source.

Script

Node 18 or later, with npm install papaparse. Works for .csv and .jsonl.

split-file.mjs

// Split a large upload file into parts of about N rows. A session never spans two parts.
// Usage: node split-file.mjs combined.csv 50000      (also works for .jsonl)
import fs from 'node:fs';
import path from 'node:path';
import Papa from 'papaparse';

const [input, size = '50000'] = process.argv.slice(2);
const perPart = Number(size);
const ext = path.extname(input).toLowerCase();
const base = input.slice(0, -ext.length);
const text = fs.readFileSync(input, 'utf8').replace(/^\uFEFF/, '');

let fields, rows;
if (ext === '.jsonl') {
  rows = text.split('\n').filter((l) => l.trim()).map((l) => JSON.parse(l));
} else {
  const parsed = Papa.parse(text, { header: true, skipEmptyLines: 'greedy' }); // keeps quoted line breaks intact
  fields = parsed.meta.fields;
  rows = parsed.data;
}

// Group rows by session, in order of first appearance.
const sessions = new Map();
for (const r of rows) {
  if (!sessions.has(r.sessionId)) sessions.set(r.sessionId, []);
  sessions.get(r.sessionId).push(r);
}

const parts = [[]];
for (const group of sessions.values()) {
  const current = parts.at(-1);
  if (current.length && current.length + group.length > perPart) parts.push([]);
  parts.at(-1).push(...group);
}

parts.forEach((part, i) => {
  const file = `${base}_part${i + 1}${ext}`;
  const body = ext === '.jsonl'
    ? part.map((r) => JSON.stringify(r)).join('\n')
    : Papa.unparse({ fields, data: part.map((r) => fields.map((f) => r[f])) }, { newline: '\n' }); // header in every part
  fs.writeFileSync(file, body + '\n');
  console.log(`${file}: ${part.length} rows`);
});
console.log(`${rows.length} rows, ${sessions.size} sessions, ${parts.length} parts`);

node split-file.mjs combined.csv 3
# combined_part1.csv: 2 rows
# combined_part2.csv: 3 rows
# 5 rows, 3 sessions, 2 parts

In Python

Standard library only, for CSV. It writes the same files.

split_file.py

# Split a large CSV into parts of about N rows. A session never spans two parts.
# Usage: python3 split_file.py combined.csv 50000
import csv, sys
from collections import OrderedDict

src, per_part = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 50000
with open(src, newline="", encoding="utf-8-sig") as f:   # utf-8-sig drops a byte order mark
    reader = csv.DictReader(f)
    fields = reader.fieldnames
    sessions = OrderedDict()
    for r in reader:
        sessions.setdefault(r["sessionId"], []).append(r)

parts = [[]]
for group in sessions.values():
    if parts[-1] and len(parts[-1]) + len(group) > per_part:
        parts.append([])
    parts[-1].extend(group)

base = src[: -len(".csv")]
for i, part in enumerate(parts, start=1):
    with open(f"{base}_part{i}.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
        w.writeheader()
        w.writerows(part)
    print(f"{base}_part{i}.csv: {len(part)} rows")

Variations

• Converting and splitting in one step: the toolkit writes parts directly with "output": { "chunkRows": 50000 }, never splitting a session.
• Files too large for memory: these scripts read the whole file. If the input is sorted by sessionId, a streaming version can write each session as soon as the next one starts. Or ask support to run the transformation for you.