# Split a large file

> Recipe: split a large file into smaller parts with the header in each, without cutting a conversation in two, when your own tools can't handle the whole file.

## When you need it

You don't need this to upload. Uploads have no size limit, and Dimension Labs splits a file for you if it is too large.

Use it when your own side can't handle the whole file: a spreadsheet or script that runs out of memory, a mail or file-share limit when you send a sample to support, or a review you want to do one part at a time.

Splitting by line count with a text tool breaks two things. A quoted field that contains a line break is cut in half, and a conversation can land half in one file and half in the next. This recipe avoids both.

## Input

Example input: the `combined.csv` written by [Combine several exports](/transform/recipe-combine-exports), split into parts of at most 3 rows so the example stays short. For real files, pick a size your tools handle; the script's default of 50,000 rows is an example, not a limit.

```csv title="combined.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,channel,store,account_id
chat-1001,cus_51,true,Is the Pearl store open late?,1771340400000,chat-1001,chat,Portland Pearl,
chat-1001,cus_51,false,It's open until 8 PM.,1771340401000,chat-1001,chat,Portland Pearl,
email-1001,cus_51,true,My wholesale invoice is wrong.,1771344000000,email-1001,email,,ACCT-204
email-1001,cus_51,false,Fixed. A new invoice is attached.,1771347600000,email-1001,email,,ACCT-204
email-1002,cus_90,true,Please pause my subscription.,1772812800000,email-1002,email,,ACCT-311
```

## Output

```csv title="combined_part1.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,channel,store,account_id
chat-1001,cus_51,true,Is the Pearl store open late?,1771340400000,chat-1001,chat,Portland Pearl,
chat-1001,cus_51,false,It's open until 8 PM.,1771340401000,chat-1001,chat,Portland Pearl,
```

```csv title="combined_part2.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,channel,store,account_id
email-1001,cus_51,true,My wholesale invoice is wrong.,1771344000000,email-1001,email,,ACCT-204
email-1001,cus_51,false,Fixed. A new invoice is attached.,1771347600000,email-1001,email,,ACCT-204
email-1002,cus_90,true,Please pause my subscription.,1772812800000,email-1002,email,,ACCT-311
```

Part 1 holds only 2 rows: adding the next session, `email-1001` (2 rows), would have made it 4, over the target of 3, and a session is never divided.

## Rules

- **Parse, don't count lines.** A CSV parser knows that a line break inside quotes is part of a field.
- **Repeat the header in every part.** Each part is a file of its own and needs [a header row](/data/conversation-format#rules-that-break-uploads).
- **Never split a session.** All rows of a `sessionId` go in the same part. A part can be larger than the target when one session is very long.
- **Strip a byte order mark** from the input so it doesn't end up glued to the first column name.
- **Count before and after.** The parts' rows should add up to the input's rows.
- **If you upload the parts**, upload them one at a time into the same source.

## Script

Node 18 or later, with `npm install papaparse`. Works for `.csv` and `.jsonl`.

```js title="split-file.mjs"
// Split a large upload file into parts of about N rows. A session never spans two parts.
// Usage: node split-file.mjs combined.csv 50000      (also works for .jsonl)




const [input, size = '50000'] = process.argv.slice(2);
const perPart = Number(size);
const ext = path.extname(input).toLowerCase();
const base = input.slice(0, -ext.length);
const text = fs.readFileSync(input, 'utf8').replace(/^\uFEFF/, '');

let fields, rows;
if (ext === '.jsonl') {
  rows = text.split('\n').filter((l) => l.trim()).map((l) => JSON.parse(l));
} else {
  const parsed = Papa.parse(text, { header: true, skipEmptyLines: 'greedy' }); // keeps quoted line breaks intact
  fields = parsed.meta.fields;
  rows = parsed.data;
}

// Group rows by session, in order of first appearance.
const sessions = new Map();
for (const r of rows) {
  if (!sessions.has(r.sessionId)) sessions.set(r.sessionId, []);
  sessions.get(r.sessionId).push(r);
}

const parts = [[]];
for (const group of sessions.values()) {
  const current = parts.at(-1);
  if (current.length && current.length + group.length > perPart) parts.push([]);
  parts.at(-1).push(...group);
}

parts.forEach((part, i) => {
  const file = `${base}_part${i + 1}${ext}`;
  const body = ext === '.jsonl'
    ? part.map((r) => JSON.stringify(r)).join('\n')
    : Papa.unparse({ fields, data: part.map((r) => fields.map((f) => r[f])) }, { newline: '\n' }); // header in every part
  fs.writeFileSync(file, body + '\n');
  console.log(`${file}: ${part.length} rows`);
});
console.log(`${rows.length} rows, ${sessions.size} sessions, ${parts.length} parts`);
```

```bash
node split-file.mjs combined.csv 3
# combined_part1.csv: 2 rows
# combined_part2.csv: 3 rows
# 5 rows, 3 sessions, 2 parts
```

## In Python

Standard library only, for CSV. It writes the same files.

```python title="split_file.py"
# Split a large CSV into parts of about N rows. A session never spans two parts.
# Usage: python3 split_file.py combined.csv 50000

from collections import OrderedDict

src, per_part = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 50000
with open(src, newline="", encoding="utf-8-sig") as f:   # utf-8-sig drops a byte order mark
    reader = csv.DictReader(f)
    fields = reader.fieldnames
    sessions = OrderedDict()
    for r in reader:
        sessions.setdefault(r["sessionId"], []).append(r)

parts = [[]]
for group in sessions.values():
    if parts[-1] and len(parts[-1]) + len(group) > per_part:
        parts.append([])
    parts[-1].extend(group)

base = src[: -len(".csv")]
for i, part in enumerate(parts, start=1):
    with open(f"{base}_part{i}.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
        w.writeheader()
        w.writerows(part)
    print(f"{base}_part{i}.csv: {len(part)} rows")
```

## Variations

- **Converting and splitting in one step**: the [toolkit](/transform/toolkit) writes parts directly with `"output": { "chunkRows": 50000 }`, never splitting a session.
- **Files too large for memory**: these scripts read the whole file. If the input is sorted by `sessionId`, a streaming version can write each session as soon as the next one starts. Or [ask support](/transform#get-help-from-dimension-labs) to run the transformation for you.