# The toolkit

> One configurable script, the toolkit, that turns CSV, TSV, JSONL, JSON or XLSX exports into upload-ready files, with a report of every row it dropped or fixed.

One-off conversion scripts are easy to write and hard to reuse: column positions, sheet names and file paths end up hard-coded, and next month's export needs a new script. The toolkit, `to-dimensionlabs.mjs`, keeps everything that changes in a small JSON mapping file. You write the mapping once per export type and rerun it every time.

- Download the script: [to-dimensionlabs.mjs](/templates/to-dimensionlabs.mjs)
- Download an example mapping: [to-dimensionlabs.example.json](/templates/to-dimensionlabs.example.json)

For a quick one-time conversion without code, the in-browser [converter](/data/convert-a-file) may be all you need. Use the toolkit when the conversion repeats, or you need HTML stripping, skipped system rows, transcript splitting, a sheet other than the first, a header that isn't in row 1, a time zone the converter doesn't list, nested JSON paths or split output.

## What it does

- Reads **CSV, TSV, JSONL, JSON or XLSX** (the format is picked from the file extension).
- Maps your columns to `text`, `userId`, `incoming`, `sessionId`, `conversationId` and `dimensionlabs_timestamp`, and keeps the other columns as metadata (or the ones you list).
- Sets `incoming` from value lists (`"customer"` is `true`, `"agent"` and `"bot"` are `false`), or a constant.
- Fills a blank `userId` with the `sessionId`, and gives the agent's or bot's replies the customer's `userId`, so both sides of a conversation carry the end user's id.
- Reads timestamps as Unix milliseconds, Unix seconds, ISO 8601, clock times in a named time zone, or Excel serial dates.
- Replaces colons in ids, flattens line breaks in text, optionally strips HTML, drops rows with empty text.
- Splits transcripts with a speaker pattern, and reshapes wide surveys into question and answer rows, taking the question wording from the export's own wording row.
- Removes duplicate messages, keeps messages in a session at least 1,000 ms apart, and splits output into parts of about N rows without cutting a session.
- Writes a **report**: rows in, messages out, every dropped row counted by reason, every fix counted, and a note when a kept column looks like personal data.

It writes CSV by default, or JSONL. A JSONL line has the same shape as a [universal API](/api/tracker#the-message) message: extra fields go in a nested `platformJson` object and `intent` is an object (`{ "name": "...", "confidence": 0.94 }`). Uploads accept that shape, so either format works for a file upload.

## Install and run

Node 18 or later.

```bash
npm install papaparse      # always
npm install xlsx           # only if you read .xlsx files
node to-dimensionlabs.mjs --config mapping.json --input export.csv --out out/export
```

`--input` and `--out` override `input.path` and `output.path` in the mapping. Output files are `<out>-1.csv`, `<out>-2.csv` and so on, plus `<out>.report.json`. The script holds the whole input in memory; for an input larger than your machine's memory, [split it](/transform/recipe-split-large-file) first or [ask support](/transform#get-help-from-dimension-labs).

## Example: a help-desk export

Example input, `lumen-tickets.csv`:

```csv title="lumen-tickets.csv"
ticket_id,author_id,author_type,body,created_at,store_name,customer_email,customer_phone,priority
4471,c-88,end-user,"<div>My card was charged twice<br>for one flat white.</div>",1/3/2026 9:12 AM,Portland Pearl,ana@example.com,555-0100,high
4471,a-2,agent,Refund is on its way.,1/3/2026 9:20 AM,Portland Pearl,ana@example.com,555-0100,high
4471,sys,system,Ticket closed,1/3/2026 9:21 AM,Portland Pearl,ana@example.com,555-0100,high
```

The example mapping, [to-dimensionlabs.example.json](/templates/to-dimensionlabs.example.json). Keys that start with `_` are notes and are ignored:

```json title="to-dimensionlabs.example.json"
{
  "_about": "Example mapping for to-dimensionlabs.mjs: a Lumen Coffee help-desk export, one row per message. Keys that start with _ are ignored. Docs: https://docs.dimensionlabs.io/transform/toolkit",

  "input": {
    "format": "auto",
    "_format": "auto picks from the extension: .csv, .tsv, .jsonl, .json, .xlsx",
    "sheet": "Messages",
    "headerRow": 1,
    "skipRowsAfterHeader": 0,
    "_skipRowsAfterHeader": "survey tools often add a row of question text under the column names: set 1 to skip it (Qualtrics adds two: set 2)"
  },

  "shape": "messages",
  "_shape": "messages = one row per message; transcript = one row per conversation with the whole transcript in a cell; survey = one row per respondent with one column per question",

  "columns": {
    "text": "body",
    "userId": "author_id",
    "sessionId": "ticket_id",
    "conversationId": "ticket_id",
    "timestamp": "created_at",
    "_columns": "source column names; use a dot path such as sender.id for nested JSON. Never use an email, phone number or name as userId or sessionId. Optional: intent and intentConfidence, for an intent-based chatbot"
  },

  "userIdForReplies": "customer",
  "_userIdForReplies": "customer (default) gives the agent's and bot's replies the end user's userId, so both sides of a conversation carry the customer's id; own keeps each row's userId",

  "incoming": {
    "column": "author_type",
    "true": ["customer", "end-user"],
    "false": ["agent", "bot", "system"],
    "_incoming": "values are matched case-insensitively; rows with any other value are dropped and counted. Use \"incoming\": true when every row is from the customer"
  },

  "timestamp": {
    "format": "auto",
    "timeZone": "America/Los_Angeles",
    "dayFirst": false,
    "_timestamp": "auto reads Unix ms, Unix seconds, ISO 8601 and Excel serial dates. timeZone applies to times written without a zone, such as 1/3/2026 9:12 AM"
  },

  "clean": {
    "flattenNewlines": true,
    "stripHtml": true,
    "replaceColonsWith": "-"
  },

  "metadata": {
    "keep": "unmapped",
    "drop": ["customer_email", "customer_phone"],
    "rename": { "store_name": "store" },
    "_metadata": "keep: unmapped (every column not used above), none, or a list of column names. Drop personal data you don't need"
  },

  "constants": { "channel": "email" },

  "skip": [
    { "column": "author_type", "values": ["system"] }
  ],
  "_skip": "rows whose column holds one of these values are dropped and counted, for example automatic ticket-closed notes",

  "dedupe": true,
  "spaceMessagesMs": 1000,

  "transcript": {
    "_used_when": "shape is transcript",
    "column": "transcript",
    "speakerPattern": "^(Agent|Customer):\\s*",
    "incomingSpeakers": ["Customer"],
    "outgoingSpeakers": ["Agent"],
    "secondsApart": 1,
    "userId": "session"
  },

  "survey": {
    "_used_when": "shape is survey",
    "questionsFromRow": 2,
    "_questionsFromRow": "the file row (counting from 1) that holds the question wording; it must be one of the rows skipped by input.skipRowsAfterHeader. A question's own text, if set, wins",
    "questions": [
      { "column": "why_score", "text": "Why did you give that score?" },
      { "column": "improve", "text": "What one thing should we improve?" }
    ]
  },

  "output": {
    "format": "csv",
    "path": "out/lumen-tickets",
    "chunkRows": 50000,
    "_output": "csv or jsonl. chunkRows 0 writes one file; otherwise files of about that many rows, never splitting a session"
  }
}
```

```bash
node to-dimensionlabs.mjs --config to-dimensionlabs.example.json --input lumen-tickets.csv
```

```text
Rows in: 3  Messages out: 2  Sessions: 1  Dropped: 1
  dropped 1: skipped: author_type is system
  fixed 1: html removed from text
  fixed 1: line breaks in text replaced with spaces
  fixed 1: replies given the customer's userId
  wrote out/lumen-tickets-1.csv (2 rows)
  report: out/lumen-tickets.report.json
```

```csv title="out/lumen-tickets-1.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,store,priority,channel
4471,c-88,true,My card was charged twice for one flat white.,1767460320000,4471,Portland Pearl,high,email
4471,c-88,false,Refund is on its way.,1767460800000,4471,Portland Pearl,high,email
```

The email and phone columns were dropped, `store_name` was renamed, the automatic "Ticket closed" note was skipped, `channel` was added to every row, and the agent's reply carries the customer's id, `c-88`.

## Mapping reference

| Key | What it does |
|---|---|
| `input.format` | `auto` (default, from the extension), `csv`, `tsv`, `jsonl`, `json` or `xlsx`. |
| `input.sheet` | XLSX: the sheet name. Default: the first sheet. |
| `input.headerRow` | The row that holds the column names, counting from 1. Default `1`. |
| `input.skipRowsAfterHeader` | Rows to skip under the header, such as a row of question wording. Default `0`. |
| `input.delimiter` | Force a delimiter. Default: detected (tab for `.tsv`). |
| `input.recordsPath` | JSON: the key that holds the array of records, if the file isn't an array. Default `data`. |
| `shape` | `messages` (one row per message, default), `transcript` (one row per conversation, transcript in a cell) or `survey` (one row per respondent). |
| `columns.text`, `.userId`, `.sessionId`, `.conversationId`, `.timestamp` | Source column names. Dot paths such as `sender.id` read nested JSON. `conversationId` defaults to the `sessionId`. Never map an email address, phone number or name to `userId` or `sessionId`: use the tool's internal ids. |
| `columns.intent`, `.intentConfidence` | Only for an intent-based chatbot: the intent name and its confidence. JSONL output writes them as `"intent": { "name": "...", "confidence": 0.94 }`. |
| `userIdForReplies` | `customer` (default): every outgoing message in a session takes the `userId` of the session's first incoming message (or the `sessionId` if there is none), so the bot's and agent's replies carry the end user's id. `own` keeps each row's own `userId`. |
| `incoming` | `{ "column": "...", "true": [...], "false": [...] }`, matched case-insensitively. With only one list, every other value is the other direction. A value in neither list drops the row. Or `true` / `false` for the whole file. |
| `timestamp.format` | `auto` (default), `ms`, `s` or `excel`. `auto` reads 13-digit milliseconds, 10-digit seconds, numbers below 1,000,000 as Excel serial dates, and date text. Values in scientific notation are dropped, because their digits are lost. |
| `timestamp.timeZone` | The zone of clock times without one, such as `America/Chicago`. Default `UTC`, with a note in the report. |
| `timestamp.dayFirst` | `true` when `03/01/2026` means 3 January. Default `false`. |
| `clean.flattenNewlines` | Replace line breaks in text with spaces. Default `true`. |
| `clean.stripHtml` | Remove HTML tags and decode entities. Default `false`. |
| `clean.replaceColonsWith` | Replacement for `:` in `sessionId` and `conversationId`. Default `-`. |
| `metadata.keep` | `unmapped` (default: every column not used above), `none`, or a list of columns. |
| `metadata.drop` | Columns never to keep, such as emails and phone numbers. |
| `metadata.rename` | `{ "source name": "new name" }`. |
| `constants` | Columns added to every row, such as `{ "channel": "email" }`. |
| `skip` | `[{ "column": "...", "values": [...] }]`: rows to drop and count, such as system notes. |
| `dedupe` | `true` removes repeated messages (same session, direction, time and text). Default `false`. |
| `spaceMessagesMs` | Minimum gap between messages in a session; later messages are moved. Default `1000`. `0` turns it off. |
| `transcript.column`, `.speakerPattern`, `.incomingSpeakers`, `.outgoingSpeakers`, `.secondsApart`, `.userId`, `.stripQuotes` | For `shape: transcript`. The pattern's first group is the speaker label. Lines without a label join the turn above. `userId: "speaker"` gives each speaker their own id; the default keeps the row's `userId` (or the session id) on every turn. |
| `survey.questions` | For `shape: survey`: a list of question columns, as names (`["why_score", "improve"]`) or as `{ "column": "...", "text": "the question as asked" }`. Each answered question becomes an outgoing question row and an incoming answer row. Blank answers are skipped. |
| `survey.questionsFromRow` | The file row, counting from 1, that holds the question wording, such as `2` when the export puts it under the column names. It must be one of the rows skipped by `input.skipRowsAfterHeader`. A question's own `text` wins over it; with neither, the column name is used and the report says so. |
| `output.format` | `csv` (default) or `jsonl` (one universal API message per line, metadata nested in `platformJson`). Both upload. |
| `output.path` | Output prefix. Default: next to the input. |
| `output.chunkRows` | Split output into parts of about this many rows, never cutting a session. Default `0` (one file). |

## Tested examples

Each example below was run exactly as shown.

### Value lists, HTML, colons, seconds, duplicates and empty rows

```csv title="tickets.csv"
ticket_id,author,author_type,body,created_at,store,customer_email
LC:1001,cust-17,Customer,"<p>My oat latte was cold &amp; late.</p>",2026-01-03T17:12:00Z,Portland Pearl,ana@example.com
LC:1001,agent-4,Agent,"Sorry about that!
We've refunded you.",2026-01-03T17:12:00Z,Portland Pearl,ana@example.com
LC:1001,cust-17,Customer,,2026-01-03T17:14:00Z,Portland Pearl,ana@example.com
LC:1002,,customer,Do you sell decaf beans?,1767460320,Austin Eastside,bo@example.com
LC:1002,agent-9,Bot,Yes: our Night Owl decaf.,1767460380000,Austin Eastside,bo@example.com
LC:1002,agent-9,Bot,Yes: our Night Owl decaf.,1767460380000,Austin Eastside,bo@example.com
LC:1003,cust-22,Visitor,Is the Pearl store open on Sunday?,2026-01-04T10:00:00Z,Portland Pearl,cy@example.com
```

```json title="tickets.json"
{
  "input": { "format": "auto" },
  "shape": "messages",
  "columns": { "text": "body", "userId": "author", "sessionId": "ticket_id", "timestamp": "created_at" },
  "incoming": { "column": "author_type", "true": ["customer"], "false": ["agent", "bot"] },
  "timestamp": { "format": "auto", "timeZone": "UTC" },
  "clean": { "stripHtml": true, "flattenNewlines": true, "replaceColonsWith": "-" },
  "metadata": { "keep": "unmapped", "drop": ["customer_email"] },
  "constants": { "channel": "email" },
  "dedupe": true,
  "spaceMessagesMs": 1000,
  "output": { "format": "csv", "chunkRows": 0 }
}
```

```bash
node to-dimensionlabs.mjs --config tickets.json --input tickets.csv --out out/tickets
```

```text
Rows in: 7  Messages out: 4  Sessions: 2  Dropped: 3
  dropped 1: empty text
  dropped 1: incoming value not recognized: "visitor"
  dropped 1: duplicate message
  fixed 3: ids with colons rewritten
  fixed 1: html removed from text
  fixed 1: line breaks in text replaced with spaces
  fixed 1: timestamps in seconds converted to milliseconds
  fixed 1: blank userId filled from sessionId
  fixed 2: replies given the customer's userId
  fixed 1: messages moved to be 1000 ms apart
  wrote out/tickets-1.csv (4 rows)
  report: out/tickets.report.json
```

```csv title="out/tickets-1.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,store,channel
LC-1001,cust-17,true,My oat latte was cold & late.,1767460320000,LC-1001,Portland Pearl,email
LC-1001,cust-17,false,Sorry about that! We've refunded you.,1767460321000,LC-1001,Portland Pearl,email
LC-1002,LC-1002,true,Do you sell decaf beans?,1767460320000,LC-1002,Austin Eastside,email
LC-1002,LC-1002,false,Yes: our Night Owl decaf.,1767460380000,LC-1002,Austin Eastside,email
```

The agent's and the bot's replies carry the customer's id: `cust-17` in `LC-1001`, and in `LC-1002`, where the customer's id was blank, the session id.

### Clock times in a named time zone, TSV in, JSONL out

```text title="chat.tsv (tab-separated)"
conv	sender	who	message	sent
c1	u1	customer	Hi, is the Pearl store open?	1/3/2026 9:12 AM
c1	bot	bot	Yes, until 6 PM.	1/3/2026 9:12 AM
c2	u2	customer	Summer hours?	7/4/2026 12:30 PM
c3	u3	customer	Loyalty card lost	Oct 3 2025  9:12AM
```

```json title="chat.json"
{
  "columns": { "text": "message", "userId": "sender", "sessionId": "conv", "timestamp": "sent" },
  "incoming": { "column": "who", "true": ["customer"] },
  "timestamp": { "timeZone": "America/Los_Angeles" },
  "output": { "format": "jsonl" }
}
```

```bash
node to-dimensionlabs.mjs --config chat.json --input chat.tsv --out out/chat
```

```text
Rows in: 4  Messages out: 4  Sessions: 3  Dropped: 0
  fixed 1: replies given the customer's userId
  fixed 1: messages moved to be 1000 ms apart
  wrote out/chat-1.jsonl (4 rows)
  report: out/chat.report.json
```

```json title="out/chat-1.jsonl"
{"text":"Loyalty card lost","userId":"u3","incoming":true,"sessionId":"c3","conversationId":"c3","dimensionlabs_timestamp":1759507920000}
{"text":"Hi, is the Pearl store open?","userId":"u1","incoming":true,"sessionId":"c1","conversationId":"c1","dimensionlabs_timestamp":1767460320000}
{"text":"Yes, until 6 PM.","userId":"u1","incoming":false,"sessionId":"c1","conversationId":"c1","dimensionlabs_timestamp":1767460321000}
{"text":"Summer hours?","userId":"u2","incoming":true,"sessionId":"c2","conversationId":"c2","dimensionlabs_timestamp":1783193400000}
```

The bot's reply carries the customer's id, `u1`, not `bot`. `1/3/2026 9:12 AM` in Los Angeles is `1767460320000` (17:12 UTC, standard time); `7/4/2026 12:30 PM` is `1783193400000` (19:30 UTC, daylight saving time).

### Excel: a named sheet, a title row, serial dates, long ids and an error cell

The workbook `chats.xlsx` has a **Read me** sheet and a **Chats** sheet with a title in row 1. Build it with:

```js title="make-xlsx.mjs"


XLSX.set_fs(fs);
const rows = [
  ['Lumen Coffee chat export'],
  ['Chat ID', 'Visitor', 'Role', 'Message', 'Sent at', 'Order no'],
  [880011223344, 'v-1', 'Visitor', 'Can I change my pickup store?', 46025.38333333333, 4400123456789],
  [880011223344, '', 'Barista', 'Sure, which store?', 46025.38333333333, 4400123456789],
  [880011223345, 'v-2', 'Visitor', 'Gift card balance?', 46026.5, { t: 'e', v: 0x2A }],
];
const ws = XLSX.utils.aoa_to_sheet(rows);
// mark the date column as a date format so it looks like a real export
for (const r of [3, 4, 5]) { const c = ws['E' + r]; c.z = 'm/d/yyyy h:mm'; }
ws['F5'] = { t: 'e', v: 0x2A, w: '#N/A' };
const wb = XLSX.utils.book_new();
XLSX.utils.book_append_sheet(wb, XLSX.utils.aoa_to_sheet([['notes']]), 'Read me');
XLSX.utils.book_append_sheet(wb, ws, 'Chats');
XLSX.writeFile(wb, 'chats.xlsx');
console.log('ok');
```

```json title="xlsx.json"
{
  "input": { "sheet": "Chats", "headerRow": 2 },
  "columns": { "text": "Message", "userId": "Visitor", "sessionId": "Chat ID", "timestamp": "Sent at" },
  "incoming": { "column": "Role", "true": ["visitor"], "false": ["barista"] },
  "timestamp": { "format": "auto", "timeZone": "America/Chicago" },
  "metadata": { "rename": { "Order no": "order_number" } }
}
```

```bash
node make-xlsx.mjs
node to-dimensionlabs.mjs --config xlsx.json --input chats.xlsx --out out/xlsx
```

```text
Rows in: 3  Messages out: 3  Sessions: 2  Dropped: 0
  fixed 3: excel serial dates converted
  fixed 1: blank userId filled from sessionId
  fixed 1: replies given the customer's userId
  fixed 1: messages moved to be 1000 ms apart
  note: read sheet "Chats"
  wrote out/xlsx-1.csv (3 rows)
  report: out/xlsx.report.json
```

```csv title="out/xlsx-1.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,order_number
880011223344,v-1,true,Can I change my pickup store?,1767453120000,880011223344,4400123456789
880011223344,v-1,false,"Sure, which store?",1767453121000,880011223344,4400123456789
880011223345,v-2,true,Gift card balance?,1767549600000,880011223345,
```

The order number in the last row was a `#N/A` cell and is left empty. The barista's reply had no id and now carries the visitor's, `v-1`.

### Transcripts

```csv title="calls.csv"
call_id,started_at,store,transcript
call-501,2026-02-10T15:00:00Z,Austin Eastside,"Agent: Thanks for calling Lumen Coffee, this is Sam.
Customer: Hi, my mobile order never showed up.
It was a cold brew and a scone.
Agent: I'm sorry. Let me look that up.
Customer: Thanks."
call-502,2026-02-11T16:30:00Z,Portland Pearl,"Customer: Do you roast on site?
Agent: We do, every Tuesday."
```

```json title="calls.json"
{
  "shape": "transcript",
  "columns": { "sessionId": "call_id", "timestamp": "started_at" },
  "transcript": { "column": "transcript", "speakerPattern": "^(Agent|Customer):\\s*", "incomingSpeakers": ["Customer"], "secondsApart": 5 },
  "constants": { "channel": "phone" }
}
```

```bash
node to-dimensionlabs.mjs --config calls.json --input calls.csv --out out/calls
```

Transcript turns all carry the row's `userId`, or the session id when there is none, so both sides already share one id.

```csv title="out/calls-1.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,store,channel,speaker
call-501,call-501,false,"Thanks for calling Lumen Coffee, this is Sam.",1770735600000,call-501,Austin Eastside,phone,Agent
call-501,call-501,true,"Hi, my mobile order never showed up. It was a cold brew and a scone.",1770735605000,call-501,Austin Eastside,phone,Customer
call-501,call-501,false,I'm sorry. Let me look that up.,1770735610000,call-501,Austin Eastside,phone,Agent
call-501,call-501,true,Thanks.,1770735615000,call-501,Austin Eastside,phone,Customer
call-502,call-502,true,Do you roast on site?,1770827400000,call-502,Portland Pearl,phone,Customer
call-502,call-502,false,"We do, every Tuesday.",1770827405000,call-502,Portland Pearl,phone,Agent
```

### A wide survey with a row of question wording

The export puts the question wording in row 2, under the column names. `questionsFromRow` takes the question text from that row, and `skipRowsAfterHeader` keeps it from being read as a respondent.

```csv title="survey.csv"
response_id,submitted_at,nps,why_score,improve
Response ID,Submitted,How likely are you to recommend Lumen Coffee?,Why did you give that score?,What one thing should we improve?
R-1,2026-03-01 08:15,9,Friendly baristas.,Faster mobile pickup.
R-2,2026-03-01 09:40,4,,More oat milk options.
R-3,2026-03-02 18:05,7,,
```

```json title="survey.json"
{
  "input": { "skipRowsAfterHeader": 1 },
  "shape": "survey",
  "columns": { "sessionId": "response_id", "timestamp": "submitted_at" },
  "timestamp": { "timeZone": "America/New_York" },
  "survey": { "questionsFromRow": 2, "questions": ["why_score", "improve"] }
}
```

```bash
node to-dimensionlabs.mjs --config survey.json --input survey.csv --out out/survey
```

```text
Rows in: 3  Messages out: 6  Sessions: 2  Dropped: 3
  dropped 3: blank survey answer (question and answer skipped)
  fixed 6: blank userId filled from sessionId
  note: question wording read from row 2
  note: some respondents answered none of the listed questions and produce no rows
  wrote out/survey-1.csv (6 rows)
  report: out/survey.report.json
```

```csv title="out/survey-1.csv"
sessionId,userId,incoming,text,dimensionlabs_timestamp,conversationId,nps,question
R-1,R-1,false,Why did you give that score?,1772370900000,R-1,9,why_score
R-1,R-1,true,Friendly baristas.,1772370901000,R-1,9,why_score
R-1,R-1,false,What one thing should we improve?,1772370902000,R-1,9,improve
R-1,R-1,true,Faster mobile pickup.,1772370903000,R-1,9,improve
R-2,R-2,false,What one thing should we improve?,1772376000000,R-2,4,improve
R-2,R-2,true,More oat milk options.,1772376001000,R-2,4,improve
```

A Qualtrics CSV has two rows under the column names: the question wording, then a row of `{"ImportId":...}` values. Use `"skipRowsAfterHeader": 2` and `"questionsFromRow": 2`.

### Nested JSONL and split output

```json title="messages.jsonl (the last line is deliberately broken)"
{"ticket":{"id":"T-1"},"sender":{"type":"customer","id":"u-301"},"body":"Where is my bag of Night Owl?","created":"2026-04-01T10:00:00Z","tags":["shipping"]}
{"ticket":{"id":"T-1"},"sender":{"type":"agent","id":"a-12"},"body":"It ships today.","created":"2026-04-01T10:05:00Z","tags":["shipping"]}
{"ticket":{"id":"T-2"},"sender":{"type":"customer","id":"u-302"},"body":"Cancel my subscription","created":"2026-04-02T11:00:00Z","tags":[]}
{"ticket":{"id":"T-3"},"sender":{"type":"customer","id":"u-303"},"body":"Thanks!","created":"2026-04-03T11:00:00Z","tags":[]}
{"ticket":{"id":"T-3"},"sender":{"type":"agent","id":"a-12"},"body":"Anytime.","created":"2026-04-03T11:01:00Z","tags":[]}
not json
```

```json title="jsonl.json"
{
  "columns": { "text": "body", "userId": "sender.id", "sessionId": "ticket.id", "timestamp": "created" },
  "incoming": { "column": "sender.type", "true": ["customer"], "false": ["agent"] },
  "metadata": { "keep": ["tags"] },
  "output": { "format": "csv", "chunkRows": 2 }
}
```

```bash
node to-dimensionlabs.mjs --config jsonl.json --input messages.jsonl --out out/nested
```

```text
Rows in: 6  Messages out: 5  Sessions: 3  Dropped: 1
  dropped 1: line 6 is not valid JSON
  fixed 2: replies given the customer's userId
  wrote out/nested-1.csv (2 rows)
  wrote out/nested-2.csv (1 row)
  wrote out/nested-3.csv (2 rows)
  report: out/nested.report.json
```

With `chunkRows` set to 2, ticket `T-1` (2 rows), `T-2` (1 row) and `T-3` (2 rows) each land in their own part: no part can take the next ticket without going over 2 rows.

## Read the report

Every run writes `<out>.report.json`:

```json title="out/lumen-tickets.report.json"
{
  "input": "lumen-tickets.csv",
  "rowsIn": 3,
  "messagesOut": 2,
  "sessions": 1,
  "dropped": {
    "skipped: author_type is system": 1
  },
  "fixed": {
    "html removed from text": 1,
    "line breaks in text replaced with spaces": 1,
    "replies given the customer's userId": 1
  },
  "notes": [],
  "files": [
    {
      "file": "out/lumen-tickets-1.csv",
      "rows": 2
    }
  ],
  "firstTimestamp": "2026-01-03T17:12:00.000Z",
  "lastTimestamp": "2026-01-03T17:20:00.000Z"
}
```

- **`rowsIn` and `messagesOut`** should make sense together. For `messages`, out is in minus dropped. Transcripts and surveys write several rows per input row.
- **`dropped`** lists every reason a row (or a survey answer) was left out. An unexpected `incoming value not recognized` means a value is missing from your lists.
- **`fixed`** counts each fix. Large numbers are fine when you expect them (seconds converted to milliseconds on every row of a file that uses seconds); a surprise is worth a look.
- **`notes`** flags things to check, such as times read as UTC because no time zone was set, or kept columns that look like personal data (add those to `metadata.drop`; see [PII redaction](/data/pii-redaction)).
- **`firstTimestamp` and `lastTimestamp`** should match the date range of your export.

Then run the [file checker](/data/check-a-file) on the output and go through [Before you send it](/transform/checklist).

## Monthly exports

Keep one mapping file per export type next to your data, under a name such as `helpdesk-tickets.mapping.json`. Next month, run the same command on the new file and upload it to the same source. If the new export overlaps last month's, that's fine: Dimension Labs removes duplicate messages when it ingests them. If the export's columns change, the run tells you: a missing column shows up as dropped rows (`no sessionId`, `empty text`) instead of quietly shifted data.

## What it doesn't do

- Cut quoted replies from emails: see [Clean ticket and email text](/transform/recipe-clean-text).
- Shift dates by a known error, such as a wrong year: see [Fix timestamps](/transform/recipe-fix-timestamps).
- Join business fields from a second file: see [Attach business fields](/transform/recipe-business-fields).
- Combine several exports: run it once per export, then see [Combine several exports](/transform/recipe-combine-exports).
- Name speakers' sides in call transcripts that use people's names: see [Turn call transcripts into turns](/transform/recipe-call-transcripts).