# Before you send it

> The checks to run on a converted file before you upload it or send it to the API, the counts to compare, and the mistakes that most often slip through.

A converted file can look right and still be wrong in ways that only show up weeks later as odd numbers. Run these checks every time, in this order.

## 1. Run the file checker

Drop the output into the [file checker](/data/check-a-file). It runs in your browser and finds the format problems behind most failed uploads: empty text, `TRUE`/`FALSE` in capitals, timestamps in seconds or scientific notation, colons in ids, line breaks in text, blank `userId`s, messages less than a second apart. Fix what it reports, or take its fixed copy. Codes are explained in [Upload errors](/data/upload-errors).

## 2. Compare the counts

Write these down for the export and for the converted file. If you used the [toolkit](/transform/toolkit), most of them are in its report.

| Count | What you expect |
|---|---|
| Rows in the export | Matches what the source system says it exported |
| Rows written | Rows in, minus rows dropped (for one row per message); more than rows in for transcripts and surveys |
| Rows dropped, by reason | Every reason makes sense: empty text, system notes, unanswered questions |
| Distinct `sessionId`s | About the number of conversations, tickets, calls or respondents in the source |
| Rows per session | No session with one huge count (a sign that the session key is too broad, such as a customer id instead of a ticket id) and not every session with exactly one row (a sign that the transcript wasn't split) |
| `incoming: true` share | Plausible for the channel: reviews are all `true` except owner replies; surveys are half `true`, one answer per question; chats and tickets have both sides |
| First and last timestamp | The export's date range, in the right year, not 1970 |
| Rows per day | No single day holding everything (a sign that times were missing and filled with one date) |

## 3. Read ten conversations

Pick ten sessions at random and read them top to bottom in the converted file. Check that:

- each message is on the right side (`incoming` is `true` only for the customer);
- the messages are in the order they happened;
- no message is cut in half or glued to the next one;
- the text has no HTML, quoted replies or placeholder text;
- the extra columns hold the values you expect.

## 4. Check the ids

- `sessionId` is the ticket, chat, call or response id, and it is the same on every rerun.
- No `sessionId` or `conversationId` contains a colon.
- In bot and agent conversations, the replies carry the customer's `userId`, not the bot's or agent's.
- No id is an email address, phone number or name.
- Long ids are whole numbers, not `1.70324E+12` or `12345.0`.
- If you will join business data, the join key has the same format as in the business file: the [file checker's join test](/data/check-a-file#test-a-join) shows the match rate.

## 5. Check what you're sending

- No personal data you don't need in the extra columns: emails, phone numbers, addresses, names, access tokens. See [PII redaction](/data/pii-redaction).
- Extra column names are consistent across files and months. Each one becomes a field the agent can filter on, so `Store`, `store` and `store_name` would be three different fields.
- Values in extra columns use one encoding: `Yes`, `yes` and `1` are three different values ([conversation format](/data/conversation-format#small-rules-that-keep-results-clean)).

## 6. After the upload

Ask the agent *"How many rows are in my new source, and what date range does it cover?"* and compare with your counts. A file can land partly without an error ([Upload a file](/data/upload-a-file#after-you-upload)). An uploaded file is not enriched until you start a [backfill](/dimensions/backfill) from the **Backfill** tab.

## The mistakes conversions make most

These come from real conversion scripts that each worked once:

| Mistake | What it looks like afterwards | Prevention |
|---|---|---|
| Columns read by position | Text in the timestamp field, or ids where the text should be, after the export added a column | Map columns by name |
| Direction inverted, or guessed from "who spoke first" | Customers appear to say what agents said; complaint Dimensions land on agents | Map from a role column or a list of your people; read ten conversations |
| Random ids for users or sessions | Every rerun creates new users and new copies of the same conversations, which duplicate removal can't catch | Build ids from ids in the export |
| Month off by one in hand-built dates | Everything a month late | Parse dates with a library, or check the first and last timestamp |
| Clock times read in the machine's time zone | Times off by hours, different on each computer | Name the time zone |
| Timestamps rewritten to be one second apart across a whole history | Weeks between messages become seconds; time-based questions go wrong | Keep real times; only move messages that collide |
| Placeholder text for empty answers or bodies | `[No Response]` enriched as if a customer wrote it | Drop empty rows |
| One session per customer instead of per ticket | A few giant conversations | Use the ticket, chat or call id |
| Business records sent as messages | Orders and CRM notes enriched as customer text | Upload them as [business data](/data/business-data-format) or as extra columns |
| Excel resaved as CSV | Long ids and timestamps in scientific notation, digits lost | Read the `.xlsx` directly |
| Output appended to last run's files | Your counts don't match the export, and last month's rows go up again | Write to a fresh output folder |
| Personal data left in metadata, or used as an id | Emails and addresses stored with every message | Drop them, or hash a key you need to join on |

## Sending through the API instead

If you send the converted rows to the [universal API](/api/tracker) rather than uploading a file:

- Send anything older than an hour to `/trackhistorical`, and run a [backfill](/dimensions/backfill) from the **Backfill** tab afterwards.
- Test with a handful of messages first and check they arrived before sending the rest.
- Send sequentially or with low concurrency. Rate limits are not published.
- Process rows in a fixed order and log what you've sent, so a run that stops halfway can resume where it stopped. Re-sending is safe: duplicates are removed when they are ingested.
- Retry only failures that a retry can fix (network timeouts, HTTP `429` and `5xx` responses), with exponential backoff, and wait as long as a `Retry-After` header says. Any other `4xx` response means the request itself was rejected: fix the message, don't resend it unchanged.
- Write every message that still failed to a file, so you can resend exactly those.

## Still stuck

Email **support@dimensionlabs.io** with the details listed in [Get help](/transform#get-help-from-dimension-labs). Dimension Labs technical support can run large or complex transformations for you.