# Amazon S3

> Send conversation files dropped in Amazon S3 to Dimension Labs with the universal API.

If your systems drop files of conversations into an S3 bucket (a nightly export, a vendor's data dump), you can send them to Dimension Labs from there. A scheduled job on your side lists the files that are new since its last run, reads each one, and posts every row as a message to the [universal API](/api/tracker). Files can be CSV or JSONL (one JSON object per line).

## What you need

- An AWS role that can create IAM roles and policies.
- An IAM role for the job, such as a Lambda execution role, or an IAM user's access key for a job that runs outside AWS.
- A Dimension Labs API key for a new **Universal API** source. See [Your API key](/api/tracker#your-api-key).

## 1. Get access

The job needs two permissions. Listing a bucket needs `s3:ListBucket` ([ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)), and reading a file needs `s3:GetObject` ([GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)). Add this policy to the job's role, with your bucket name and folder:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::YOUR_BUCKET" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::YOUR_BUCKET/conversations/*" }
  ]
}
```

If the files are encrypted with an AWS KMS key, the role also needs permission to use that key ([GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)). Outside AWS, keep the access key in a secrets manager, never in code you commit.

## 2. List new files

Use the AWS SDK for JavaScript v3 client, `@aws-sdk/client-s3`. Each run lists the files under your folder and keeps the ones written since the last run, using each file's `LastModified` as the cursor.

```bash
GET /?list-type=2&prefix=conversations/ HTTP/1.1
Host: YOUR_BUCKET.s3.YOUR_REGION.amazonaws.com
```

- `ListObjectsV2` returns up to 1,000 keys per call. When there are more, pass the `NextContinuationToken` back as `continuation-token`. The SDK's `paginateListObjectsV2` does this for you ([SDK for JavaScript v3 S3 examples](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/javascript_s3_code_examples.html)).
- Each listed object has a `Key` and a `LastModified` date, the object's creation date ([Object](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html)).
- `GetObject` returns the file, and `Body.transformToString()` reads it as text ([SDK for JavaScript v3 S3 examples](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/javascript_s3_code_examples.html)).

The job lists the whole folder on every run. If it builds up many thousands of files, move processed files elsewhere, or write each day's files under a dated folder and point the job at it.

## 3. Map the columns

This guide assumes each file holds one row (CSV) or one line (JSONL) per message, with these fields. Rename them to match yours.

| Field | Holds |
|---|---|
| `message_id` | The message's own id |
| `conversation_id` | The conversation, ticket or chat the message belongs to |
| `customer_id` | Your internal id for the customer in that conversation, on every row, including agent and bot rows. Not an email address or phone number. |
| `sender_type` | Who wrote the message, for example `customer`, `agent` or `bot` |
| `body` | The message text |
| `channel` | Chat, email, voice and so on |
| `sent_at` | When the message was sent: ISO 8601 text such as `2026-01-02T09:30:00Z`, or Unix seconds or milliseconds. Text without a time zone is read as UTC. |

| Dimension Labs | File field | Notes |
|---|---|---|
| `text` | `body` | Skip rows where it is empty. Strip HTML if bodies are HTML. |
| `userId` | `customer_id` | If it is empty, use the `sessionId`. |
| `incoming` | `sender_type` | `true` only when the customer wrote it. Agent and bot rows are `false`. |
| `sessionId` | `conversation_id` | Replace any colons. |
| `conversationId` | `conversation_id` | Same value as `sessionId`. |
| `dimensionlabs_timestamp` | `sent_at` | Pass it through `toMillis`. |
| `platformJson` | `message_id`, `channel`, the file's key | Add any join keys you have, such as an account or store id. |

**Personal data.** Leave email, phone and name fields out of the files, or drop them in the function. If your only customer key is an email, send a consistent hash of it instead. Message text can still hold personal details: redact them in the function before you send. See [PII redaction](/data/pii-redaction).

## 4. Send it

Install the clients in your project first: `npm install @aws-sdk/client-s3 csv-parse`. CSV files are read with `csv-parse`: `columns: true` takes the field names from the first line, and `bom: true` strips a byte order mark if the file has one ([csv-parse sync API](https://csv.js.org/parse/api/sync/), [columns](https://csv.js.org/parse/options/columns/), [bom](https://csv.js.org/parse/options/bom/)).

```js
// amazon-s3.mjs. Credentials: the AWS SDK's default chain (for example the Lambda role).
// Environment: S3_BUCKET, S3_PREFIX (the folder, for example conversations/), DL_API_KEY.




const HOUR = 60 * 60 * 1000;
const SETTLE = 5 * 60 * 1000; // leave files from the last 5 minutes for the next run
const s3 = new S3Client({}); // the SDK retries throttling and server errors itself
const clean = (s) => String(s ?? '').replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim();
const safeId = (s) => String(s).replaceAll(':', '-');

function readRows(key, body) {
  if (/\.csv$/i.test(key)) return parse(body, { columns: true, bom: true });
  return body.split('\n').filter((line) => line.trim()).map((line) => JSON.parse(line)); // .jsonl
}

// Sends every file written after `since` (Unix ms). Returns the new `since`.
export async function syncAmazonS3({ since, historical = false }) {
  const Bucket = process.env.S3_BUCKET;
  const cutoff = Date.now() - SETTLE;
  const files = [];
  const pages = paginateListObjectsV2({ client: s3 }, { Bucket, Prefix: process.env.S3_PREFIX || '' });
  for await (const page of pages) {
    for (const obj of page.Contents ?? []) {
      const modified = obj.LastModified.getTime();
      if (modified > since && modified <= cutoff && /\.(csv|jsonl)$/i.test(obj.Key)) files.push({ key: obj.Key, modified });
    }
  }
  files.sort((a, b) => a.modified - b.modified);

  let newest = since;
  for (const { key, modified } of files) {
    const res = await s3.send(new GetObjectCommand({ Bucket, Key: key }));
    for (const row of readRows(key, await res.Body.transformToString())) {
      const text = clean(row.body);
      if (!text || !row.conversation_id || !row.sent_at) continue;
      const id = safeId(row.conversation_id);
      const ts = toMillis(row.sent_at);
      await sendToDimensionLabs({
        text,
        userId: row.customer_id ? safeId(row.customer_id) : id,
        incoming: row.sender_type === 'customer',
        sessionId: id,
        conversationId: id,
        dimensionlabs_timestamp: ts,
        platformJson: { message_id: String(row.message_id ?? ''), channel: row.channel ?? null, file: key },
      }, { historical: historical || ts < Date.now() - HOUR });
    }
    newest = Math.max(newest, modified);
  }
  return newest;
}
```

The job skips files written in the last five minutes and picks them up on the next run, so files that land at the same moment are never split between two runs. A run that fails part way sends some files again on the next run. That's fine: duplicates are removed automatically.

## 5. Run it

Save the code as `amazon-s3.mjs` in a project set up as in [Set up your project](/api/tracker#set-up-your-project), and run it with the [scheduled sync](/api/scheduled-sync) runner: `import { syncAmazonS3 as sync } from './amazon-s3.mjs';`. On AWS Lambda, add the policy from step 1 to the function's execution role.

- **History first:** `node run.mjs 2025-01-01` sends every file written since that date to `/trackhistorical`. Then start a [backfill](/dimensions/backfill) from the **Backfill** tab.
- **Then every 15 minutes:** the scheduled job sends new files. Messages less than an hour old go to `/track` and are enriched automatically every day. A file usually lands after its messages were sent: a nightly export, for example, holds messages that are hours old, so they go to `/trackhistorical`. Start a [backfill](/dimensions/backfill) after each batch arrives, or ask for files more often.
- **Re-runs are safe:** duplicates are removed automatically.
- **Large files:** the job reads each file into memory. Split very large exports into several files.

You can also [upload a file](/data/upload-a-file) directly. Run it through the [file checker](/data/check-a-file) first.

## Amazon S3 reference

- [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)
- [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)
- [Object](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html)
- [SDK for JavaScript v3: Amazon S3 examples](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/javascript_s3_code_examples.html)
- [csv-parse: sync API](https://csv.js.org/parse/api/sync/)

Checked against AWS's and csv-parse's documentation on 2026-10-26.