Amazon S3

If your systems drop files of conversations into an S3 bucket (a nightly export, a vendor’s data dump), you can send them to Dimension Labs from there. A scheduled job on your side lists the files that are new since its last run, reads each one, and posts every row as a message to the universal API. Files can be CSV or JSONL (one JSON object per line).

What you need

• An AWS role that can create IAM roles and policies.
• An IAM role for the job, such as a Lambda execution role, or an IAM user’s access key for a job that runs outside AWS.
• A Dimension Labs API key for a new Universal API source. See Your API key.

1. Get access

The job needs two permissions. Listing a bucket needs s3:ListBucket (ListObjectsV2), and reading a file needs s3:GetObject (GetObject). Add this policy to the job’s role, with your bucket name and folder:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::YOUR_BUCKET" },
    { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::YOUR_BUCKET/conversations/*" }
  ]
}

If the files are encrypted with an AWS KMS key, the role also needs permission to use that key (GetObject). Outside AWS, keep the access key in a secrets manager, never in code you commit.

2. List new files

Use the AWS SDK for JavaScript v3 client, @aws-sdk/client-s3. Each run lists the files under your folder and keeps the ones written since the last run, using each file’s LastModified as the cursor.

GET /?list-type=2&prefix=conversations/ HTTP/1.1
Host: YOUR_BUCKET.s3.YOUR_REGION.amazonaws.com

• ListObjectsV2 returns up to 1,000 keys per call. When there are more, pass the NextContinuationToken back as continuation-token. The SDK’s paginateListObjectsV2 does this for you (SDK for JavaScript v3 S3 examples).
• Each listed object has a Key and a LastModified date, the object’s creation date (Object).
• GetObject returns the file, and Body.transformToString() reads it as text (SDK for JavaScript v3 S3 examples).

The job lists the whole folder on every run. If it builds up many thousands of files, move processed files elsewhere, or write each day’s files under a dated folder and point the job at it.

3. Map the columns

This guide assumes each file holds one row (CSV) or one line (JSONL) per message, with these fields. Rename them to match yours.

Field	Holds
message_id	The message’s own id
conversation_id	The conversation, ticket or chat the message belongs to
customer_id	Your internal id for the customer in that conversation, on every row, including agent and bot rows. Not an email address or phone number.
sender_type	Who wrote the message, for example customer, agent or bot
body	The message text
channel	Chat, email, voice and so on
sent_at	When the message was sent: ISO 8601 text such as 2026-01-02T09:30:00Z, or Unix seconds or milliseconds. Text without a time zone is read as UTC.

Dimension Labs	File field	Notes
text	body	Skip rows where it is empty. Strip HTML if bodies are HTML.
userId	customer_id	If it is empty, use the sessionId.
incoming	sender_type	true only when the customer wrote it. Agent and bot rows are false.
sessionId	conversation_id	Replace any colons.
conversationId	conversation_id	Same value as sessionId.
dimensionlabs_timestamp	sent_at	Pass it through toMillis.
platformJson	message_id, channel, the file’s key	Add any join keys you have, such as an account or store id.

Personal data. Leave email, phone and name fields out of the files, or drop them in the function. If your only customer key is an email, send a consistent hash of it instead. Message text can still hold personal details: redact them in the function before you send. See PII redaction.

4. Send it

Install the clients in your project first: npm install @aws-sdk/client-s3 csv-parse. CSV files are read with csv-parse: columns: true takes the field names from the first line, and bom: true strips a byte order mark if the file has one (csv-parse sync API, columns, bom).

// amazon-s3.mjs. Credentials: the AWS SDK's default chain (for example the Lambda role).
// Environment: S3_BUCKET, S3_PREFIX (the folder, for example conversations/), DL_API_KEY.
import { S3Client, GetObjectCommand, paginateListObjectsV2 } from '@aws-sdk/client-s3';
import { parse } from 'csv-parse/sync';
import { sendToDimensionLabs, toMillis } from './dimensionlabs.mjs';

const HOUR = 60 * 60 * 1000;
const SETTLE = 5 * 60 * 1000; // leave files from the last 5 minutes for the next run
const s3 = new S3Client({}); // the SDK retries throttling and server errors itself
const clean = (s) => String(s ?? '').replace(/<[^>]+>/g, ' ').replace(/\s+/g, ' ').trim();
const safeId = (s) => String(s).replaceAll(':', '-');

function readRows(key, body) {
  if (/\.csv$/i.test(key)) return parse(body, { columns: true, bom: true });
  return body.split('\n').filter((line) => line.trim()).map((line) => JSON.parse(line)); // .jsonl
}

// Sends every file written after `since` (Unix ms). Returns the new `since`.
export async function syncAmazonS3({ since, historical = false }) {
  const Bucket = process.env.S3_BUCKET;
  const cutoff = Date.now() - SETTLE;
  const files = [];
  const pages = paginateListObjectsV2({ client: s3 }, { Bucket, Prefix: process.env.S3_PREFIX || '' });
  for await (const page of pages) {
    for (const obj of page.Contents ?? []) {
      const modified = obj.LastModified.getTime();
      if (modified > since && modified <= cutoff && /\.(csv|jsonl)$/i.test(obj.Key)) files.push({ key: obj.Key, modified });
    }
  }
  files.sort((a, b) => a.modified - b.modified);

  let newest = since;
  for (const { key, modified } of files) {
    const res = await s3.send(new GetObjectCommand({ Bucket, Key: key }));
    for (const row of readRows(key, await res.Body.transformToString())) {
      const text = clean(row.body);
      if (!text || !row.conversation_id || !row.sent_at) continue;
      const id = safeId(row.conversation_id);
      const ts = toMillis(row.sent_at);
      await sendToDimensionLabs({
        text,
        userId: row.customer_id ? safeId(row.customer_id) : id,
        incoming: row.sender_type === 'customer',
        sessionId: id,
        conversationId: id,
        dimensionlabs_timestamp: ts,
        platformJson: { message_id: String(row.message_id ?? ''), channel: row.channel ?? null, file: key },
      }, { historical: historical || ts < Date.now() - HOUR });
    }
    newest = Math.max(newest, modified);
  }
  return newest;
}

The job skips files written in the last five minutes and picks them up on the next run, so files that land at the same moment are never split between two runs. A run that fails part way sends some files again on the next run. That’s fine: duplicates are removed automatically.

5. Run it

Save the code as amazon-s3.mjs in a project set up as in Set up your project, and run it with the scheduled sync runner: import { syncAmazonS3 as sync } from './amazon-s3.mjs';. On AWS Lambda, add the policy from step 1 to the function’s execution role.

• History first: node run.mjs 2025-01-01 sends every file written since that date to /trackhistorical. Then start a backfill from the Backfill tab.
• Then every 15 minutes: the scheduled job sends new files. Messages less than an hour old go to /track and are enriched automatically every day. A file usually lands after its messages were sent: a nightly export, for example, holds messages that are hours old, so they go to /trackhistorical. Start a backfill after each batch arrives, or ask for files more often.
• Re-runs are safe: duplicates are removed automatically.
• Large files: the job reads each file into memory. Split very large exports into several files.

You can also upload a file directly. Run it through the file checker first.

Amazon S3 reference

• ListObjectsV2
• GetObject
• Object
• SDK for JavaScript v3: Amazon S3 examples
• csv-parse: sync API

Checked against AWS’s and csv-parse’s documentation on 2026-10-26.