Scheduled sync

Sending one message to the tracker is a single POST. Doing it reliably for a live system needs a small piece of infrastructure: a job that runs on a schedule, pulls what’s new, and sends it. You host and run this job yourself. This page gives you one runner that works with every integration guide.

How it fits together

Every universal API guide exports one function with the same shape:

export async function syncZendesk({ since, historical = false }) { /* ... */ return newSince; }

• since is Unix time in milliseconds. The function sends everything from that point on.
• historical: true sends everything to /trackhistorical. Without it, each message goes to /track, unless it is more than an hour old when the job sees it, in which case it goes to /trackhistorical.
• It returns the new since: store it, and pass it to the next run. That stored value is the cursor.

The runner loads the cursor, calls the guide’s function, and saves the cursor. If a run fails, the cursor isn’t saved, so the next run tries again from the same point. That is safe: duplicates are removed automatically, and the helper already retries the tracker’s 429 and 5xx responses.

Your project

Set up the project as in Set up your project, then add the guide’s code and the two runner files. For Zendesk:

dl-sync/
  package.json         "type": "module"
  dimensionlabs.mjs    the helper, downloaded
  zendesk.mjs          the code from the guide's "Send it" step
  run.mjs              runs it from your computer, cron or GitHub Actions
  handler.mjs          runs it on AWS Lambda

In both runner files, change the one import line to your guide, for example import { syncQualtrics as sync } from './qualtrics.mjs';. Each guide’s opening comment lists the environment variables it reads.

Load history first

run.mjs runs a guide from your computer or a server. With a date, it loads history; without one, it picks up from the saved cursor.

// run.mjs. History: node run.mjs 2026-01-01   Since the last run: node run.mjs
import { readFile, writeFile } from 'node:fs/promises';
import { syncZendesk as sync } from './zendesk.mjs';

const FILE = new URL('./cursor.json', import.meta.url);
const arg = process.argv[2];
let since, historical = false;
if (arg) {
  since = Date.parse(arg);
  if (Number.isNaN(since)) throw new Error(`Can't read the date "${arg}". Use YYYY-MM-DD.`);
  historical = true;
} else {
  try { ({ since } = JSON.parse(await readFile(FILE, 'utf8'))); }
  catch { since = Date.now() - 60 * 60 * 1000; } // first run: start an hour ago
}
const next = await sync({ since, historical });
await writeFile(FILE, JSON.stringify({ since: next }));
console.log(`Sent everything from ${new Date(since).toISOString()}. The next run starts at ${new Date(next).toISOString()}.`);

To load history:

1. Set the environment variables: export DL_API_KEY=... and the vendor’s variables from the guide.
2. Run node run.mjs 2026-01-01 with the oldest date you want. Everything goes to /trackhistorical. A long history can take hours: the code waits whenever the vendor or the tracker asks it to.
3. When it finishes, start a backfill from the Backfill tab. A backfill typically shows up within the hour, depending on how much data is being enriched.

Start the scheduled job before or after the history load: where the two overlap, the duplicates are removed. Some vendors cap how far back their API reaches (the guide says so); for anything older, upload a file instead.

Run it every 15 minutes on AWS Lambda

handler.mjs keeps the cursor in AWS Systems Manager Parameter Store:

// handler.mjs. Lambda handler: handler.handler
import { SSMClient, GetParameterCommand, PutParameterCommand } from '@aws-sdk/client-ssm';
import { syncZendesk as sync } from './zendesk.mjs';

const ssm = new SSMClient({});
const Name = process.env.CURSOR_PARAM; // for example /dimensionlabs/zendesk/since

export const handler = async () => {
  let since;
  try {
    since = Number((await ssm.send(new GetParameterCommand({ Name }))).Parameter.Value);
  } catch (err) {
    if (err.name !== 'ParameterNotFound') throw err;
    since = Date.now() - 60 * 60 * 1000; // first run: start an hour ago
  }
  const next = await sync({ since });
  await ssm.send(new PutParameterCommand({ Name, Value: String(next), Type: 'String', Overwrite: true }));
  return { since, next };
};

1. Package the code

In your project folder:

npm install @aws-sdk/client-ssm
zip -r function.zip . -x "cursor.json" -x ".git/*"

The zip holds package.json (with "type": "module"), dimensionlabs.mjs, your guide’s file, handler.mjs and node_modules.

2. Create the function

1. In the AWS console, open Lambda and select Create function.
2. Choose Author from scratch, enter a name such as dl-sync-zendesk, and pick a Node.js runtime (Node.js 22.x, or the newest offered). Select Create function.
3. On the Code tab, choose Upload from > .zip file and upload function.zip.
4. Under Runtime settings, select Edit and set Handler to handler.handler (the file handler.mjs, the export handler).

3. Configure it

On the Configuration tab:

1. General configuration > Edit. Raise Timeout from the 3-second default to 5 minutes (Lambda allows up to 15 minutes). Set Memory to 256 MB. Save.

2. Environment variables > Edit. Add DL_API_KEY, CURSOR_PARAM (for example /dimensionlabs/zendesk/since) and the vendor variables from the guide’s opening comment. Lambda encrypts environment variables at rest.

3. Permissions. Open the execution role (the link under Role name), choose Add permissions > Create inline policy, switch to JSON, and paste the following, with your region and account id. Name it and create it.

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": ["ssm:GetParameter", "ssm:PutParameter"],
       "Resource": "arn:aws:ssm:YOUR_REGION:YOUR_ACCOUNT_ID:parameter/dimensionlabs/*"
     }]
   }

   Guides that read from AWS (Amazon Connect’s S3 bucket, Redshift’s Data API) need their own permissions on this role too.

4. Concurrency > Edit. Choose Reserve concurrency and enter 1, so two runs never overlap.

5. Asynchronous invocation > Edit. Set Retry attempts to 0. The next scheduled run picks up anything a failed run missed.

On the Test tab, select Test to run it once. The result shows since and next; the log shows any error from the vendor or the tracker.

4. Schedule it

1. In the AWS console, open Amazon EventBridge. Under Scheduler in the left menu, choose Schedules, then Create schedule.
2. Enter a name. Under Schedule pattern, choose Recurring schedule, then Rate-based schedule, and enter 15 minutes. Set Flexible time window to Off. Select Next.
3. Under Target detail, choose AWS Lambda Invoke and pick your function. Select Next.
4. Under Permissions, choose Create new role for this schedule. EventBridge Scheduler creates a role that is allowed to invoke the function. Select Next, then Create schedule.

5. Check the first runs

1. In the function’s Monitor tab, select View CloudWatch logs and look at the first few runs for errors.
2. Ask the agent “How many messages arrived in this source today?” and compare with your source system.

If a run hits the timeout, it saves no cursor and the next run starts from the same point. That only happens when a backlog is too big for one run: catch up from your computer with node run.mjs and a date, then let the schedule carry on.

Run it with cron

On a Linux server, run.mjs keeps the cursor in cursor.json next to it. Put the environment variables in env.sh (export DL_API_KEY=... and so on), then add this line with crontab -e:

*/15 * * * * cd /opt/dl-sync && . ./env.sh && flock -n /tmp/dl-sync.lock node run.mjs >> sync.log 2>&1

flock -n skips a run while the last one is still going.

Run it with GitHub Actions

Put your project in a private repository with a package-lock.json (from npm install), and add the API keys under Settings > Secrets and variables > Actions. Don’t commit cursor.json: the workflow keeps it in the Actions cache.

.github/workflows/dl-sync.yml

name: Dimension Labs sync
on:
  schedule:
    - cron: '*/15 * * * *'
  workflow_dispatch:
concurrency: dl-sync
jobs:
  sync:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - uses: actions/cache/restore@v4
        with:
          path: cursor.json
          key: dl-cursor-${{ github.run_id }}
          restore-keys: dl-cursor-
      - run: node run.mjs
        env:
          DL_API_KEY: ${{ secrets.DL_API_KEY }}
          ZENDESK_SUBDOMAIN: ${{ secrets.ZENDESK_SUBDOMAIN }}
          ZENDESK_EMAIL: ${{ secrets.ZENDESK_EMAIL }}
          ZENDESK_API_TOKEN: ${{ secrets.ZENDESK_API_TOKEN }}
      - uses: actions/cache/save@v4
        with:
          path: cursor.json
          key: dl-cursor-${{ github.run_id }}

GitHub runs scheduled workflows on a best-effort basis, so runs can start late. A late run still picks up from the saved cursor. If the cache is ever lost, the next run starts an hour back; run node run.mjs with a date to fill a longer gap.

On Google Cloud, the same run.mjs can run as a Cloud Run job on a Cloud Scheduler trigger; see Google’s Execute jobs on a schedule. Keep the cursor somewhere that outlives the job, such as a Cloud Storage object.

Make it safe to re-run

• Use your system’s own conversation ids for sessionId and conversationId, so a re-sent message lands in the same conversation. The guides do this.
• Re-sending is safe: duplicates are removed automatically.
• Retries are built in: the helper retries the tracker, and each guide’s get() retries the vendor’s 429 and 5xx responses, honoring Retry-After. Any other error stops the run, and the next run tries again.
• Keep the clock in milliseconds and in UTC. toMillis reads dates without a time zone as UTC.
• Redact personal information in the function, before you send. See PII redaction.

Stuck? Email [email protected] with the source name, the error from the log, and one sample payload (with keys removed).