# Clean ticket and email text

> Recipe: strip HTML, quoted replies, sign-offs, odd spaces and placeholder text from message bodies before upload.

## When you need it

Email and ticket exports carry more than the customer's words: HTML tags, entities like ` `, the whole previous thread quoted under every reply, "Sent from my iPhone", Windows line endings and placeholder bodies such as `[No message body]`. Enrichment reads every word you send, so a quoted reply gets analyzed again on every message that repeats it.

## Input

Example input, `emails.csv`, one row per email. The first email quotes the previous one and has a mobile sign-off; the second is HTML; the third contains a non-breaking space between "Thanks" and "Sam"; the fourth is a placeholder.

```csv title="emails.csv"
thread_id,author_id,from_customer,body,sent_at
e-1,cus_51,true,"Hi team,

The beans arrived stale.

Sent from my iPhone

On Tue, Mar 3, 2026 at 9:00 AM Lumen Coffee <help@lumen.example> wrote:
> Your order has shipped!
> Track it here.",2026-03-04T09:15:00Z
e-1,agt_3,false,<p>So sorry!&nbsp;We&#39;ll send a fresh bag.</p><p>Sam<br>Lumen Coffee Support</p>,2026-03-04T10:02:00Z
e-1,cus_51,true,Thanks Sam!,2026-03-04T10:30:00Z
e-2,cus_90,true,[No message body],2026-03-05T08:00:00Z
```

## Output

```csv title="emails-clean.csv"
thread_id,author_id,from_customer,body,sent_at
e-1,cus_51,true,"Hi team, The beans arrived stale.",2026-03-04T09:15:00Z
e-1,agt_3,false,So sorry! We'll send a fresh bag. Sam Lumen Coffee Support,2026-03-04T10:02:00Z
e-1,cus_51,true,Thanks Sam!,2026-03-04T10:30:00Z
```

The columns keep their names here so the recipe can sit in front of any other step. Map `thread_id`, `author_id`, `from_customer`, `body` and `sent_at` to the required fields with the [converter](/data/convert-a-file) or the [toolkit](/transform/toolkit). When you do, the agent's reply should carry the customer's id (`cus_51`), not `agt_3`: the toolkit does that by default, and in the converter you can leave **User id** empty so every row in a thread gets the thread id.

## Rules

- **Plain text only.** Turn `<br>`, `</p>`, `</div>` and similar into line breaks, remove every other tag, and decode entities (`&amp;`, `&#39;`, `&nbsp;`). The [universal API](/api/tracker#rules-that-trip-up-integrations) asks for the same.
- **Cut the quoted thread.** Stop at the `On ... wrote:` line, and drop lines that start with `>`. Each email then holds only what was new in it.
- **Drop sign-offs that add nothing**, such as "Sent from my iPhone". Keep real signatures only if you need them; an agent's name and title are usually noise.
- **Normalize spaces.** Non-breaking spaces (U+00A0) and Windows line endings look like ordinary text but don't match it.
- **One line per message.** After cleaning, replace the remaining line breaks with spaces: [line breaks inside a field](/data/conversation-format#rules-that-break-uploads) split records.
- **Placeholders become empty, and empty rows go.** A body of `[No message body]` is not something the customer wrote. Add your system's own placeholder strings to the list.
- **Personal data:** cleaning is also the place to redact. See [PII redaction](/data/pii-redaction).

## Script

Node 18 or later, with `npm install papaparse`.

```js title="clean-text.mjs"
// Clean email and ticket text before upload: HTML, entities, quoted replies, sign-offs, placeholders.
// Usage: node clean-text.mjs emails.csv emails-clean.csv



const [input, output] = process.argv.slice(2);
const PLACEHOLDERS = new Set(['[no message body]', '[no text]', 'no response', '[no response]']);
const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', '#39': "'" };

function cleanMessage(raw) {
  let s = String(raw ?? '').replace(/\r\n?/g, '\n');                       // Windows line endings
  s = s.replace(/<br\s*\/?>|<\/(p|div|li|tr)>/gi, '\n').replace(/<[^>]+>/g, ''); // HTML to plain text
  s = s.replace(/&(amp|lt|gt|quot|apos|nbsp|#39);/g, (_, e) => ENTITIES[e]);
  s = s.replace(/\u00a0/g, ' ');                                            // non-breaking spaces
  const kept = [];
  for (const line of s.split('\n')) {
    if (/^\s*On .+ wrote:\s*$/.test(line)) break;                            // the quoted reply starts here
    if (/^\s*>/.test(line)) continue;                                         // quoted lines
    if (/^\s*Sent from my \w+/i.test(line)) continue;                         // mobile sign-off
    kept.push(line);
  }
  const text = kept.join(' ').replace(/\s+/g, ' ').trim();                   // one line per message
  return PLACEHOLDERS.has(text.toLowerCase()) ? '' : text;
}

const { data } = Papa.parse(fs.readFileSync(input, 'utf8'), { header: true, skipEmptyLines: 'greedy' });
const rows = data.map((r) => ({ ...r, body: cleanMessage(r.body) })).filter((r) => r.body); // drop what is left empty
fs.writeFileSync(output, Papa.unparse(rows, { newline: '\n' }) + '\n');
console.log(`${data.length} messages in, ${rows.length} out, ${data.length - rows.length} empty after cleaning`);
```

```bash
node clean-text.mjs emails.csv emails-clean.csv
# 4 messages in, 3 out, 1 empty after cleaning
```

## In Python

Standard library only. It writes the same file.

```python title="clean_text.py"
# Clean email and ticket text before upload. Usage: python3 clean_text.py emails.csv emails-clean.csv


PLACEHOLDERS = {"[no message body]", "[no text]", "no response", "[no response]"}

def clean_message(raw):
    s = (raw or "").replace("\r\n", "\n").replace("\r", "\n")
    s = re.sub(r"<br\s*/?>|</(p|div|li|tr)>", "\n", s, flags=re.I)
    s = re.sub(r"<[^>]+>", "", s)
    s = html.unescape(s).replace("\u00a0", " ")
    kept = []
    for line in s.split("\n"):
        if re.match(r"^\s*On .+ wrote:\s*$", line):
            break
        if re.match(r"^\s*>", line) or re.match(r"^\s*Sent from my \w+", line, re.I):
            continue
        kept.append(line)
    text = " ".join(" ".join(kept).split())
    return "" if text.lower() in PLACEHOLDERS else text

src, dst = sys.argv[1], sys.argv[2]
with open(src, newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    fields = reader.fieldnames
    rows = [{**r, "body": clean_message(r["body"])} for r in reader]
rows = [r for r in rows if r["body"]]
with open(dst, "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
    w.writeheader()
    w.writerows(rows)
print(f"{len(rows)} messages kept")
```

## Variations

- **Other reply headers**: Outlook writes `From: ... Sent: ...` blocks instead of `On ... wrote:`. Add a pattern such as `/^\s*From: .+/` to the break condition, and test it on a sample first so you don't cut real text.
- **Other languages**: reply headers are translated (`Le ... a écrit :`, `Am ... schrieb ...`). Add the ones your customers use.
- **The [toolkit](/transform/toolkit)** strips HTML with `"clean": { "stripHtml": true }`, and by default replaces non-breaking spaces and line breaks. It does not cut quoted replies; run this recipe first if you need that.