Clean ticket and email text

Recipe: strip HTML, quoted replies, sign-offs, odd spaces and placeholder text from message bodies before upload.

When you need it

Email and ticket exports carry more than the customer’s words: HTML tags, entities like  , the whole previous thread quoted under every reply, “Sent from my iPhone”, Windows line endings and placeholder bodies such as [No message body]. Enrichment reads every word you send, so a quoted reply gets analyzed again on every message that repeats it.

Input

Example input, emails.csv, one row per email. The first email quotes the previous one and has a mobile sign-off; the second is HTML; the third contains a non-breaking space between “Thanks” and “Sam”; the fourth is a placeholder.

thread_id,author_id,from_customer,body,sent_at
e-1,cus_51,true,"Hi team,

The beans arrived stale.

Sent from my iPhone

On Tue, Mar 3, 2026 at 9:00 AM Lumen Coffee <[email protected]> wrote:
> Your order has shipped!
> Track it here.",2026-03-04T09:15:00Z
e-1,agt_3,false,<p>So sorry!&nbsp;We&#39;ll send a fresh bag.</p><p>Sam<br>Lumen Coffee Support</p>,2026-03-04T10:02:00Z
e-1,cus_51,true,Thanks Sam!,2026-03-04T10:30:00Z
e-2,cus_90,true,[No message body],2026-03-05T08:00:00Z

Output

thread_id,author_id,from_customer,body,sent_at
e-1,cus_51,true,"Hi team, The beans arrived stale.",2026-03-04T09:15:00Z
e-1,agt_3,false,So sorry! We'll send a fresh bag. Sam Lumen Coffee Support,2026-03-04T10:02:00Z
e-1,cus_51,true,Thanks Sam!,2026-03-04T10:30:00Z

The columns keep their names here so the recipe can sit in front of any other step. Map thread_id, author_id, from_customer, body and sent_at to the required fields with the converter or the toolkit. When you do, the agent’s reply should carry the customer’s id (cus_51), not agt_3: the toolkit does that by default, and in the converter you can leave User id empty so every row in a thread gets the thread id.

Rules

Plain text only. Turn <br>, </p>, </div> and similar into line breaks, remove every other tag, and decode entities (&, ',  ). The universal API asks for the same.
Cut the quoted thread. Stop at the On ... wrote: line, and drop lines that start with >. Each email then holds only what was new in it.
Drop sign-offs that add nothing, such as “Sent from my iPhone”. Keep real signatures only if you need them; an agent’s name and title are usually noise.
Normalize spaces. Non-breaking spaces (U+00A0) and Windows line endings look like ordinary text but don’t match it.
One line per message. After cleaning, replace the remaining line breaks with spaces: line breaks inside a field split records.
Placeholders become empty, and empty rows go. A body of [No message body] is not something the customer wrote. Add your system’s own placeholder strings to the list.
Personal data: cleaning is also the place to redact. See PII redaction.

Script

Node 18 or later, with npm install papaparse.

// Clean email and ticket text before upload: HTML, entities, quoted replies, sign-offs, placeholders.
// Usage: node clean-text.mjs emails.csv emails-clean.csv
import fs from 'node:fs';
import Papa from 'papaparse';

const [input, output] = process.argv.slice(2);
const PLACEHOLDERS = new Set(['[no message body]', '[no text]', 'no response', '[no response]']);
const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', '#39': "'" };

function cleanMessage(raw) {
  let s = String(raw ?? '').replace(/\r\n?/g, '\n');                       // Windows line endings
  s = s.replace(/<br\s*\/?>|<\/(p|div|li|tr)>/gi, '\n').replace(/<[^>]+>/g, ''); // HTML to plain text
  s = s.replace(/&(amp|lt|gt|quot|apos|nbsp|#39);/g, (_, e) => ENTITIES[e]);
  s = s.replace(/\u00a0/g, ' ');                                            // non-breaking spaces
  const kept = [];
  for (const line of s.split('\n')) {
    if (/^\s*On .+ wrote:\s*$/.test(line)) break;                            // the quoted reply starts here
    if (/^\s*>/.test(line)) continue;                                         // quoted lines
    if (/^\s*Sent from my \w+/i.test(line)) continue;                         // mobile sign-off
    kept.push(line);
  }
  const text = kept.join(' ').replace(/\s+/g, ' ').trim();                   // one line per message
  return PLACEHOLDERS.has(text.toLowerCase()) ? '' : text;
}

const { data } = Papa.parse(fs.readFileSync(input, 'utf8'), { header: true, skipEmptyLines: 'greedy' });
const rows = data.map((r) => ({ ...r, body: cleanMessage(r.body) })).filter((r) => r.body); // drop what is left empty
fs.writeFileSync(output, Papa.unparse(rows, { newline: '\n' }) + '\n');
console.log(`${data.length} messages in, ${rows.length} out, ${data.length - rows.length} empty after cleaning`);

node clean-text.mjs emails.csv emails-clean.csv
# 4 messages in, 3 out, 1 empty after cleaning

In Python

Standard library only. It writes the same file.

# Clean email and ticket text before upload. Usage: python3 clean_text.py emails.csv emails-clean.csv
import csv, html, re, sys

PLACEHOLDERS = {"[no message body]", "[no text]", "no response", "[no response]"}

def clean_message(raw):
    s = (raw or "").replace("\r\n", "\n").replace("\r", "\n")
    s = re.sub(r"<br\s*/?>|</(p|div|li|tr)>", "\n", s, flags=re.I)
    s = re.sub(r"<[^>]+>", "", s)
    s = html.unescape(s).replace("\u00a0", " ")
    kept = []
    for line in s.split("\n"):
        if re.match(r"^\s*On .+ wrote:\s*$", line):
            break
        if re.match(r"^\s*>", line) or re.match(r"^\s*Sent from my \w+", line, re.I):
            continue
        kept.append(line)
    text = " ".join(" ".join(kept).split())
    return "" if text.lower() in PLACEHOLDERS else text

src, dst = sys.argv[1], sys.argv[2]
with open(src, newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    fields = reader.fieldnames
    rows = [{**r, "body": clean_message(r["body"])} for r in reader]
rows = [r for r in rows if r["body"]]
with open(dst, "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=fields, lineterminator="\n")
    w.writeheader()
    w.writerows(rows)
print(f"{len(rows)} messages kept")

Variations

Other reply headers: Outlook writes From: ... Sent: ... blocks instead of On ... wrote:. Add a pattern such as /^\s*From: .+/ to the break condition, and test it on a sample first so you don’t cut real text.
Other languages: reply headers are translated (Le ... a écrit :, Am ... schrieb ...). Add the ones your customers use.
The toolkit strips HTML with "clean": { "stripHtml": true }, and by default replaces non-breaking spaces and line breaks. It does not cut quoted replies; run this recipe first if you need that.