Engine Precision Analysis

How ground truth was derived

There is no pre-labeled test set for this task. Instead, ground truth is derived by structurally analyzing what the human annotator actually did when transforming original → reworked — a behavioral labeling approach. The idea of inferring labels from before/after pairs is related to weak supervision (programmatic labeling) in NLP literature; what is specific here is the ordered 4-phase reconstruction of split/merge/modify/add/delete events from timed transcript data.

Dataset scope

14 datasets from transcription annotation jobs. All datasets include NSE (non-speech event) markers, overlap tags ([overlap]), timestamps, speaker labels, and metadata tags.

Class distribution

The dataset is heavily skewed toward MODIFIED, which makes up over half of all GT events. This is a natural property of transcript reworking — annotators correct rather than restructure the majority of segments.

The engine also shows a mild bias toward predicting MODIFIED: 3 GT-UNCHANGED rows are classified as MODIFIED (over-detecting changes), and several GT-ADDED rows are pulled into MODIFIED or SPLIT. This is partly by design — MODIFIED is the safest fallback when similarity is moderate — but it reduces recall for UNCHANGED and ADDED.

Per-category precision / recall / F1

Computed across 185 GT events from 14 datasets. TP = engine and GT agree. FP = engine predicted this category but GT says otherwise. FN = GT says this category but engine predicted something else.

Confusion matrix

Rows = GT labels. Columns = engine predictions. · denotes zero. UNMATCHED = GT row had no engine output at that anchor (absorbed as merge source / split child, or fell below all similarity gates).

The UNMATCHED column accounts for 18 events (9.7%): 8 GT-MODIFIED, 10 GT-ADDED. These are cases where the engine produces fewer output rows than GT expects — the most common cause is DS12, which has 1 original row expanding to 15 reworked rows, and the engine simply cannot produce enough events.

Root cause analysis

Breakdown of the 51 misclassifications by failure pattern and the engine parameter that controls each risk.

Similarity metric benchmarking

Computed against 6 datasets (DS6–DS11), 116 original rows, 1,754 total (original, reworked) pairs.

matchScore = w_time × timeSim + w_text × txtSim
timeSim  = time-proximity score (saturates beyond a max time delta)
txtSim   = Jaccard bigram similarity after NFKC + diacritics normalization
[blocked] if txtSim falls below a minimum text-floor threshold

Pairwise score coverage

Accepted match confidence distribution (64 matches)

Decision margin above acceptance threshold

Limitations

• •

  185 GT events is a small sample. Per-category numbers for UNCHANGED (6 events) and SPLIT (6 events) carry very wide confidence intervals — a single additional dataset could shift their F1 by 10+ points.

• •

  Ground truth is derived, not hand-labeled. The derivation algorithm itself has tunable parameters that influence which events appear in GT. Different parameter choices would produce different GT labels and different precision numbers.