Hello. What would you like to examine today?
01 Local ML, fully inspectable
Train the model.
Inspect the decision.
ELIZA Lab is a complete intent-classification pipeline in Rust: validated data, deterministic training, versioned weights, measured uncertainty and a hard abstention path.
- 525 reviewed prompts
- 105 paraphrase families
- No prompt telemetry
02 Local inference
Make one prediction.
See what moved it.
Use a harmless, fictional prompt. The inspector exposes the predicted class, probability, margin and strongest positive feature contributions. Input stays in this tab.
03 Reproducible pipeline
Every stage is code.
Every artifact can be checked.
Reject bad rows.
The parser enforces equal family support, unique model features and bounded near-duplicate review.
Separate the roles.
Whole groups enter train, development, calibration or ID-test. Every class remains represented.
Build the feature space.
Word uni- and bigrams combine with character 3–5 grams, TF-IDF weighting and L2 normalization.
Fit real weights.
Only the 315 training rows fit TF-IDF and multinomial logistic-regression parameters.
Calibrate, then stop.
Calibration fits temperature. Development plus OOD-development select the abstention policy.
01Grouped TSV525 prompts / 105 families
02SplitPlan315 / 70 / 70 / 70
03Fit + scaleweights / temperature
04Freeze policydevelopment + OOD-dev
05Open testsID + OOD + contrast
04 Verified model report
Choose, calibrate and test
on different evidence.
The v3 experiment gives training, development, probability calibration and ID-test their own group-disjoint partitions. OOD threshold data, OOD-test and the paired contrast test stay separate too.
REPORT LOADING
- ID-test accuracy
- —available after bundle verification
- ID-test macro F1
- —available after bundle verification
- Learned vs unigram
- —macro-F1 difference on the same ID-test
- Decision coverage
- —accepted rows after the frozen gate
- ID-test NLL
- —measured only after the policy was frozen
- OOD-test AUROC
- —domain-cluster interval after verification
- OOD-test FPR @ 95 TPR
- —lower is better
- OOD-test coverage
- —reported across 36 untouched prompts
- Contrast pair accuracy
- —both meaning-changing variants correct
WHAT THE CONTRACT PROVES — AND WHAT IT DOES NOT
No test row chooses a weight, temperature or threshold.
Temperature scaling sees only calibration rows. Confidence and probability-margin thresholds see only development plus OOD-development. The ID, OOD and paired contrast tests are opened afterward. Family and domain-cluster intervals preserve dependence instead of pretending every sentence is an independent experiment.
The report also reconstructs majority and unigram baselines and separates semantic, capability and noise OOD behaviour. Fourteen paired contrasts probe lexical shortcuts without entering any tuning API. All prompts remain synthetic and English-only. This is an auditable experiment, not a production-language claim.
Inspect the complete v3 metrics05 Pre-test selection stability
The less flattering result
belongs here too.
Eleven outer folds hold out one whole family per class. Five inner folds then compare all nine candidates without access to calibration, ID-test, OOD or contrast data. Every selection-pool row gets one genuinely out-of-family prediction.
AUDIT LOADING
- Out-of-fold accuracy
- —385 rows · before abstention
- Out-of-fold macro F1
- —seven balanced labels
- Accuracy interval
- —1,000 family-cluster resamples
- Most stable choice
- —outer folds selecting the same candidate
06 Post-training robustness
Stable under format.
Measured under noise.
A fixed metamorphic suite now probes the verified model after every training and policy choice is frozen. It checks exact preprocessing invariants separately from controlled typo stress, then reports only aggregate drift.
07 A hard boundary
A classifier is not a clinician.
ELIZA Lab is educational software. It does not understand a person, assess risk, provide treatment or replace qualified care.
A small list of explicit safety phrases runs before learned inference, stops the simulation and points toward immediate human help. It is a boundary, not crisis detection.
Read the safety and privacy model08 Refactored with intent
The useful question survived.
The unsafe product did not.
The original 2023 Telegram prototype stored conversations and implied therapeutic use. This edition removes Telegram, MySQL, OpenAI and the PHP admin panel. What remains is a reproducible Rust ML pipeline, verified open-set inference, visible abstention and a transparent browser lab.