validation F1
Fixed stratified UCI rebuild holdout.
← All work
Applied machine learningRebuild tested
SpamShield ML
Cost-aware SMS spam classification where probability quality, threshold policy, artifact integrity, and deployment are evaluated as separate decisions.
Applied machine learning2026-08-29
SpamShield ML
0.9291validation F1
0.9788average precision
0.0347Brier score
Quick read
Project summary
Problem
A good classifier can still make the wrong operating decision. A false positive can hide a legitimate message, so threshold policy should reflect an explicit cost assumption rather than defaulting silently to 0.50.
My role
I rebuilt and evaluated the UCI classification workflow, compared models and thresholds, defined the artifact contract, and designed the FastAPI, Docker, and Streamlit delivery boundaries.
Result
Validation F1 0.9291 with average precision 0.9788 on the fixed stratified rebuild holdout.
Stack
Python · scikit-learn · FastAPI · Docker · Streamlit · TF-IDF
Status
Rebuild tested
average precision
Ranking evidence for the imbalanced target.
Brier score
Probability accuracy evidence.
simulated cost reduction
Compared with threshold 0.50 under the same 5:1 cost policy. Not financial savings.
Overview
SpamShield treats the model, predicted probability, operating threshold, and deployment artifact as separate engineering decisions.
Problem
A good classifier can still make the wrong operating decision. A false positive can hide a legitimate message, so threshold policy should reflect an explicit cost assumption rather than defaulting silently to 0.50.
Why it matters
Classification quality alone does not define a safe operating decision. Probability quality, threshold policy, artifact integrity, and deployment behavior all affect how the model should be used.
My role
I rebuilt and evaluated the UCI classification workflow, compared models and thresholds, defined the artifact contract, and designed the FastAPI, Docker, and Streamlit delivery boundaries.
Product
The rebuild packages a reproducible classifier behind testable API and Streamlit delivery surfaces while preserving the historical public demo as a separate reference.
- FastAPI health and prediction endpoints with request-size limits, validation, stable errors, and safe logging.
- Streamlit on the same artifact-backed runtime, with guards against intentionally logging or persisting submitted messages.
- Non-root Docker runtime with pinned base-image digest and pinned release dependencies.
Data
The rebuild uses an approved UCI retrieval path with bounded downloads and archive validation before training begins.
- ZIP signature and archive-member validation.
- Decompressed-size limits before the dataset is accepted.
- Fixed stratified holdout for the reviewed rebuild evidence.
Method
TF-IDF preprocessing feeds Naive Bayes and Logistic Regression comparison. Threshold selection is then evaluated separately under the recorded false-positive cost policy.
- Historical model record: Logistic Regression, F1 0.8550, threshold 0.40.
- Rebuild selected threshold: 0.30.
- Selected simulated decision cost: 28 units versus 44 units at threshold 0.50 on 1,115 validation messages.
Results
Rebuild numbers are reported as validation evidence, not production performance.
- Validation F1: 0.9291.
- 95% bootstrap percentile interval for F1: 0.8953 to 0.9580 using 2,000 stratified resamples.
- Average precision: 0.9788.
- Brier score: 0.0347.
- Simulated cost reduction: 36.36% versus threshold 0.50 under the same policy. This is not measured financial savings.
Validation
The evaluation is reproducible, but the same fixed holdout participates in model comparison and threshold selection, so selection optimism can remain.
- Bootstrap uncertainty describes sampling variability for the fixed holdout and does not remove selection bias.
- A separate untouched test set or nested validation is the next stronger design.
- Versioned model, vectorizer, metadata, threshold, training manifest, and SHA-256 integrity checks protect artifact compatibility.
Architecture
Training, artifacts, API delivery, container runtime, and Streamlit presentation share explicit contracts rather than relying on notebook state.
- Guarded UCI retrieval and dataset validation.
- Reproducible preprocessing and model training.
- Versioned integrity-checked artifact bundle.
- FastAPI and Streamlit surfaces consuming the same artifact contract.
- Non-root Docker runtime.
- Guarded UCI retrievalBounded download and archive checks
- Dataset validationSchema and decompressed-size controls
- Training and evaluationTF-IDF, model comparison, threshold analysis
- Artifact bundleVersioned files with SHA-256 integrity checks
- Delivery surfacesFastAPI, Docker, and Streamlit
Tradeoffs
The cost policy makes the operating decision interpretable, but it also means the preferred threshold depends on a stated business assumption rather than a universally optimal cutoff.
Limitations
The rebuild is not described as production performance. The historical public demo is a reference until the rebuilt artifact deployment and live privacy behavior are verified.
Live Product
A historical Streamlit demo URL is recorded, but direct availability could not be independently reverified in this environment. It is withheld from recruiter-facing links and is not evidence that the 2026 rebuilt artifacts are deployed.
Source
The rebuilt source and deployment path are not linked in this portfolio prototype until remote source access and the deployed artifact identity are reverified together.
Technical status: Rebuild tested, historical demo availability unverified
Repository status: Historical public demo is available, but rebuilt source deployment remains a separate verification milestone.
Next milestone
Produce the intended artifact bundle, deploy the rebuilt FastAPI path, verify live health and prediction behavior, reconnect Streamlit to the same artifact contract, and verify that submitted SMS text is absent from live logs and persistent storage.