The job
Bottom line
Use it as a local redaction layer, not as a privacy program.
OpenAI Privacy Filter is for teams that need to mask PII and secrets before text enters an LLM, vector index, training set, logging system, support export, review queue, or public data-sharing workflow. It looks genuinely useful: the official numbers are strong, the architecture matches the job, and the deployment paths cover browser, laptop, Apple Silicon, CPU, and GPU/server use. The hard boundary is that it is not anonymization, legal compliance, or DLP on its own.
Audience
Who this is for
| Audience | Why it fits | What to watch |
|---|---|---|
| AI platform teams | Redact user text before LLM calls, eval storage, traces, and RAG indexing. | Preserve enough context for downstream task quality. |
| Data and ML teams | Sanitize corpora before training, fine-tuning, labeling, or external review. | Benchmarks do not prove your domain is safe. |
| Security and infra teams | Detect secrets and personal identifiers in logs, tickets, dumps, and code-adjacent text. | Secret span F1 is materially lower than token recall; pair with dedicated secret scanners. |
| Support and ops teams | Mask PII in support tickets, exports, transcripts, and debugging bundles. | False positives remove needed context; false negatives leak customer data. |
| Local-first app builders | Run redaction on a laptop, browser/WebGPU app, Mac mini sidecar, or on-prem service before upload. | Extraction, OCR, review UI, and caching matter as much as the model call. |
| Compliance and privacy teams | Use as one technical control in a privacy-by-design program. | It is not a compliance certification or anonymization guarantee. |
Use-cases
Where it fits best
| Use-case | Fit | Reason |
|---|---|---|
| Pre-send LLM prompt filtering | Strong | Local redaction before content leaves the machine or tenant. |
| RAG and document ingestion | Strong | 128k context makes long documents less awkward than chunk-only approaches. |
| Support-ticket and CRM export scrubbing | Strong | The native taxonomy covers names, addresses, phones, emails, URLs, dates, account numbers, and secrets. |
| Log and trace scrubbing | Good | Useful for observability pipelines, especially with deterministic patterns for known internal formats. |
| Agent egress filtering | Good | Can sit before external tools, MCP calls, browser posting, ticket creation, Slack, or GitHub issue writes. |
| Codebase secret discovery | Good but incomplete | Credential recall is strong, but exact span scores are weaker; use GitHub secret scanning, gitleaks, trufflehog, or vendor DLP too. |
| Screenshot or image redaction | Conditional | Works only after OCR/text extraction; Hugging Face demos show the pattern but the model itself is text-only. |
| Fully automated legal, medical, or finance redaction | Risky | High-stakes domains need in-domain eval, fine-tuning, deterministic rules, audit, and human review. |
Hardware
Browser, Mac mini, laptop, CPU, and GPU fit
| Runtime | Will it run? | Practical expectation | Recommendation |
|---|---|---|---|
| Browser / WebGPU | Yes. Official examples use Transformers.js with WebGPU and q4; official ONNX q4/q4f16 artifacts exist. | The quantized files are still around 809-917 MB, so first load is heavy but private repeated use is credible. | Good for browser extensions, paste boxes, and local web apps where text should not leave the browser. |
| Mac mini M4 / Apple Silicon desktop | Yes by official laptop positioning plus MLX conversion support. | A base M4 Mac mini should run 4-bit inference; 24 GB+ unified memory is the safer floor for a local sidecar service. | Best local always-on target for vault, repo, support, or agent-log redaction. |
| MacBook Air / Pro | Yes. OpenAI positions the model for laptop use and MLX packages exist. | Good for local scripts, interactive review, and private document scrubbing; Pro/Max chips should be better for batch throughput. | Good developer-workstation target; benchmark before continuous background scanning. |
| CPU-only machine | Yes. The GitHub CLI exposes a CPU flag. | Fine for correctness checks, low-volume scripts, CI smoke tests, and fallback. | Do not choose CPU-only for high-throughput corpus cleanup unless latency does not matter. |
| NVIDIA/CUDA server | Yes through the Python/Transformers path and the repo's GPU-default CLI behavior. | Best fit for a central redaction service, bulk data cleanup, batching, and fine-tuning. | Use when throughput and organization-wide pipelines matter. |
| Mobile | Not a primary first-party target in this source pass. | Plausible with community/mobile runtimes, but not the main supported path today. | Treat as experimental unless adopting a maintained port and benchmark. |
Footprint
Model size is manageable, but not tiny
| Artifact | Size / shape | What it implies |
|---|---|---|
| Base safetensors | 2.8 GB plus a 27.9 MB tokenizer | Manageable on laptops and servers; too heavy for casual browser delivery. |
| ONNX q4 | 917 MB data file | Browser-capable but big; caching and load UX matter. |
| ONNX q4f16 | 809 MB data file | Better-looking browser delivery target if quality and runtime check out. |
| MLX 4-bit | 790 MB package | Best-looking Apple Silicon target for lightweight local inference. |
| MLX 8-bit | 1.49 GB package | Larger Apple Silicon target if 4-bit quality or boundaries are not acceptable. |
| Architecture | 1.5B total / 50M active, 8 layers, sparse MoE | Much smaller active compute than a normal dense 1.5B model, but not a regex replacement. |
Performance
No public chars/sec headline, but the architecture is right for throughput
I did not find first-party tokens/sec or chars/sec numbers. The useful signal is that Privacy Filter labels all tokens in one forward pass, has only 50M active parameters, uses banded local attention, and already has browser ONNX and Apple Silicon MLX formats.
| Workload | Fit | Why |
|---|---|---|
| Interactive paste redaction | Strong | Load once, redact many snippets locally. |
| Browser extension / private web app | Strong if model caching is acceptable | Privacy story is excellent, but the first model load is a real product constraint. |
| Vault or repo snapshot scrubber on a Mac mini | Strong | Text-only batch job, local data, enough memory, and easy sidecar deployment. |
| Continuous log firehose on CPU | Weak | It will run, but CPU is not the high-throughput target. |
| Enterprise batch sanitization | Strong on GPU | Use CUDA/server deployment and evaluate on real documents. |
| Fine-tuning | Server or GPU preferred | Fine-tuning is supported, but not a browser or tiny-edge-box workflow. |
Quality
It is good enough to pilot, not good enough to blindly trust
| Signal | What it says | Read |
|---|---|---|
| PII-Masking-300k | OpenAI reports token F1 0.960 baseline and 0.974 corrected; span F1 0.926 baseline and 0.942 corrected. | Strong headline, but corrected labels depend partly on OpenAI's adjudication process. |
| CredData secret detection | Corrected token recall is 0.965, token F1 is 0.844, and span F1 is 0.624. | Good as a layer, not a standalone secret scanner. |
| Domain fine-tuning | SPY benchmark moved from 0.545 F1 to 0.962 after training on 10 percent of the training set. | Suggests strong adaptation if you can label examples. |
| Multilingual evals | Strong on common PII-Masking-300k languages, lower but usable synthetic results across more languages. | Do not assume equal quality across scripts, countries, and naming conventions. |
| Independent GLiNER comparison | A small public repo reports boundary F1 wins over GLiNER large-v2.1 on 400 English and 200 multilingual samples. | Encouraging but small, synthetic, self-published, and not a production-domain corpus. |
| Stress tests | One-hop reasoning, odd formatting, line breaks, phonetic alphabet, and clue-free identifiers degrade. | Real pipelines need adversarial tests from local data. |
Alternatives
Nearest-neighbor ladder
| Option | Use when | Tradeoff |
|---|---|---|
| Regex-only filters | You only need emails, phone numbers, SSNs, credit cards, or known deterministic patterns. | Fast and explainable, weak on ambiguous unstructured text. |
| Microsoft Presidio | You need a mature recognizer framework, country-specific IDs, custom recognizers, and structured anonymization operators. | More rules/configuration work; context understanding depends on recognizers and NLP backends. |
| GLiNER-style PII models | You want flexible entity labels and smaller NER-oriented models. | Label flexibility can be useful, but Privacy Filter looks stronger in early boundary-style comparison. |
| OpenAI Privacy Filter | You need local, context-aware text PII masking with long context, browser/laptop/on-prem deployment, and fine-tuning. | Static label policy unless fine-tuned; not compliance/anonymization; stress cases remain. |
| Hosted DLP/privacy APIs | You need managed support, dashboards, policy packs, and enterprise procurement. | Sensitive text may leave your environment; cost and lock-in rise. |
Implementation
How I would deploy it
- Run it before outbound LLM calls, RAG indexing, log export, and human review shares.
- Pair it with deterministic recognizers for high-value IDs and known internal formats.
- Pair it with a dedicated secret scanner for credentials.
- Build a local eval set from 100-300 representative examples before trusting it.
- Tune operating points by workflow: high recall for external sharing, higher precision for internal review where context matters.
- For a Mac mini sidecar, start with MLX 4-bit, benchmark a representative file bundle, then test 8-bit if quality or boundary errors are unacceptable.
- For a browser tool, design around model caching and visible load state; the first model load is part of the UX.
- Keep human review for legal, medical, finance, HR, education, government, and public-release workflows.
Failure modes
Test these before production
| Failure mode | Why it matters |
|---|---|
| Private identifiers with no category clue | Addresses, dates, phone numbers, and secrets can be harder without surrounding context. |
| Long-range alias references | The model card says recall drops as the alias definition gets farther away. |
| Weird formatting | Line breaks, spacing, at/dot obfuscation, phonetic alphabet, and symbol substitutions can distort spans. |
| Domain-specific IDs | Customer IDs, medical record numbers, policy numbers, or internal identifiers may need fine-tuning. |
| Non-English and non-Latin text | OpenAI reports multilingual results, but also warns performance can drop by language, script, and naming convention. |
| Public vs private entities | The base policy preserves some context by design; your organization may want stricter or broader redaction. |
Receipts
Claim table
| Where | Claim | Confidence | Source | Caveat |
|---|---|---|---|---|
| Bottom line | OpenAI Privacy Filter is an open-weight text model for PII detection and masking, released under Apache 2.0. | High | OpenAI | License and repository metadata can change in future versions. |
| Footprint | The model is a bidirectional token classifier with constrained span decoding, 1.5B total parameters, 50M active parameters, and 128k context. | High | Hugging Face / OpenAI | Runtime performance still depends on implementation, precision, text length, and hardware. |
| Hardware | Browser deployment is credible because the official model card includes Transformers.js WebGPU usage and the Hugging Face repository includes ONNX q4/q4f16 artifacts. | High | Hugging Face / OpenAI | First-load size is large and browser WebGPU support must be tested per target browser. |
| Footprint | Official ONNX q4/q4f16 data artifacts are roughly 809-917 MB. | High | Hugging Face / OpenAI | Hugging Face file sizes may change with future revisions. |
| Hardware | Community MLX conversions provide Apple Silicon packages, including a 790 MB 4-bit package and a 1.49 GB 8-bit package. | Medium | Hugging Face / MLX Community | Community conversions need independent quality and performance validation before production use. |
| Hardware | A base M4 Mac mini should be able to run 4-bit inference, while 24 GB+ unified memory is the safer floor for an always-on local service. | Medium | Hugging Face / MLX Community | This is inferred from model/package size and Apple Silicon runtime support; no local Mac mini benchmark was run in this pass. |
| Hardware | The official CLI supports CPU execution through a CPU device flag. | High | GitHub / OpenAI | CPU execution is not the preferred path for high-throughput production. |
| Quality | OpenAI reports strong benchmark performance on PII-Masking-300k, including corrected token F1 of 0.974 and corrected span F1 of 0.942. | Medium | OpenAI | Corrected benchmark scores depend partly on model-assisted adjudication and should not replace local evaluation. |
| Quality | OpenAI reports high corrected token recall on CredData secret detection but much lower corrected span F1. | Medium | OpenAI | Use a dedicated secret scanner alongside Privacy Filter. |
| Bottom line | OpenAI says Privacy Filter is not anonymization, compliance, or a safety guarantee. | High | OpenAI | High-risk workflows need in-domain evaluation and human review. |
| Quality | A small independent comparison reported better boundary F1 for Privacy Filter than GLiNER large-v2.1 on sampled English and multilingual synthetic data. | Medium | Gaurav Vij / GitHub | Small sample, synthetic data, and self-published methodology; treat as directional only. |
Sources