PDF redaction often gives a false sense of security when only the visible layer is checked. A black rectangle on a PDF page might look like it covers sensitive data, but the original text can still sit underneath, accessible to anyone with the right software. For anyone handling private documents, confirming that confidential data is truly removed, rather than just masked, is a critical step.
Why Visual Redaction Can Fail
Simply drawing a filled rectangle over text in a PDF does not delete the underlying content. The text stays in the file's content stream, and most PDF readers just display the box on top. Automated text extractors like PyMuPDF, pypdf, and pdfminer.six can still pull out the hidden information. Even advanced redaction tools may leave traces if the process is incomplete or if the file is saved incrementally, which can keep earlier versions inside the PDF structure.
Legal and regulatory standards demand more than visual redaction. In Trade West Construction, Inc. v. U.S. Army Corps of Engineers, No. 24-3291, decided September 30, 2026, the U.S. District Court for the District of Columbia made clear that redaction is a legal and procedural requirement, not just a formatting step. The court told the agency to "conduct a more careful redaction of the requested documents, and to release any non-exempt and reasonably segregable material contained in the records," making it clear that agencies must actually protect confidential information under FOIA Exemption 4 (U.S. Department of Justice, Office of Information Policy).
Understanding PDF Content Streams and Metadata
PDFs keep text as encoded streams, often compressed and shown in hexadecimal. For instance, an account number like 4471 2093 5568 could appear as a hex string in the content stream. Searching the PDF file directly with tools like grep will not find the number unless the stream is decompressed and the encoding is handled. Sensitive information can also linger in document metadata, such as the author field, even after visible redaction.
Testing Redaction with Python Libraries
To check that redacted information is really gone, it helps to use more than one method. The following Python libraries are widely used for this:
- PyMuPDF: Extracts text and checks annotations and metadata.
- pypdf: Reads PDF content and pulls text from pages.
- pdfminer.six: Offers detailed text extraction, even from complex PDFs.
Each tool can uncover text that remains after redaction. For a thorough check, scan every stream in the PDF file, decompressing them to search for both plain and encoded versions of sensitive data. This step helps catch information that text extractors might miss, especially in non-standard or orphaned streams.
Common Redaction Pitfalls
Several situations can leave redacted data exposed:
- Drawing a box over text without deleting the underlying content.
- Adding redaction marks but not applying them, so the text stays in place.
- Saving the PDF incrementally, which adds new content but keeps earlier versions in the file.
- Leaving sensitive data in metadata fields like author or title.
For example, using saveIncr() in PyMuPDF or similar incremental save features in desktop PDF editors can create multiple file revisions. The original content may still be present in earlier revisions, even if PDF readers no longer display it.
Redacted PDFs are regularly published in official federal court records. In United States v. Merchant, No. 24-362, the U.S. District Court for the Eastern District of New York attached redacted versions of its orders dated January 14, 2026, and January 22, 2026, to the docket on April 1, 2026, with file sizes of about 281.3 KB and 392.7 KB. Judge Eric R. Komitee ordered these attachments, showing the ongoing use of redacted PDFs in federal court administration (U.S. Government Publishing Office).
In another case, DeLeon v. Phillips et al., No. 24-234, in the U.S. District Court for the Eastern District of California, a redacted document was filed as part of a civil action. This confirms that redacted PDFs are distributed through official federal-record systems, where document construction and metadata can affect confidentiality.
What Proper Redaction Looks Like
When redaction is done correctly and the file is fully sanitized, the sensitive text is removed from all content streams and metadata. In the content stream, the space once used by the text is replaced with a positional adjustment, so the layout stays the same but the data is gone. Black rectangles may still be drawn for appearance, but nothing remains underneath. Metadata should also be cleared to prevent leaks through document properties.
Federal court rulings have set the standards for lawful redaction. In Trade West Construction, the court stated that "even where an FOIA Exemption applies, an agency may only withhold exempt information if the agency reasonably foresees that disclosure would harm an interest protected by an exemption." The decision also explained that Exemption 3 applies only to material "specifically exempted from disclosure by statute," and required agencies to separate protected content from information that can be released.
Automating Redaction Checks with Python
One way to automate these checks is to use a Python script that combines text extraction, annotation review, metadata checks, and stream scanning. The following function shows this approach:
import re, sys, zlib
import pymupdf
from pypdf import PdfReader
from pdfminer.high_level import extract_text
def problems(path, secrets):
raw = open(path, 'rb').read()
doc = pymupdf.open(path)
flat = lambda t: re.sub(r'\s+', ' ', t.replace('\xa0', ' '))
texts = {
'pymupdf': flat(''.join(p.get_text() for p in doc)),
'pypdf': flat(''.join(p.extract_text() for p in PdfReader(path).pages)),
'pdfminer': flat(extract_text(path)),
}
out = []
for s in secrets:
out += [f'{name}: text contains {s!r}' for name, t in texts.items() if s in t]
for p in doc:
for a in p.annots():
if s in (a.info.get('content') or ''):
out.append(f'annotation on page {p.number + 1} contains {s!r}')
out += [f'metadata {k} = {v!r}' for k, v in doc.metadata.items() if v and s in v]
if raw.count(b'%%EOF') > 1:
out.append(f'{raw.count(b"%%EOF")} revisions: earlier versions are still in the file')
if any(p.first_annot and p.first_annot.type[1] == 'Redact' for p in doc):
out.append('redaction annotations were never applied')
for m in re.finditer(rb'stream\r?\n(.*?)\r?\nendstream', raw, re.S):
try:
body = zlib.decompress(m.group(1))
except zlib.error:
body = m.group(1)
for s in secrets:
if s.encode() in body or s.encode().hex().encode() in body.lower():
out.append(f'stream at byte {m.start()} still holds {s!r}')
return out
This script checks for the presence of specified secrets in extracted text, annotations, metadata, and decompressed streams. It also detects multiple file revisions and un-applied redaction annotations. If any issues are found, the script lists them; otherwise, it reports the file as clean.
Limitations and Best Practices
This method only finds the exact strings provided, so changes in formatting or encoding may need extra checks. Secrets inside images will not be found unless optical character recognition (OCR) is used first. For PDFs using CID fonts, some text may be missed by stream scanning but still picked up by text extractors. Always clear metadata and attachments during redaction, and use tools that fully rewrite the PDF instead of saving incrementally.
Based on the original technical analysis, the osFoundry PDF Editor's Apply and Sanitize features created a file with no remaining sensitive data in content streams or metadata. The editor works locally, so documents are processed on the user's machine and do not leave the device.
