Before encrypting, masking, or deleting personal data, organizations need to pinpoint exactly where that information lives. Many teams struggle to identify which tables or files hold sensitive details like card numbers or national IDs. Relying on memory or incomplete documentation falls short, especially since GDPR Article 30 demands a thorough record of processing activities and data categories. Under Article 30 of the General Data Protection Regulation, controllers and processors must keep records that cover processing purposes, categories of data subjects, types of personal data, recipients, transfers, retention periods, and security measures. This requirement puts data discovery at the heart of governance, since organizations cannot document processing they have not actually found in their databases, files, or applications. The European Data Protection Board (EDPB) has made clear that Article 30 is reviewed alongside GDPR definitions, processing principles, lawfulness requirements, and consent rules, which raises the stakes for building accurate and complete data inventories.
Why Regex Alone Falls Short for PII Discovery
Many teams reach for regular expressions to scan databases and exported files. Regex can spot patterns that look like credit card numbers or social security numbers, but it often floods teams with false positives. Order numbers, timestamps, ZIP+4 codes, and product SKUs can all match similar patterns, making it tough to separate real personal data from unrelated values. The EDPB has not set a universal technical rule to use regex, sampling, or automated scanners for compliance, but organizations are expected to use effective safeguards and document their processing activities in a way that meets regulatory expectations.
Designing a Smarter Python PII Scanner
A better method combines pattern matching with validation logic. The Python scanner described here works with both SQLite databases and folders of CSV exports. It uses reservoir sampling to efficiently sample column values, which keeps memory usage predictable and results reproducible even with large datasets.
After finding candidate values with regex, the scanner runs specific validation checks. For example, it uses the Luhn algorithm and network prefixes to confirm valid card numbers, ISO 7064 mod-97 for IBANs, SSA rules for SSNs, and the phonenumbers library for phone numbers. This validation step weeds out most false positives and highlights columns that actually contain sensitive data. The EDPB notes in its guidance on anonymization and pseudonymization that finding a value that looks like an identifier does not automatically mean it is still personal data, has been irreversibly anonymized, or is pseudonymized and potentially linkable with extra information. A practical reporting system should log the detected field, validation evidence, surrounding metadata, and whether a re-identification key exists.
Column Scoring and Reporting
To sharpen accuracy, the scanner also looks at column names as a signal. It calculates a confidence score for each column based on the share of validated values and how relevant the column name is. The final data inventory is saved in both JSON and Markdown formats, with classification labels and masked examples to avoid exposing raw PII in reports. The EDPB points out that a column-level confidence score can help prioritize review, but it cannot by itself decide the legal status of processing or guarantee anonymization. Inventories and discovery reports should keep collection dates, source locations, retention status, and evidence for each classification, as seen in regulatory decisions like the Belgian Data Protection Authority's handling of online directory complaints.
Comparison with Microsoft Presidio
For benchmarking, the scanner's results are compared with Microsoft Presidio using the same synthetic test data generated by Faker. Presidio offers context-sensitive detection and works with spaCy for natural language processing. The comparison shows differences in speed, accuracy, and how well each tool validates structured data. While Presidio is powerful, the custom Python scanner gives more control over validation logic and reporting, making it a good fit for production use with scheduled scans and change tracking.
Implementation Details and Extensibility
The scanner is built with Python 3.13 and uses libraries such as Faker 40.44.0 for test data, phonenumbers 9.0.42 for phone validation, and pytest 9.2.0 for testing. For the Presidio comparison, presidio-analyzer 2.2.366 and spaCy 3.8.18 with en_core_web_sm 3.8.0 are used. All test data is synthetic and reproducible. The code is set up to let developers add new detectors for other national IDs or data types, and it encourages human review for borderline cases. The EDPB's published resources also include guidance for personal-data processing in complex, distributed systems such as connected vehicles and mobility-related applications, where inventories must account for telemetry, location data, user accounts, device identifiers, and service-provider access. A scanner limited to SQLite files and CSV exports would only cover part of an organization's processing landscape, and regulatory documentation still requires organizations to understand processing activities and data categories across all relevant systems, including those from applications, devices, or external providers. For further reading, see official EDPB decision materials.
Practical Steps for Building Your Own Scanner
- Connect to your SQLite database or load CSV files from a folder.
- Sample values from each column using reservoir sampling.
- Apply regex patterns to find candidate PII values.
- Validate candidates with appropriate algorithms or libraries.
- Score columns based on validation results and column names.
- Generate a data inventory report in JSON and Markdown, masking sensitive values.
- Optionally, compare results with tools like Microsoft Presidio for additional context.
Developers looking to strengthen their Python skills can benefit from exploring advanced topics like data sampling and validation logic. Those interested in testing their knowledge of related tools may find the advanced Git quiz useful for broadening their understanding of developer workflows.
