Static site generators help developers quickly roll out tool pages and resource guides, but templates can quietly introduce a problem: pages that seem distinct to people may look almost identical to search engines. This can drag down indexing and limit how easily users find your content. Google Search Central defines a canonical URL as the address Google picks as the main version among duplicate or near-duplicate pages. Instead of flagging duplicate content as spam, Google usually groups similar pages and shows just one in search results. For static sites, this means every page needs genuinely different content or a clear canonical tag to avoid being filtered out.
Why Duplicate Content Hurts Static Sites
When templates generate similar pages—like image compressors for different file sizes or translators for language pairs—the only real changes might be a few numbers or names. Instructions, FAQs, and privacy notes often stay the same. Google may crawl these pages but skip indexing them, treating them as duplicates. New sites can end up with many pages stuck in a "Crawled – currently not indexed" state in Google Search Console, meaning Google fetched the page but didn't add it to the index. This differs from "Discovered – currently not indexed," where the URL is known but not crawled, and from "Duplicate, Google chose different canonical than user," which means Google picked another URL as canonical. These issues need page-level checks and don't signal a manual penalty or a sitewide problem.
Measuring Content Similarity with Python
Guesswork won't reveal which pages are too similar, but Python can analyze your HTML output and measure duplication. Two main checks help here:
- Counting repeated sentences: Pull out the main content, split it into sentences, normalize them (lowercase, letters and digits only), and count how many pages each sentence of eight or more words appears on. For example:
def sentences(text):
out = set()
for s in re.split(r'(?<=[.!?])\s+', text):
w = re.findall(r"[a-z0-9']+", s.lower())
if len(w) >= 8:
out.add(" ".join(w))
return out- Measuring page overlap: Break each page into five-word shingles and use the Jaccard similarity to compare every pair. This shows how much phrasing is shared between pages:
def shingles(text, n=5):
w = re.findall(r"[a-z0-9']+", text.lower())
return {" ".join(w[i:i+n]) for i in range(len(w) - n + 1)}
def jaccard(a, b):
return len(a & b) / max(1, len(a | b))Running these checks can reveal patterns you might miss. In one audit, a privacy statement showed up on 21 pages, and some compressor pages shared over half their phrasing. Converter and translator pages also had heavy overlap. A Python audit can spot exact duplicate output with content hashes and near-duplicates with token-based similarity, letting a build fail or warn before deployment. Shingle-based Jaccard checks work well for template-heavy pages, since they measure shared word sequences instead of just full-document matches. Sentence frequency counts also flag which instructional, FAQ, or legal lines repeat across many pages. These measurements set engineering thresholds, since Google doesn't publish a fixed percentage for what counts as a duplicate.
Refactoring Templates for Unique Content
To cut duplication, start by removing boilerplate that repeats everywhere. If a privacy promise already appears in a key UI spot, an extra paragraph may be redundant. Then, move page-specific content—like unique FAQs or context—to a separate data file. For instance, a page_extras.py can store custom HTML blocks and FAQs for each page slug. This lets each page address its own use case, such as signature compression for a 20 KB image tool or file format quirks for a HEIC converter.
After these updates, repeated sentences should drop sharply. Ideally, no sentence appears on more than two pages, and the most similar pair of pages might fall from over 50% overlap to under 10%. Google Search Central says some duplicate content is normal and doesn't break spam policies, but static-site templates can amplify duplication when page-specific fields are too small to change the document’s overall value. Editorial and engineering teams should separate harmless URL duplication from thin pages that lack unique substance.
Automating Checks in Continuous Integration
To keep duplication from creeping back in, add the checker to your CI pipeline. For example, run a script on every pull request that fails the build if a post shares any eight-word sentence with another page or if two posts share more than 25% of their shingles. The script can print the repeated sentence and the affected file, making fixes straightforward before deployment. This step matters as new content gets added, whether written by hand or with AI tools. Automated checks help keep content quality and uniqueness high. For static sites, a CI check can also validate canonical tags, sitemap URLs, redirects, and internal-link consistency alongside text similarity, as outlined in Google's canonicalization documentation.
Impact on Indexing and Further Steps
Removing duplicate content isn't the only factor in search indexing—new sites also need time and inbound links—but it removes a clear reason for search engines to skip your pages. After making these changes, request re-validation in Search Console and watch for improved indexing. Google’s documented fixes depend on whether multiple URLs must stay live. If a duplicate URL can be retired, a permanent redirect is a strong signal that the destination should be canonical. If both URLs need to stay up, a rel="canonical" annotation can point to the main version. Internal links should go straight to the canonical URL, and sitemap entries should reinforce the same choice. Google recommends against using robots.txt for canonicalization, since blocking crawling doesn't tell Google which page should represent the set.
It's also necessary to make sure error pages and fallback templates return the right error status codes, not just a 200 OK with generic text. Google Search Central advises delivering <title> and <link rel="canonical"> tags directly in the initial server-side HTML, not relying on client-side JavaScript. Static generation usually makes this easier, but build pipelines still need tests to catch empty, fallback, or repeated output.
For another example of using Python for site maintenance, see this earlier breakdown on verifying PDF redactions with Python tools.
For developers generating pages from templates, running a content similarity check over your build output is a simple way to spot hidden duplication and help your site perform better in search. For more technical details on canonicalization and duplicate content, see Google's official guidance.
