Google Scaled Content Abuse: A Site Audit Checklist

Content Quality, Search Engines, SEO, SEO Content | 0 comments

Google defines scaled content abuse as producing many pages mainly to manipulate search rankings, with little or no added value for users. The deciding factors are purpose and user value, not whether pages use AI-generated content or automation.

Table of Contents

For your site audit, the priority is usefulness, not a target publishing limit. Review page groups, find repeated weaknesses, and assign a clear action to each affected URL. Start with the policy, then compare it with what your pages deliver.

Key Takeaways

Scaled content abuse involves producing many pages mainly to manipulate search rankings, with little or no added value for visitors. Google sets no page-count threshold, publishing limit, or acceptable ratio of AI-generated content.
AI tools, automation, and shared templates aren't violations by themselves. Assess each page's purpose, accuracy, and usefulness to visitors.
Audit by page group, review representative pages and exceptions, and document the distinct value each page adds.
Choose an appropriate action for each group: keep, improve, consolidate, noindex, or remove. Update internal links and sitemaps after consolidating or removing pages.
Check Search Console for a manual action notice before responding. Fix the process behind affected pages, then submit a reconsideration request for a manual action or monitor affected URLs after algorithmic losses.

What Scaled Content Abuse Means

Google's spam policies focus on why publishers create content at scale and the value visitors receive under this spam policy. Content can violate the policy when it's mass-produced mainly to influence rankings rather than help people.

Google sets no page-count threshold, daily publishing limit, or acceptable ratio of AI-generated content. A large catalog can provide useful information. A smaller collection of repetitive, unhelpful pages can warrant review.

How the policy changed

In March 2024, Google introduced the policy on March 5 alongside expired domain abuse and site reputation abuse. It replaced the narrower rule against spammy automatically generated content.

Google's March 2024 announcement projected a 40% reduction in low-quality, unoriginal content after its changes. After the rollout ended on April 19, Google reported a 45% reduction.

Those figures describe changes across Google Search, not the share of websites penalized.

What counts as a warning sign

Repeated copy, empty templates, and unsupported claims can indicate low-quality content, but these patterns warrant investigation, not an automatic conclusion.

A short page can answer a question completely. A long page can repeat information without helping anyone. Assess what the page accomplishes before deciding what to change.

The Five Patterns Google Lists

Google gives five examples of scaled content abuse. Use them to identify content groups that need closer inspection:

Using generative AI to produce pages isn't a violation by itself. The concern is producing them at scale mainly to manipulate rankings without meaningful value for readers.
Scraped feeds or search results become new pages without meaningful added value.
Material from different webpages is stitched together without useful additions.
Multiple websites conceal the scaled nature of the content.
Pages contain search keywords but make little sense to readers.

During an audit, compare these patterns with your publishing workflow. Examine where information comes from, how pages are generated, and what editors add.

For syndicated content, permission to republish doesn't establish search value. Review whether the published version gives visitors something useful beyond the original.

When Programmatic Pages Provide Real Value

Programmatic SEO uses templates and data to create pages efficiently. Templates aren't a violation by themselves. Their value depends on the information each page provides.

Useful data can justify separate pages

Property listings can offer different prices, photographs, availability, and property details. Sports results can provide distinct scores and match records. Product pages can show specifications, stock information, and purchasing options.

Those differences help visitors make different decisions. A shared layout doesn't erase that value.

For local businesses, our guidance on valuable programmatic local pages focuses on accurate service information and meaningful local details. A city name alone isn't enough to justify another page.

Empty combinations deserve review

Review pages created for every service, location, or filter combination. Some combinations have no available products, no real service coverage, or no distinct answer.

Look for identical paragraphs with substituted keywords, generic images, mismatched phone numbers, and invented testimonials.

Ask what changes for the visitor on this particular page. If the answer is only the heading, consolidation or withholding publication may be more useful than rewriting synonyms.

Manual Action or Algorithmic Traffic Loss?

Google uses automated systems and human review to enforce its spam policies. A manual action and an algorithmic traffic loss require different responses, so start by determining whether Google issued a notice.

A manual action has a notice

Open Search Console's Manual Actions report. A notice identifies the issue and whether it affects a section or the whole site.

After correcting the problem, submit a reconsideration request through that report. Review the full affected content group, not only the example URLs Google provides.

Quality raters are separate from manual-action reviewers. Google's quality-rater guidelines describe Low and Lowest quality ratings, but individual rater assessments don't directly determine rankings.

Algorithmic losses have no notice

An empty report means there is no reported manual action. It doesn't establish that your content meets every quality expectation.

A core update, a spam update, or traffic loss alone doesn't prove a policy violation or a Google penalty.

Compare affected directories, publication dates, queries, and indexing changes. Also check server errors, accidental noindex tags, rendering problems, and changes in search demand.

According to ppc.land's enforcement summary, Ian Nuttall's sample of 49,345 sites included 837 deindexed sites, about 1.7%. This is an observation, not evidence that scaled content abuse caused every removal.

A Numbered Site Audit Checklist

We recommend auditing by page group before making individual edits. This helps identify the template or workflow behind repeated problems.

1. Build a complete URL inventory

Combine a site crawl, CMS export, XML sitemaps, and Search Console data. Screaming Frog can collect status codes, titles, headings, canonical tags, and index directives.

Record each URL's template, purpose, publication date, traffic, conversions, backlinks, and content source. Include orphan pages and generated URLs outside your main navigation.

Our small business content audit checklist helps prioritize important service pages, location pages, and content tied to customer inquiries.

2. Review representative pages and exceptions

Sample each location, service, template version, and publishing batch. Include strong performers, weak performers, and recently created pages.

Compare rendered HTML with the database record. A populated field doesn't help if visitors or Google can't see it.

Inspect visible copy, titles, headings, structured data, contact details, internal links, and robots directives. Repeated templates or rising "Discovered, currently not indexed" counts call for investigation, but don't establish a policy finding.

3. Document the value each page adds

Assign one main visitor need to each page. Then record the information that helps someone complete that task.

Google's people-first content guidance emphasizes original information and useful contributions beyond other sources.

Look for empty inventory, copied descriptions, unsupported claims, and pages competing to answer the same question. Record concrete evidence instead of labeling pages as thin content or low-quality content based only on low traffic or indexing status.

Finish with an action, an owner, and a completion date for each affected group.

Choose the Right Remediation for Each Page

Don't delete pages just because they receive few clicks. Consider traffic, backlinks, customer use, and business relevance when choosing an action.

Use these five categories to organize your repair plan.

Action	Appropriate situation	Practical change
Keep	The page provides distinct, accurate value.	Maintain its information and links.
Improve	A useful topic has weak execution.	Add missing facts, evidence, or task guidance.
Consolidate	Several pages serve the same need.	Merge useful content and redirect redundant URLs.
noindex	Visitors need the page, but Search doesn't.	Add a noindex directive and remove it from sitemaps.
Remove	The page has no useful purpose.	Delete it and return 404 or 410.

When improving a page, add information that helps visitors make a decision. More words, replacement adjectives, or another generic introduction won't fix the underlying weakness.

Use canonical tags for duplicate versions that need to stay accessible. A canonical is a preference signal, not a dependable way to exclude abusive content.

Google's policy calls for excluding violating content from Search. If you keep it, use noindex; otherwise, remove it. Our guide to using noindex without blocking crawlers explains the setup.

Google must be able to crawl a page to read that directive. Blocking it first in robots.txt can prevent Google from discovering it.

After consolidating or removing pages, update internal links, breadcrumbs, and sitemaps. Redirect a URL only when a relevant replacement exists, rather than sending every deleted URL to the homepage.

Add a Pre-Publication Quality Gate

An audit helps repair existing problems. A publishing gate helps keep them from returning.

Before approving another generated page group, require clear answers to these questions:

What visitor need does each page address?
Which accurate information distinguishes it from related pages?
Can visitors complete their task without an unnecessary intermediate page?
Who reviewed the facts and approved publication?
What happens when required data is missing or outdated?

Hold incomplete pages back instead of publishing empty templates. Missing service coverage, unavailable inventory, or unsupported claims should stop publication.

Google's guidance on generative AI content says the method of creation alone doesn't determine whether content violates its policies. Assess each page's accuracy, usefulness, and purpose.

We recommend approving the rendered page, not only the draft. Metadata and structured data can contain inaccurate claims even when the visible copy looks correct.

Recovery Starts With the Publishing Process

For a manual action, clean up the affected content before requesting review. Describe completed work, not promises.

Fix the source of the problem

Pause the workflow responsible for the affected pages. Correct template logic, data requirements, editorial review, or moderation rules before restarting it.

Apply the repair across the affected group. Editing a few examples leaves the wider issue unresolved.

Save affected URL lists and document improvements, exclusions, removals, and validation results.

Submit a clear reconsideration request

Explain what caused the violation, what you found, what you changed, and how you tested the repair. Describe the safeguards that prevent recurrence.

Google doesn't publish a guaranteed recovery timeline. Removing a manual action also doesn't guarantee previous rankings.

For algorithmic losses, there is no reconsideration request route. After repairs, monitor useful traffic and check whether affected URLs are recrawled, reindexed, or remain deindexed.

Related Risks: Expired Domains and Advertising

Keep adjacent policies separate during your audit. Site reputation abuse, expired domain abuse, and scaled content abuse describe different behaviors, though they can overlap.

Site reputation abuse concerns third-party content published with little or no first-party oversight or control, primarily to manipulate rankings.

Expired domain abuse involves buying an expired domain and repurposing it for low-value content to manipulate rankings. Google's examples include affiliate content on a former government agency website and casino content on a former elementary school website.

Google Publisher Policies prohibit Google-served ads, including AdSense, AdMob, and Ad Manager, on screens that violate Search spam policies.

Include affected advertising destinations in the cleanup inventory so repairs address more than organic traffic.

Frequently Asked Questions

What counts as scaled content abuse?

It involves producing many pages mainly to manipulate search rankings, with little or no added value for users. Repeated copy, empty templates, and unsupported claims are warning signs to investigate, not automatic proof of a violation.

Is AI-generated or programmatic content automatically a violation?

No. Google says the method of creation alone doesn't determine whether content violates its policies, and templates aren't a violation by themselves. Assess each page's purpose and the useful information it provides.

How should I audit pages that may have little value?

Start by grouping pages by template, purpose, and publishing workflow, then inspect representative pages and exceptions. Record the value each page adds and assign an action, an owner, and a completion date to affected groups.

How can I tell a manual action from an algorithmic traffic loss?

A manual action appears as a notice in Search Console's Manual Actions report. An empty report means there is no reported manual action, but it doesn't establish that every page meets Google's quality expectations.

What should I do after receiving a manual action?

Correct the affected content and the workflow that produced it, then submit a reconsideration request through the Manual Actions report. Describe what caused the issue, what you changed, and how you tested the repairs.

Keep Useful Pages, Fix Repeatable Problems

A scaled content abuse audit should produce a repair plan for each affected page group. Keep useful pages, improve worthwhile content, and consolidate or exclude pages that don't justify separate search visibility.

Start with the workflow creating the weakest pages. Correcting that process protects your improvements and gives future content a stronger foundation.

Submit a Comment Cancel reply

Latest Articles

Google Scaled Content Abuse: A Site Audit ChecklistGoogle defines scaled content abuse as producing many pages mainly to manipulate search rankings, with little or no added value for users. The deciding factors are purpose and user value, not whether pages use AI-generated content or automation. Table of Contents Toggle Key TakeawaysWhat Scaled Content Abuse MeansHow the policy changedWhat counts as a warning signThe Five Patterns Google ListsWhen Programmatic Pages Provide Real ValueUseful data can justify separate pagesEmpty combinations deserve reviewManual Action or Algorithmic Traffic Loss?A manual action has a noticeAlgorithmic losses have no noticeA Numbered Site Audit Checklist1. Build a complete URL inventory2. Review representative pages and exceptions3. Document the value each page addsChoose the Right Remediation for Each PageAdd a Pre-Publication Quality GateRecovery Starts With the Publishing ProcessFix the source of the problemSubmit a clear reconsideration requestRelated Risks: Expired Domains and AdvertisingFrequently Asked QuestionsWhat counts as scaled content abuse?Is AI-generated or programmatic content automatically a violation?How should I audit pages that may have little value?How can I tell a manual action from an algorithmic traffic loss?What should I do after receiving a manual action?Keep Useful Pages, Fix Repeatable Problems For your site audit, the priority is usefulness, not a target publishing limit. Review page groups, find repeated weaknesses, and assign a clear action to each affected URL. Start with the policy, then compare it with what your pages deliver. Key Takeaways Scaled content abuse involves producing many pages mainly to manipulate search rankings, with little or no added value for visitors. Google sets no page-count threshold, publishing limit, or acceptable ratio of AI-generated content. AI tools, automation, and shared templates aren't violations by themselves. Assess each page's purpose, accuracy, and usefulness to visitors. Audit by page group, review representative pages and exceptions, and document the distinct value each page adds. Choose an appropriate action for each group: keep, improve, consolidate, noindex, or remove. Update internal links and sitemaps after consolidating or removing pages. Check Search Console for a manual action notice before responding. Fix the process behind affected pages, then submit a reconsideration request for a manual action or monitor affected URLs after algorithmic losses. What Scaled Content Abuse Means Google's spam policies focus on why publishers create content at scale and the value visitors receive under this spam policy. Content can violate the policy when it's mass-produced mainly to influence rankings rather than help people. Google sets no page-count threshold, daily publishing limit, or acceptable ratio of AI-generated content. A large catalog can provide useful information. A smaller collection of repetitive, unhelpful pages can warrant review. How the policy changed In March 2024, Google introduced the policy on March 5 alongside expired domain abuse and site reputation abuse. It replaced the narrower rule against spammy automatically generated content. Google's March 2024 announcement projected a 40% reduction in low-quality, unoriginal content after its changes. After the rollout ended on April 19, Google reported a 45% reduction. Those figures describe changes across Google Search, not the share of websites penalized. What counts as a warning sign Repeated copy, empty templates, and unsupported claims can indicate low-quality content, but these patterns warrant investigation, not an automatic conclusion. A short page can answer a question completely. A long page can repeat information without helping anyone. Assess what the page accomplishes before deciding what to change. The Five Patterns Google Lists Google gives five examples of scaled content abuse. Use them to identify content groups that need closer inspection: Using generative AI to produce pages isn't a violation by itself. The concern is producing them at scale mainly to manipulate rankings without meaningful value for readers. Scraped feeds or search results become new pages without meaningful added value. Material from different webpages is stitched together without useful additions. Multiple websites conceal the scaled nature of the content. Pages contain search keywords but make little sense to readers. During an audit, compare these patterns with your publishing workflow. Examine where information comes from, how pages are generated, and what editors add. For syndicated content, permission to republish doesn't establish search value. Review whether the published version gives visitors something useful beyond the original. When Programmatic Pages Provide Real Value Programmatic SEO uses templates and data to create pages efficiently. Templates aren't a violation by themselves. Their value depends on the information each page provides. Useful data can justify separate pages Property listings can offer different prices, photographs, availability, and property details. Sports results can provide distinct scores and match records. Product pages can show specifications, stock information, and purchasing options. Those differences help visitors make different decisions. A shared layout doesn't erase that value. For local businesses, our guidance on valuable programmatic local pages focuses on accurate service information and meaningful local details. A city name alone isn't enough to justify another page. Empty combinations deserve review Review pages created for every service, location, or filter combination. Some combinations have no available products, no real service coverage, or no distinct answer. Look for identical paragraphs with substituted keywords, generic images, mismatched phone numbers, and invented testimonials. Ask what changes for the visitor on this particular page. If the answer is only the heading, consolidation or withholding publication may be more useful than rewriting synonyms. Manual Action or Algorithmic Traffic Loss? Google uses automated systems and human review to enforce its spam policies. A manual action and an algorithmic traffic loss require different responses, so start by determining whether Google issued a notice. A manual action has a notice Open Search Console's Manual Actions report. A notice identifies the issue and whether it affects a section or the whole site. After correcting the problem, submit a reconsideration request through that report. Review the full affected content group, not only the example URLs Google provides. Quality raters are separate from manual-action reviewers. Google's quality-rater guidelines describe Low and Lowest quality ratings, but individual rater assessments don't directly determine rankings. Algorithmic losses have no notice An empty report means there is no reported manual action. It doesn't establish that your content meets every quality expectation. A core update, a spam update, or traffic loss alone doesn't prove a policy violation or a Google penalty. Compare affected directories, publication dates, queries, and indexing changes. Also check server errors, accidental noindex tags, rendering problems, and changes in search demand. According to ppc.land's enforcement summary, Ian Nuttall's sample of 49,345 sites included 837 deindexed sites, about 1.7%. This is an observation, not evidence that scaled content abuse caused every removal. A Numbered Site Audit Checklist We recommend auditing by page group before making individual edits. This helps identify the template or workflow behind repeated problems. 1. Build a complete URL inventory Combine a site crawl, CMS export, XML sitemaps, and Search Console data. Screaming Frog can collect status codes, titles, headings, canonical tags, and index directives. Record each URL's template, purpose, publication date, traffic, conversions, backlinks, and content source. Include orphan pages and generated URLs outside your main navigation. Our small business content audit checklist helps prioritize important service pages, location pages, and content tied to customer inquiries. 2. Review representative pages and exceptions Sample each location, service, template version, and publishing batch. Include strong performers, weak performers, and recently created pages. Compare rendered HTML with the database record. A populated field doesn't help if visitors or Google can't see it. Inspect visible copy, titles, headings, structured data, contact details, internal links, and robots directives. Repeated templates or rising "Discovered, currently not indexed" counts call for investigation, but don't establish a policy finding. 3. Document the value each page adds Assign one main visitor need to each page. Then record the information that helps someone complete that task. Google's people-first content guidance emphasizes original information and useful contributions beyond other sources. Look for empty inventory, copied descriptions, unsupported claims, and pages competing to answer the same question. Record concrete evidence instead of labeling pages as thin content or low-quality content based only on low traffic or indexing status. Finish with an action, an owner, and a completion date for each affected group. Choose the Right Remediation for Each Page Don't delete pages just because they receive few clicks. Consider traffic, backlinks, customer use, and business relevance when choosing an action. Use these five categories to organize your repair plan. ActionAppropriate situationPractical changeKeepThe page provides distinct, accurate value.Maintain its information and links.ImproveA useful topic has weak execution.Add missing facts, evidence, or task guidance.ConsolidateSeveral pages serve the same need.Merge useful content and redirect redundant URLs.noindexVisitors need the page, but Search doesn't.Add a noindex directive and remove it from sitemaps.RemoveThe page has no useful purpose.Delete it and return 404 or 410. When improving a page, add information that helps visitors make a decision. More words, replacement adjectives, or another generic introduction won't fix the underlying weakness. Use canonical tags for duplicate versions that need to stay accessible. A canonical is a preference signal, not a dependable way to exclude abusive content. Google's policy calls for excluding violating content from Search. If you keep it, use noindex; otherwise, remove it. Our guide to using noindex without blocking crawlers explains the setup. Google must be able to crawl a page to read that directive. Blocking it first in robots.txt can prevent Google from discovering it. After consolidating or removing pages, update internal links, breadcrumbs, and sitemaps. Redirect a URL only when a relevant replacement exists, rather than sending every deleted URL to the homepage. Add a Pre-Publication Quality Gate An audit helps repair existing problems. A publishing gate helps keep them from returning. Before approving another generated page group, require clear answers to these questions: What visitor need does each page address? Which accurate information distinguishes it from related pages? Can visitors complete their task without an unnecessary intermediate page? Who reviewed the facts and approved publication? What happens when required data is missing or outdated? Hold incomplete pages back instead of publishing empty templates. Missing service coverage, unavailable inventory, or unsupported claims should stop publication. Google's guidance on generative AI content says the method of creation alone doesn't determine whether content violates its policies. Assess each page's accuracy, usefulness, and purpose. We recommend approving the rendered page, not only the draft. Metadata and structured data can contain inaccurate claims even when the visible copy looks correct. Recovery Starts With the Publishing Process For a manual action, clean up the affected content before requesting review. Describe completed work, not promises. Fix the source of the problem Pause the workflow responsible for the affected pages. Correct template logic, data requirements, editorial review, or moderation rules before restarting it. Apply the repair across the affected group. Editing a few examples leaves the wider issue unresolved. Save affected URL lists and document improvements, exclusions, removals, and validation results. Submit a clear reconsideration request Explain what caused the violation, what you found, what you changed, and how you tested the repair. Describe the safeguards that prevent recurrence. Google doesn't publish a guaranteed recovery timeline. Removing a manual action also doesn't guarantee previous rankings. For algorithmic losses, there is no reconsideration request route. After repairs, monitor useful traffic and check whether affected URLs are recrawled, reindexed, or remain deindexed. Related Risks: Expired Domains and Advertising Keep adjacent policies separate during your audit. Site reputation abuse, expired domain abuse, and scaled content abuse describe different behaviors, though they can overlap. Site reputation abuse concerns third-party content published with little or no first-party oversight or control, primarily to manipulate rankings. Expired domain abuse involves buying an expired domain and repurposing it for low-value content to manipulate rankings. Google's examples include affiliate content on a former government agency website and casino content on a former elementary school website. Google Publisher Policies prohibit Google-served ads, including AdSense, AdMob, and Ad Manager, on screens that violate Search spam policies. Include affected advertising destinations in the cleanup inventory so repairs address more than organic traffic. Frequently Asked Questions What counts as scaled content abuse? It involves producing many pages mainly to manipulate search rankings, with little or no added value for users. Repeated copy, empty templates, and unsupported claims are warning signs to investigate, not automatic proof of a violation. Is AI-generated or programmatic content automatically a violation? No. Google says the method of creation alone doesn't determine whether content violates its policies, and templates aren't a violation by themselves. Assess each page's purpose and the useful information it provides. How should I audit pages that may have little value? Start by grouping pages by template, purpose, and publishing workflow, then inspect representative pages and exceptions. Record the value each page adds and assign an action, an owner, and a completion date to affected groups. How can I tell a manual action from an algorithmic traffic loss? A manual action appears as a notice in Search Console's Manual Actions report. An empty report means there is no reported manual action, but it doesn't establish that every page meets Google's quality expectations. What should I do after receiving a manual action? Correct the affected content and the workflow that produced it, then submit a reconsideration request through the Manual Actions report. Describe what caused the issue, what you changed, and how you tested the repairs. Keep Useful Pages, Fix Repeatable Problems A scaled content abuse audit should produce a repair plan for each affected page group. Keep useful pages, improve worthwhile content, and consolidate or exclude pages that don't justify separate search visibility. Start with the workflow creating the weakest pages. Correcting that process protects your improvements and gives future content a stronger foundation. [...]
Read more...
Product Page SEO Beyond Manufacturer DescriptionsEffective product page SEO combines useful buying information with clear search listings, crawlable URLs, and accurate structured data. Rewriting manufacturer descriptions helps when you add decision-making value, such as compatibility details, practical benefits, and answers shoppers can't find elsewhere. Table of Contents Toggle Key Takeaways1. Write product descriptions that answer buying questionsConnect features to customer needsKeep specifications clear and useful2. Give search listings clear product detailsStart title tags with the product identityWrite a useful search snippet3. Add FAQs and reviews that reduce uncertaintyAnswer purchase-related questions directlyMake customer reviews useful and credible4. Make product schema match the page5. Choose one URL strategy for variants6. Improve product images and mobile usability7. Handle unavailable products without misleading shoppers8. Audit product page SEO across your catalogFix access and indexing firstMeasure changes against a stable baselineMake Useful Product Information the Priority Copied supplier text often leaves important questions unanswered, while adding paragraphs solely for search engines makes shopping harder. We recommend starting with those missing answers, then improving the technical elements that help people find your products. Key Takeaways Rewrite manufacturer descriptions when you can add decision-making value. Answer buying questions, explain benefits and limitations, and keep specifications accurate. Give priority products clear title tags and meta descriptions. Use genuine FAQs and reviews to address concerns, and treat search snippets and enhanced results as possibilities, not guarantees. Keep Product structured data aligned with visible catalog details. Choose a clear indexing and canonical strategy for variants and duplicate product URLs. Improve product images and mobile usability, and keep availability, offers, and discontinued-product details accurate. Audit crawling and indexing before making broad copy changes. Prioritize products with meaningful sales potential and measure results against a stable baseline. 1. Write product descriptions that answer buying questions Manufacturer specifications are a starting point for a useful product description that helps customers decide. Keep accurate measurements, model numbers, and safety information, then add explanations that help shoppers choose. Duplicate supplier copy isn't automatically a Google penalty. The practical problem is that your page gives shoppers little reason to choose your store over another seller. Connect features to customer needs First, identify what matters to the buyer. Customer-service questions, return reasons, and product reviews reveal concerns that generic descriptions often miss. For example, the Logitech MX Master 3S has an adjustable sensor reaching 8,000 DPI. That specification identifies a capability. Explaining adjustable pointer sensitivity helps shoppers understand how it supports their preferred working style. Describe suitability and limitations alongside benefits. Compatibility, fit, installation requirements, and included accessories can influence purchases more than broad claims like "premium quality." We recommend placing each benefit next to the feature that supports it. Keep claims grounded in documented capabilities, rather than implying results the product can't guarantee. Keep specifications clear and useful Separate scannable specifications from explanatory copy. Materials, dimensions, capacity, care requirements, and what's included shouldn't disappear inside a long paragraph. Keep model numbers and measurements unchanged. Distinctive copy doesn't require rewriting established facts into unusual language. There's no required description length. A simple replacement part may need little explanation. A technical product may need specifications, compatibility guidance, and several answers. Long buying guides belong on separate pages that link to relevant products. Keep product pages focused on the purchase decision. 2. Give search listings clear product details Your title tag and meta description introduce the product before someone visits your store. Each product page needs accurate, distinctive search details that are easy to understand. Start title tags with the product identity Aim for about 30 to 60 characters, roughly 285 to 575 pixels, with important product terms near the beginning. These are working targets, not Google's fixed limits. A useful pattern is brand, product name or model, then one meaningful attribute. For example: "Logitech MX Master 3S Wireless Mouse | 8,000 DPI." The visible H1 can use the simpler product name. It doesn't need to match the title tag exactly. Google often rewrites search titles. Crimson estimates this happens around 60% of the time, but that isn't a Google-published rate. Use consistent product wording in the title, H1, and visible copy instead of stuffing several keyword variations into one field. Our title tag optimization guide covers how to balance clarity with available space. Write a useful search snippet Aim for a meta description of approximately 70 to 155 characters, roughly 430 to 920 pixels. Include an accurate benefit and a natural call to action. For the same mouse: "Explore the Logitech MX Master 3S wireless mouse with an adjustable 8,000 DPI sensor. View specifications and choose your setup." Write unique descriptions for priority products. Avoid promises about free shipping, stock, or returns unless your store provides them. Google may display different page text instead. Treat the meta description as a useful introduction, not a guaranteed search snippet or ranking boost. 3. Add FAQs and reviews that reduce uncertainty Good product page SEO supports the questions people ask before buying. FAQs and product reviews can provide details your main description doesn't cover. Answer purchase-related questions directly Use genuine questions about fit, compatibility, assembly, care, and included accessories. Each answer should resolve a specific concern. A mouse page might explain whether wireless connectivity uses Bluetooth or a receiver. A furniture page could state whether assembly is required. Explain shipping and returns briefly, then link to your full policies. Repeating entire policies on every product page adds clutter. FAQs remain useful without FAQ structured data. Most ecommerce sites don't qualify for Google's FAQ rich results, and the markup doesn't guarantee enhanced results. Make customer reviews useful and credible Invite honest reviews and make them easy to browse. Product reviews can help shoppers assess sizing, installation, durability, and everyday use. BrightLocal's consumer review research examines local-business decisions. Yoast cites an earlier finding that 74% consulted at least two review sites. That supports the value of reviews, but it isn't a universal product-shopping statistic. Show review dates and distinguish verified purchases when your system supports it. Never invent reviews or ratings for search markup. Use recurring review concerns to improve the description. Repeated compatibility questions often point to missing product information. 4. Make product schema match the page Structured data gives search engines a consistent way to interpret product details. It can make a page eligible for enhanced results, but it doesn't guarantee prices or stars will appear. For a useful JSON-LD implementation, include Product properties such as name, image, description, sku, and brand. Add a valid GTIN when the product has one. Include aggregateRating only when genuine ratings are visible on the page. Use the actual ratingValue and reviewCount or ratingCount. Individual Review entries should contain the real author and reviewRating. For purchasable products, add an Offer with the current price, priceCurrency, and availability. A US-dollar offer uses USD; availability can use Schema.org's InStock, OutOfStock, or PreOrder values. Google's Product structured data guidance explains the supported search features and their requirements. Product schema should draw from the same catalog data that controls the visible page. Manually maintained prices and availability can become outdated quickly. Use BreadcrumbList to match the page's visible breadcrumbs. Where appropriate, ProductGroup can describe related variants. Clear product facts remain useful for conventional search and AI-assisted shopping. Unsupported claims and repetitive "AI-friendly" paragraphs don't improve the buying experience. 5. Choose one URL strategy for variants Color and size variants need a deliberate indexing decision. A different URL alone doesn't make a page meaningfully different. Use this framework when deciding which versions deserve separate pages. SituationRecommended approachVariants have distinct demand and useful contentKeep separate indexable pages with a self-referencing canonical.Variants differ only slightly and repeat the same contentConsolidate indexing signals toward one primary URL.One product appears in several categoriesLink every category to the same primary product URL. For separately indexed variants, provide distinct images, variant-specific descriptions, and a unique SKU. Use each variant's assigned GTIN where available; never invent identifiers. Otherwise, canonicalize duplicate variant URLs to the primary version. Shoppers should still be able to select and share the appropriate variant. Google's canonical URL guidance explains how these signals consolidate duplicate pages. A canonical is a preference signal, not a guaranteed instruction. The same principle applies when category pages generate multiple URLs for one product. Canonicalize them to the primary URL, then keep breadcrumbs, internal links, and sitemaps consistent with it. 6. Improve product images and mobile usability Images should explain the product as well as display it. Useful angles, scale references, and close-ups help shoppers assess the product and improve the user experience. Use descriptive filenames containing the product name, such as logitech-mx-master-3s-front.webp. Write alt text that accurately describes the image, including the product name where appropriate. Avoid repeating keywords across every image. Use compressed WebP or JPG files with suitable dimensions. Include product images in an image sitemap, either separately or through image entries in your existing sitemap. On mobile, keep price, availability, variant selection, and the purchase button easy to find. Our mobile-first indexing guide covers keeping important content accessible. Baymard's product-page UX research reports that only 49% of benchmarked sites achieved decent or good performance. Speed deserves attention too. Google's 2017 mobile speed study found a 123% increase in bounce probability as load time rose from one to ten seconds. Conductor also cites Amazon's historical finding that another 100 milliseconds cost about 1% in sales. These figures aren't forecasts for your store. Use PageSpeed Insights alongside real mobile behavior to identify worthwhile fixes. 7. Handle unavailable products without misleading shoppers A temporary stock shortage usually doesn't justify deleting a useful product page. Keep it accessible, show the shortage clearly, and update the offer to OutOfStock. For permanently discontinued products, retain valuable pages when they still attract relevant visitors or provide useful specifications. Explain the status and offer suitable alternatives. If no offer remains, remove offers from the Product markup. This keeps the data accurate, and Google doesn't impose an automatic penalty because a product is discontinued. Redirect only when a close replacement meets the same need. Sending every retired product to a broad category can disappoint shoppers. 8. Audit product page SEO across your catalog Start with product pages where improvements can support meaningful sales. Conductor's Pareto guidance suggests roughly 20% of categories often generate 80% of revenue. Treat that as a prioritization idea, not a rule every store follows. Fix access and indexing first We prioritize technical access before rewriting hundreds of descriptions. Follow this sequence: Crawl priority products with Screaming Frog. Identify blocked URLs, noindex tags, errors, duplicate titles, and canonical conflicts. Inspect representative URLs in Google Search Console. Primary indexable products should be crawlable, return 200 OK, and use self-referencing canonicals. Review sitemaps and internal links. Include canonical, indexable products and connect them through relevant categories and breadcrumbs. Google's sitemap requirements limit each file to 50,000 URLs or 50 MB uncompressed. Smaller files can simplify diagnosis. Oliver Mason's approach used chronological groups of 10,000 URLs, an organizational choice rather than a requirement. Measure changes against a stable baseline Record Google Search Console impressions, clicks, click-through rate, and average position for product URLs and queries. Separate branded searches from broader product searches where possible. Then track organic add-to-cart activity, purchases, and revenue in your analytics platform. Change a defined group of comparable pages, record the deployment date, and compare equivalent periods. Account for promotions, seasonality, availability, and ranking changes. Our SEO title and description testing guidance explains why snippet tests need careful interpretation. More clicks are useful, but qualified visits and completed purchases provide a stronger business outcome. Make Useful Product Information the Priority Strong product page SEO gives shoppers better answers while making those answers accessible to search engines. Distinctive descriptions earn their place through practical usefulness, not extra length. We recommend improving your priority products first, then applying successful changes across the catalog. Keep product facts, availability, markup, and buying guidance aligned as your store changes. [...]
Read more...
YouTube SEO Checklist for Search-Focused VideosA useful video can miss search traffic when its title promises something the content doesn't deliver. Our YouTube SEO checklist starts with search intent: choose a clear question, answer it well, and make that answer easy to recognize. Table of Contents Toggle Start with How YouTube Search WorksChoose a Search Query Before FilmingDefine the Question Your Video Will AnswerUse Tools to Refine the TopicMake the Video Deliver the AnswerShow the Value EarlyMake Each Step Easy to FollowOptimize the Title, Description, and ThumbnailWrite a Clear, Accurate TitleGive the Description a Useful OpeningMake the Thumbnail Match the ContentComplete the Upload Settings Without Overdoing ThemMeasure YouTube Search PerformanceSeparate Search Traffic from Other ViewsUse Retention to Choose the Next ImprovementKeep Website Video SEO on a Separate TrackKey Takeaways: The Publishing ChecklistFrequently Asked QuestionsCan Shorts Receive YouTube Search Traffic?Should Titles and Descriptions Be Translated?Conclusion: Build Around the Searcher’s Question You don't need to fill every field with keywords or chase a perfect optimization score. You need a focused video and a reliable way to measure its performance. First, let's look at what YouTube Search rewards. Start with How YouTube Search Works YouTube Search considers relevance, engagement, and quality. Its official explanation of how YouTube Search works connects relevance with the title, description, tags, and video content. Engagement includes signals such as watch time for a particular search. Quality considers whether a channel demonstrates expertise, authoritativeness, and trustworthiness on a topic. What does that mean for your checklist? Your metadata should describe the answer your video provides. The video then needs to fulfill that promise. Search results can also reflect a viewer's search and watch history. Two people can receive different results for the same query, so your own search position isn't a complete performance report. We recommend focusing on relevant search traffic and viewer response. A checklist can improve your preparation, but it can't guarantee a position or view count. Choose a Search Query Before Filming Define the Question Your Video Will Answer Start with one main problem. Write down what the viewer should understand or complete after watching. Then enter related phrases into YouTube Search. Review autocomplete suggestions and the videos appearing for those searches. Look at their format, audience level, and scope. Are viewers looking for a walkthrough, comparison, explanation, or troubleshooting guide? That distinction should shape your recording. Include the product, task, or version when those details change the answer. Our guidance on using long-tail keywords explains how focused phrases help you address a clearer need. They don't automatically mean low competition. Use Tools to Refine the Topic YouTube Studio's Trends tab can help you explore audience searches. Use those insights alongside the questions customers ask and the search terms already bringing viewers to your channel. Tools such as vidIQ and TubeBuddy can add keyword and competition insights. Their scores are research aids, not official YouTube ranking scores. Our overview of YouTube keyword research tools can help you compare options. Start with native YouTube data before adding subscriptions. Lastly, choose a topic you can explain accurately. A popular search won't help if your video offers a weaker answer than the existing results. Make the Video Deliver the Answer Show the Value Early Your opening should confirm that viewers have found the right video. State the task, show the outcome when possible, and explain any requirement that affects the instructions. For software tutorials, identify the interface or version shown when it matters. This helps viewers recognize whether the steps apply to them. We recommend moving introductions, channel updates, and lengthy promotional messages away from the opening. Search viewers have already told you what they need through their query. Say the topic naturally when introducing it. Repeating a phrase throughout the recording adds little value if it interrupts the explanation. Make Each Step Easy to Follow Plan the order before recording. Show the relevant screen or object while explaining the action, and keep important details visible. Clear audio matters because viewers need to understand the answer. Remove avoidable pauses and repeated explanations, but keep enough context for beginners. Give comparisons consistent criteria. For troubleshooting, explain the conditions that make a solution relevant. Finish when the question is answered. There's no universal video length that guarantees search performance. We prefer a complete explanation without unnecessary padding. A useful ending can point viewers toward a related tutorial after the main task is complete. Optimize the Title, Description, and Thumbnail Write a Clear, Accurate Title Put the main subject where viewers can recognize it quickly. Add a meaningful qualifier, such as the product version or problem being solved, when it helps distinguish the video. Keep the promise realistic. A title suggesting a complete walkthrough should lead to a complete walkthrough. Avoid adding a year unless freshness matters and the content supports it. Updating a title's year doesn't update the instructions inside the video. Give the Description a Useful Opening Use the first lines to explain what the video covers and who it helps. Give each upload a unique description. YouTube's video description guidance recommends featuring one or two main terms prominently in the title and description. Use them in normal sentences rather than repeating a keyword list. Then add helpful resources, relevant links, and timestamps where appropriate. Keep recurring channel information below the video's own explanation. For business videos, link to the page that supports the next step. A relevant service or resource page is usually more helpful than an unrelated homepage. Make the Thumbnail Match the Content Show a recognizable subject, result, or problem. Keep the main visual clear enough to understand at a small size. The title and thumbnail should work together. One can name the task while the other makes the outcome easier to recognize. Avoid imagery that suggests results the video doesn't show. We recommend reviewing the thumbnail at mobile size before publishing. Small details that look clear on your editing screen may disappear in search results. Complete the Upload Settings Without Overdoing Them Set the video to Public when it's ready for search discovery. Private and unlisted videos aren't intended for normal public search visibility. Next, select the correct video language and review captions. Automatic captions can misinterpret product names, technical terms, and spoken instructions. Correct errors that change the meaning. Add chapters when they help viewers find distinct parts of the answer. Manual chapters must start at 00:00, contain at least three timestamps in ascending order, and give each chapter at least 10 seconds. Chapters and captions improve access and navigation. Treat them as useful viewer features rather than guaranteed ranking boosts. Tags deserve less attention. YouTube's guidance on video tags says titles, thumbnails, and descriptions matter more for discovery. Tags are particularly useful for common misspellings. Use accurate variations where needed, then move on. Don't paste a block of tags into the description. Lastly, preview the published video. Test its sound, captions, chapters, and resource links. A YouTube SEO checklist is only useful when the upload works as intended. Measure YouTube Search Performance Separate Search Traffic from Other Views Open the video's analytics in YouTube Studio and review the Reach tab. The "Traffic source: YouTube Search" card shows search terms that brought viewers to your content when data is available. Compare those terms with your intended topic. Relevant queries suggest that your video is reaching people with the problem it addresses. Keep Search separate from Browse features, Suggested videos, and external traffic. A rise in total views doesn't necessarily mean your search visibility improved. Review impressions and click-through rate where available. Read them alongside watch time and retention, not as isolated success scores. Use Retention to Choose the Next Improvement Audience retention helps you identify where viewers leave, skip, or replay. Review those moments against the content. An early drop is a reason to examine your opening and title promise. A repeated section may need clearer instructions, although replays can also show strong interest. YouTube's performance and discovery guidance supports accurate titles and thumbnails. Changing them can affect viewer response, but an edit doesn't automatically improve ranking. Keep a dated record of meaningful changes. Compare similar periods and account for seasonality or changes in demand. We recommend changing one major element at a time when practical. Small samples can't support strong conclusions, so avoid reacting to every daily fluctuation. Keep Website Video SEO on a Separate Track Embedding a YouTube video on your website creates another way to help customers. However, visibility in Google Search is a separate task from visibility in YouTube Search. Build a useful page around the video. Explain the topic, provide relevant written information, and make the page accessible through internal links. Google Search Console reports website performance and video indexing. YouTube Studio reports your channel's performance. Our guide to Google's video indexing report explains why an indexed page doesn't automatically mean its video is indexed. We recommend treating the website page as a helpful resource of its own. Duplicating the video's description without adding useful context gives visitors little reason to stay. Key Takeaways: The Publishing Checklist Use this short review before publishing and during later performance checks: Choose one main query and identify the answer viewers expect. Deliver that answer clearly, with an opening that confirms the video's purpose. Align the title, description, and thumbnail with the content. Review public visibility, captions, chapters, and relevant resource links. Track YouTube Search terms and viewer response before deciding what to change. Give the answer and packaging the most attention. Tags and optimization scores shouldn't distract you from content that solves the viewer's problem. Frequently Asked Questions Can Shorts Receive YouTube Search Traffic? Yes. YouTube Analytics includes YouTube Search as a discovery source for Shorts as well as longer videos. Choose a question that fits the format and provide a complete answer within the available time. Search performance and Shorts feed performance are different measurements. Review the traffic sources rather than assuming all Shorts views came through the feed. Should Titles and Descriptions Be Translated? Translated titles and descriptions can help viewers find videos in their language. Use YouTube's localization options instead of filling one title with several languages. Keep the viewing experience in mind. Accurate subtitles or supported audio options help people understand the content after clicking. Translate only when you can maintain the meaning of the original instructions. Conclusion: Build Around the Searcher’s Question A reliable YouTube SEO checklist connects the search query, video content, and packaging. We recommend giving each upload a clear purpose and measuring whether it attracts the right viewers. Start with the question you can answer well. Then use YouTube Search data to improve your next decision. Useful answers give your optimization work a stronger foundation than keyword repetition or a perfect tool score. [...]
Read more...
Google Sitelinks: What Small Businesses Can InfluenceWant customers to reach your services or contact page directly from Google? Google sitelinks can provide those shortcuts, but you can't switch them on or choose which pages appear. Table of Contents Toggle How Google sitelinks workWhat you control and what Google decidesOrganize the website around customer prioritiesPut important destinations in the main navigationGroup related pages logicallyUse internal links customers can understandConnect important pages through relevant contentMake the destination clear before the clickGive each page a clear, distinct purposeAlign titles, headings, and page contentAvoid pages that compete with nearly identical contentKeep important pages crawlable and indexableHandle unwanted sitelinks carefullyReview progress without chasing the layoutBuild a Clear Website Before Chasing Sitelinks We recommend focusing on clear navigation, useful pages, and meaningful internal links. These improvements help customers use your website and help Google understand its structure. First, let's separate the website improvements you control from the search results Google controls. How Google sitelinks work Sitelinks are additional links from your website grouped beneath a Google text result. They give searchers shortcuts to pages beyond the main result. According to Google's sitelinks documentation, its systems analyze your site's link structure to identify useful shortcuts. Google displays them only when they appear helpful for the search. This means sitelinks can vary by query. A search for your business name and a search for one of your services don't necessarily produce the same links. Why do they matter? A customer can reach the page they need without first visiting your homepage. That can shorten the path to an inquiry, booking, or purchase. Organic sitelinks are also separate from sitelink assets in Google Ads. Advertisers can configure assets for their campaigns. That doesn't give them control over the links beneath an organic search result. What you control and what Google decides You control your website's content, organization, and links. Google decides whether sitelinks are useful and how they appear. Here's the practical distinction: Website elementWhat you can influenceWhat Google controlsNavigationWhich pages your menus connectWhether those pages become sitelinksPage titlesClear, accurate title wordingHow links are presented in searchInternal linksDestinations and descriptive link textHow those signals inform its selectionSearch eligibilityWhether pages are accessible and indexableWhether eligible pages appearResult layoutNo direct configuration for organic sitelinksWhich links appear and their order Our advice is to improve the underlying website rather than chase a particular arrangement of links. Those improvements remain useful even when sitelinks don't appear. There isn't a Search Console setting for selecting or ordering organic sitelinks. Adding structured data doesn't create that control either. Treat sitelinks as a possible outcome of a well-organized website. They aren't a promised deliverable, and their absence alone doesn't prove your SEO is failing. Organize the website around customer priorities A clear structure makes important pages easier to find. Start with what customers need, then organize the navigation around those needs. Put important destinations in the main navigation Your main menu should provide a straightforward route to your core services, business information, and contact options. Add location pages where they help customers understand where you operate. Use familiar labels that explain the destination. "Services" and "Contact" are easier to understand than creative wording that leaves visitors guessing. We recommend reviewing the menu on mobile as well as desktop. Important links should remain easy to find when the navigation collapses. For WordPress websites, our small-business WordPress SEO checklist covers navigation and other practical foundations. Group related pages logically A service overview can introduce your main offerings and connect visitors to dedicated service pages. A location directory can organize genuine branches or service areas. Breadcrumbs can also show where a page sits within the website. Their value is clearer navigation; they don't guarantee a particular sitelink. Avoid placing every page in the main menu. A crowded menu makes priorities harder to recognize. Instead, use supporting pages and relevant links to keep the structure browseable. Customers should be able to find a service without starting another Google search. Use internal links customers can understand Think of internal links as directions inside your business. Clear directions help customers reach the right destination without unnecessary steps. Connect important pages through relevant content Menus aren't the only place to add links. Your homepage, service descriptions, and helpful articles can point visitors toward related information. We recommend linking where the next step makes sense. A service page can connect to its booking page, pricing information, or a related service. An important page shouldn't rely entirely on a sitemap for discovery. It needs a route through the website itself. Our internal linking SEO guide explains how to connect pages without turning every paragraph into a collection of links. Relevance matters more than adding links everywhere. Make the destination clear before the click Link text should tell visitors what they'll find. "View available services" gives more information than "click here." Google's crawlable links guidance recommends descriptive, reasonably concise anchor text. It also explains that Google generally crawls links built with an HTML anchor element and an href attribute. Most website editors create these links automatically when you insert a standard text link. Custom interactive elements need more care. Lastly, avoid stuffing keywords into link text. Use wording that fits the sentence and accurately describes the destination. Google doesn't provide a magic ideal number of links per page. Give each page a clear, distinct purpose Google recommends informative, relevant, compact titles and headings for sitelinks. Clear wording helps customers recognize a page's purpose before they open it. Align titles, headings, and page content A page title should identify what the page offers. Its main heading should support that same purpose, and the content should deliver on it. We recommend distinguishing service, location, and contact pages instead of repeating the same broad business description across every title. Google's title-link guidance explains that search titles are generated automatically using several sources. These include title elements, visible headings, and text in links pointing to the page. Writing a clearer title improves the information available to Google. It doesn't give you direct control over a sitelink's wording or appearance. Avoid pages that compete with nearly identical content Separate pages should answer separate customer needs. Repeating the same service description with small wording changes makes the website harder to understand. Build dedicated landing pages when customers need distinct information. Explain the service, its availability, and the next step clearly. Internal search results, shopping-cart steps, and filter combinations can generate many URLs without adding useful landing pages. These shouldn't automatically become the focus of your indexing strategy. We recommend reviewing overlapping pages before creating more. Where pages serve the same purpose, consolidation may produce a clearer destination. Preserve useful information and use appropriate redirects when replacing an existing URL. Keep important pages crawlable and indexable A useful page still needs to be accessible to Google. Crawlability means Google can access it. Indexing means Google has processed it for possible inclusion in search. These are separate stages, as Google's explanation of Search describes. Access doesn't guarantee indexing, and indexing doesn't guarantee a sitelink. Start with Google Search Console's URL Inspection tool. Review important pages for indexing status, accidental noindex instructions, access problems, and unexpected canonical selection. A canonical URL is the preferred version among duplicate or similar URLs. Your internal links and sitemap should consistently support the versions you want indexed. Keep your XML sitemap focused on preferred, indexable pages. Don't include every filter combination or internal search URL simply because the website generates it. Update modification dates when meaningful content changes occur. Submitting a sitemap helps with discovery. It doesn't nominate pages as Google sitelinks. IndexNow also doesn't replace Google's discovery methods. For an important new or substantially updated page, an indexing request can support the process. It doesn't guarantee inclusion or a particular result layout. Our technical SEO audit checklist helps you work through access and indexing problems before spending time on search appearance. Handle unwanted sitelinks carefully What can you do when Google highlights the wrong page? First, decide whether the page belongs in search at all. If the page is useful but outdated, improve its information, title, and links. Remove misleading navigation that makes an obsolete page look like a current priority. If a page genuinely shouldn't appear in search, Google's documented options include removing it or adding noindex. Neither option is a sitelink-only adjustment. A noindex instruction removes a page's eligibility for search results, not just its appearance as a sitelink. Google must be able to crawl the page to see that instruction. Blocking it through robots.txt can prevent Google from reading noindex. We don't recommend removing a valuable service page just because you dislike its position beneath a result. Consider the wider effect on search visibility and customers who already use that page. Changes also require processing time. There's no reliable deadline for sitelinks to update. Review progress without chasing the layout Search Console helps you monitor search visibility and investigate individual pages. It isn't an organic sitelinks selection tool. Use it to review clicks, impressions, and the queries associated with important pages. Your website analytics can show whether visitors reach inquiry forms, booking pages, or other useful destinations. For a practical review, we recommend this order: Identify the pages customers need to reach most often. Improve the navigation and relevant internal links leading to those pages. Resolve unclear titles, overlapping content, and indexing problems. Monitor search traffic and customer actions after meaningful changes. You can also review branded search results periodically. Treat what you see as a snapshot, since sitelinks depend on the query. The stronger measure of success is whether customers find useful information and take the next step. A larger search result doesn't automatically mean more qualified inquiries. Build a Clear Website Before Chasing Sitelinks Google sitelinks depend on Google's systems, but the quality of your website is within your control. We recommend prioritizing clear destinations, relevant links, and pages that answer real customer needs. Start with your most important service and contact pages. Make them easy to reach, easy to understand, and eligible for search. Those improvements support your business whether Google displays sitelinks today, changes them later, or shows a standard result. [...]
Read more...
AI Crawler Rules for Small Business WebsitesYour website can welcome AI search tools without giving every crawler the same instructions. Practical AI crawler rules separate search discovery, training collection, and private access, so you can make choices that support your business. Table of Contents Toggle What AI Crawler Rules Can ControlRobots.txt is a request, not a security measureCrawling and indexing are separate decisionsWhich AI Bots Handle Search, Training, and User Requests?OpenAI separates search from trainingGooglebot and Google-Extended have different rolesAnthropic provides three separate controlsHow to Set Up Robots.txt CorrectlyPut the file in the right locationUnderstand the three main directivesPractical Rules for Search Visibility and Training ChoicesAllow search discovery while declining training crawlsRestrict selected paths without blocking useful pagesWhen to Use Enforceable Access ControlsHow Llms.txt Differs From Access ControlsPublish the Rules and Monitor Their EffectsMake one controlled updateWatch access, indexing, and business outcomes separatelyKeep Crawler Choices Separate From Website Security We recommend starting with the purpose of each bot, then applying targeted controls instead of blocking everything. Robots.txt communicates your preferences, but authentication and server-side restrictions protect content that must stay private. First, let's separate what these controls can do from what they can't promise. What AI Crawler Rules Can Control Robots.txt is a request, not a security measure Think of robots.txt as instructions posted at your website's entrance. Cooperative crawlers read those instructions before requesting pages. The Robots Exclusion Protocol standard states that these rules aren't access authorization. A crawler can ignore them, and someone can still open a disallowed page directly. That distinction matters for customer records, private pricing, and unpublished documents. Listing a folder in robots.txt doesn't protect its contents. The file is public, so its entries can reveal paths you would rather keep private. We recommend using robots.txt for crawler preferences and real access controls for confidential information. Crawling and indexing are separate decisions Crawling means fetching a page. Indexing means adding information about that page to a search system. Allowing the first doesn't guarantee the second. According to Google's robots.txt guidance, a blocked URL can still appear in search if Google discovers it through links. For a public page that shouldn't appear in Google, use a supported noindex directive and allow Googlebot to read it. Blocking the page first can prevent Google from seeing that instruction. The same practical distinction applies to AI visibility: crawler access creates an opportunity for discovery, not a promise of inclusion or citation. Which AI Bots Handle Search, Training, and User Requests? Crawler-specific user agents let you give different instructions to different services. These names matter because one provider can operate several bots with different purposes. OpenAI separates search from training OpenAI's crawler documentation distinguishes OAI-SearchBot, which supports ChatGPT search discovery, from GPTBot, which collects content that may support model training. You can allow OAI-SearchBot while disallowing GPTBot. That gives you a more targeted choice than a blanket OpenAI block. ChatGPT-User handles user-triggered visits rather than automatic crawling. OpenAI says robots.txt rules may not apply to these visits. It also doesn't determine whether your content appears in ChatGPT Search. Googlebot and Google-Extended have different roles Googlebot handles Google Search crawling. Blocking it can prevent Google from accessing the content you want customers to find. Google-Extended is a robots.txt product-use token, not a separate crawler identity in HTTP requests. It controls certain Gemini training and grounding uses. Disallowing Google-Extended doesn't affect inclusion in Google Search and isn't a Search ranking signal. We recommend keeping that choice separate from your Googlebot rules. Anthropic provides three separate controls Anthropic's crawler guidance identifies ClaudeBot for potential training collection, Claude-SearchBot for search, and Claude-User for user-requested access. Anthropic says its bots honor robots.txt. Blocking Claude-SearchBot can reduce search visibility, while blocking Claude-User can prevent retrieval for user requests. Each bot needs its own targeted instruction. A rule for ClaudeBot doesn't automatically express your preference for the other two. How to Set Up Robots.txt Correctly Put the file in the right location Publish a plain-text file named robots.txt at the root of the relevant website host. Its root-relative location is /robots.txt. A file inside a blog folder won't control the entire website. Subdomains also need their own rules. Your main website's file doesn't automatically govern a separate shop or customer portal. For WordPress, a plugin or the platform may already generate a virtual robots.txt file. We recommend identifying the current source before editing it, so you don't create competing configurations. Our robots.txt SEO guide covers the surrounding crawling and indexing basics. Understand the three main directives User-agent identifies the crawler the following rules address. Disallow lists paths that crawler shouldn't fetch. Allow permits a path, including an exception within a broader restriction. For example, Disallow: / requests a complete crawling block for that group. Disallow: /wp-admin/ targets a WordPress directory instead. The wildcard in User-agent: * addresses crawlers generally. However, a crawler-specific group doesn't automatically inherit every rule from the wildcard group. Repeat restrictions that need to apply to that crawler. Path matching is case-sensitive. Keep directory names accurate, and preserve access to CSS and JavaScript that search engines need to render your pages. Practical Rules for Search Visibility and Training Choices Allow search discovery while declining training crawls For a business that wants ChatGPT search discovery but prefers to decline GPTBot collection, these two groups express that choice: User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / Place each directive on its own line in the actual file. Keep a blank line between groups. You can add separate groups using User-agent: ClaudeBot with Disallow: /, and User-agent: Google-Extended with Disallow: /, to express the corresponding preferences. These instructions govern the identified uses. They don't remove material already collected or prevent every possible source of AI access. Restrict selected paths without blocking useful pages A complete block isn't always the right choice. You can restrict a directory for a selected crawler while leaving service pages and helpful articles accessible. For WordPress, this group requests a restriction on the administration directory while allowing its AJAX endpoint: User-agent: GPTBot Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php The longer, matching allowance creates an exception to the directory restriction. This example limits that path; it doesn't decline GPTBot collection across the whole website. We recommend choosing rules around your actual goal. Search visibility, training preferences, and server-load management are different decisions. Combining them into one blanket block can close off access you intended to keep. When to Use Enforceable Access Controls What can you do when a bot ignores robots.txt? Use controls that decide whether the server delivers the content. Authentication requires a valid login before access. That's the right foundation for customer portals, private documents, and staging websites. Authorization then determines which logged-in users can access particular resources. A web application firewall, often called a WAF, can block requests based on rules. A content delivery network can also apply bot controls or rate limits before requests reach your hosting server. User-agent names alone aren't reliable proof of identity. A scraper can claim to be Googlebot or an AI crawler. Where available, use verified-bot features or documented network identity checks instead of trusting the name alone. Rate limiting is useful when request volume causes performance problems. An HTTP 429 response communicates that too many requests have been made. Keep enforcement targeted. Applying login requirements or aggressive bot blocks to public service pages can damage search access. Our guide to authentication errors and search visibility explains why those restrictions belong on the right pages. We recommend protecting private areas first, then addressing unwanted public crawling without disrupting customer access. How Llms.txt Differs From Access Controls The llms.txt proposal describes a file intended to help agents use a website. It can provide context and direct them toward useful information. For a small business, that might mean identifying your main service information, documentation, or policies. Its value depends on whether a tool retrieves and uses it. An llms.txt file doesn't authenticate visitors, block downloads, or prevent scraping. Writing "don't train on this content" inside it doesn't create an enforceable restriction. It also doesn't replace crawler-specific robots.txt groups. The files have different purposes: robots.txt communicates crawling preferences, while llms.txt supplies information for agents. We recommend treating llms.txt as an optional information resource, not a requirement for AI visibility. It doesn't guarantee citations, traffic, or better rankings. For a small website, clear public content, working internal links, and accessible service pages deserve attention first. A machine-readable summary can't compensate for missing information or blocked pages. If you publish one, keep it aligned with your website. Outdated links and descriptions can send an agent toward the wrong content rather than simplify discovery. Publish the Rules and Monitor Their Effects Make one controlled update Start with the existing configuration rather than replacing it with a generic bot list. A short, intentional file is easier to maintain. We recommend this sequence: Save the existing file so you can restore it if necessary. Add only the crawler groups and path rules that match your decisions. Publish the update and confirm the root file returns plain text successfully. Test important public pages and private areas for their intended access. If a CDN caches the file, clear the relevant cached copy after publication. Keep the change date and the reason for each restriction in your maintenance records. Watch access, indexing, and business outcomes separately Server logs show requested URLs, claimed user agents, response codes, and request volume. They help identify whether crawling is creating errors or unnecessary load. For Google visibility, use Search Console's URL Inspection tool to assess important pages. Crawling alone doesn't guarantee indexing, even when discovery happens sooner. IndexNow is a change-notification protocol for participating engines. It doesn't override access restrictions or guarantee search inclusion. For a rarely updated site, content quality and confirmed indexing problems usually deserve more attention than custom notification work. Our technical SEO audit checklist helps organize those wider checks. Crawler identities, behavior, and provider policies can change. We recommend a monthly review, with another review after hosting, security, or website changes. Measure visibility and inquiries separately from bot activity. Keep Crawler Choices Separate From Website Security Effective AI crawler rules match each bot's purpose to your business goals. You can support search discovery, express training preferences, and protect private content with different controls. We recommend starting with a small set of targeted rules and reviewing their effects. Keep confidential information behind real access controls, regardless of what robots.txt says. Your website should stay useful to customers and accessible to the services you choose. Clear rules support that goal without promising control they can't deliver. [...]
Read more...