Blocking AI Crawlers from Your Site
Control who can access and use your content by blocking AI crawlers from your site. This is particularly useful for site owners who want to prevent their content from being used to train large language models (LLMs) without permission. For example, a site owner with unique research articles might not want their content scraped and used by AI without consent.
Using the robots.txt file to block AI crawlers
Your site's robots.txt file contains instructions to allow or disallow certain requests from search engines. The “allow” command tells crawlers the links that they can follow, while the “disallow” command tells crawlers the links that they can't follow.
To set the robots.txt file to block AI crawlers:
Go to SEO & GEO in your site's dashboard.
Scroll down to Tools and settings.
Click Robots.txt Editor.
Click View File.
Click the field under Your robots.txt file.
Add your robots.txt file info by writing the following directives under This is your current file:
User-agent: CCBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: BingAI
Disallow: /
Click Save Changes.
Click Save.
Tip:
This list includes some known AI crawlers, but you can always add more as needed.
Using Robots Meta Tags to Block AI Crawlers
Meta tags are pieces of HTML code that contain page information that web browsers and search engines can read. Configuring your site's robots meta tags allows you to have page-specific control over how the HTML is crawled and indexed in search results.
To use meta tags to block AI crawlers:
Wix Editor
Click Pages & Menu
on the left side of the editor.
Click the More Actions icon
next to the relevant page.
Click SEO basics.
Click the Advanced SEO tab.
Click Robots meta tag.
Select the nosnippet checkbox to prevent your page's content from being used as a direct input for AI overviews.
Publish your site.
Studio Editor
Click Pages
on the left side of the editor.
Click the More Actions icon
next to the relevant page.
Click SEO Basics.
Click the Advanced SEO tab.
Click Robots meta tag.
Select the nosnippet checkbox to prevent your page's content from being used as a direct input for AI overviews.
Publish your site.
Learn more about robots meta tags from Google.
FAQs
Click a question below to learn more about blocking AI crawlers from your site.
Can I block all crawlers?
You can block most known crawlers by listing them in your robots.txt file. However, some crawlers may ignore these directives, so it's not an airtight solution.
Is there a way to block unauthorized crawlers completely?
No, some crawlers may ignore the robots.txt file and scrape your content anyway.
Will blocking crawlers affect my SEO?
Blocking specific AI crawlers should not significantly impact your SEO. However, always review your site's analytics to ensure there are no negative effects.
How do I know which crawlers to block?
Start with known AI crawlers like CCBot, GPTBot, ChatGPT-User, and BingAI. You can always add more as you identify them.
If I block AI crawlers, can I still use NLWeb?
Blocking AI crawlers in your robots.txt file may limit NLWeb functionality, as AI agents need access to view your structured content. Learn more about enabling NLWeb for AI Agent access.