Manage robots.txt

robots.txt is a file that tells search engines and AI bots what they may read on your site. Once your site is connected to Weekerp, you can edit robots.txt right in Weekerp, even if it is hard to change on your platform.

 

1. Turn on Weekerp management

1. In the sidebar, open Sites > Advanced. If you have several sites, pick one under Site at the top.
2. Turn on the toggle at the top right of the robots.txt card.

An editor opens, filled with your original site's current robots.txt plus the Weekerp lines (such as the blog sitemap). Turning on the toggle without saving does not change your live site.

If you have connected a blog and Weekerp lines are already being added, the toggle is on from the start.

 

2. Edit and save

1. Edit the content in the editor. You can also edit or delete the lines Weekerp added.
2. Click Save.

What you save is served exactly as is. It reaches your live site within a few minutes. To check, open https://your-domain/robots.txt.

 

Common examples

Allow all bots everywhere, block only the admin pages:

User-agent: *
Disallow: /admin

Block only a specific AI bot (for example, GPTBot):

User-agent: GPTBot
Disallow: /

 

If you add a line that blocks the whole site

If Disallow: / appears under User-agent: *, a warning shows below the editor and you are asked to confirm again when you save. If you save it as is, your site drops out of search results and AI answers. Lines that block only part of the site, such as /admin, do not trigger this.

 

Turn off Weekerp management

When you turn off the toggle, the Turn off robots.txt management dialog opens. Type Turn off and click Turn off, and your original site's robots.txt is served as is. The blog sitemap lines are removed too.

 

Good to know

• You can save up to 100KB.
• Settings are saved separately for each site (address).
• To see whether AI bots actually read your site, go to Analytics > Bot crawling.