Overview of Weekerp Crawlers

Learn about Weekerp-SitemapBot, including its purpose, crawling behavior, User-Agent information, robots.txt rules, and how to control crawler access.

Last updated: November 8, 2026

Weekerp operates web crawlers (bots) to generate and manage sitemaps for customer websites.

Weekerp crawlers operate only on websites where customers have enabled sitemap management. They check page accessibility, search engine directives, and content changes to keep sitemaps up to date.

Weekerp Crawler List

User Agent Description and Details
Weekerp-SitemapBot A web crawler used for Weekerp's sitemap management service. It visits pages to determine whether they are eligible for inclusion in a sitemap and checks for content changes. By default, it operates only on websites that customers have connected to Weekerp.

 

Weekerp-SitemapBot

Weekerp-SitemapBot may request URLs beginning with /__weekerp_404_ to check how a website responds to non-existent pages. Seeing these URLs in your server logs is normal behavior.

User-Agent string:

Mozilla/5.0 (compatible; Weekerp-SitemapBot/1.0; +https://weekerp.com/docs/bots)

robots.txt User-Agent token:

Weekerp-SitemapBot

 

Controlling Crawler Access

Weekerp-SitemapBot respects robots.txt rules and applies them in the following order of priority:

Example:

  1. If rules for User-agent: Weekerp-SitemapBot exist, the crawler follows those rules.

  2. Otherwise, it follows the rules for User-agent: Googlebot, if present.

  3. If neither exists, it follows the rules for User-agent: *.

To block only the Weekerp crawler, add the following rules to your robots.txt file:

User-agent: Weekerp-SitemapBot
Disallow: /

Changes to robots.txt are reflected within one hour. Weekerp customers can also stop crawler visits by disabling sitemap management in the Weekerp console.

 

Crawler Verification and Support

User-Agent strings displayed in server logs can be impersonated by other programs. Therefore, a User-Agent string alone cannot verify whether a request actually originated from Weekerp.

If you notice unusual behavior from requests identifying themselves as Weekerp crawlers, please contact our support team.