# ============================================ # POLITICAL CAMPAIGN ROBOTS.TXT # Jonny Larsen For Congress - Utah's 4th District # ============================================ # Default - allow all legitimate crawlers User-agent: * Allow: / Disallow: /analysis Disallow: /summary # NOTE: No Disallow for /donate, /campaign-email-thank-you, /volunteer-thank-you, # /delegate-yes, /delegate-no. These paths either 301-redirect (donate → ActBlue, # thank-you pages → home, delegate-yes/no → home) so we WANT Google to crawl # them and see the redirect for proper URL consolidation. Previously disallowed, # which kept these URLs stuck in "known but not indexed" limbo forever. # # /analysis and /summary ARE Disallow'd because they're password-gated downloads # with no public content to consolidate (both sit behind the SAME gate; only the # document differs). Triple-layer noindex: this Disallow + # on the page + X-Robots-Tag: noindex on the function response. # ---- TRADITIONAL SEARCH ENGINES (ESSENTIAL) ---- User-agent: Googlebot User-agent: Googlebot-Image User-agent: Googlebot-News User-agent: bingbot User-agent: DuckDuckBot User-agent: Applebot Allow: / Disallow: /analysis Disallow: /summary # ---- AI SEARCH CRAWLERS (VISIBILITY) ---- # ChatGPT Search and Training User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot Allow: / Disallow: /analysis Disallow: /summary # Claude AI User-agent: ClaudeBot User-agent: Claude-Web Allow: / Disallow: /analysis Disallow: /summary # Perplexity AI User-agent: PerplexityBot Allow: / Disallow: /analysis Disallow: /summary # Google Gemini/Bard User-agent: Google-Extended Allow: / Disallow: /analysis Disallow: /summary # Other AI Platforms User-agent: YouBot User-agent: Amazonbot User-agent: Applebot-Extended Allow: / Disallow: /analysis Disallow: /summary # ---- SOCIAL MEDIA CRAWLERS (SHARING PREVIEWS) ---- User-agent: facebookexternalhit User-agent: Twitterbot User-agent: LinkedInBot User-agent: WhatsApp User-agent: Slackbot User-agent: TelegramBot Allow: / Disallow: /analysis Disallow: /summary # ---- AGGRESSIVE BOTS (BLOCK) ---- User-agent: Bytespider User-agent: CCBot User-agent: Baiduspider User-agent: PetalBot User-agent: MJ12bot User-agent: AhrefsBot User-agent: DataForSeoBot User-agent: DotBot User-agent: SemrushBot Disallow: / # ---- KNOWN SCRAPERS (BLOCK) ---- User-agent: HTTrack User-agent: Scrapy User-agent: wget User-agent: curl Disallow: / # Sitemap location Sitemap: https://jonnyutah4congress.com/sitemap.xml