# Standard crawlers
User-Agent: *
Disallow: /backend4130/
Disallow: /downloads/
Disallow: /dev/
Disallow: /newsletter.php
# Reduce crawler load on faceted and sorted listings and on-site search.
# These params multiply every listing into a near-infinite crawl space and drove
# origin overload (522s).
#
# Pagination is deliberately NOT among them, and the rule that used to cover it was
# broader than it looked: these patterns match by substring, so /*page= also blocked
# specials_page=, products_new_page=, n_page=, p_page=, c_page= and m_page= - i.e.
# EVERY paginated listing in the shop, not just category and brand pages.
#
# All of it is bounded and small: 47 extra URLs per language for categories, 49 for
# brands, and low double digits for the sale, deals, super sale, new products, news
# and comparison-index listings. Five pages deep at most on a category. Against that,
# blocking it cut the only internal link path to the ~1.600 products sitting past
# page 1 of their category and the ~1.900 past page 1 of their brand - reachable
# through the sitemap alone, indexed but cut off from the link weight of the listing
# above them - and it kept the comparison index from being crawled past its first 60
# entries, which is the discovery path for every /vergleich/ page.
#
# The combinations that caused the overload stay out regardless, because a URL
# carrying a blocked param is blocked whether or not it also carries page=. The
# facet params (1o<id>, filter_id[], pp) are not listed at all - they come from
# checkboxes in a GET form, and a crawler never submits one.
#
# Canonical clean SEO URLs and Shopping-feed landing URLs (...&st=feed_*) stay crawlable.
Disallow: /*sort=
Disallow: /*availability=
Disallow: /*specials=
Disallow: /*tipp=
Disallow: /advanced_search_result.php
Allow: /

# AI Crawlers - Explicitly allow for visibility in AI search/answers
User-Agent: GPTBot
User-Agent: ChatGPT-User
User-Agent: OAI-SearchBot
User-Agent: ClaudeBot
User-Agent: anthropic-ai
User-Agent: Google-Extended
User-Agent: PerplexityBot
User-Agent: Applebot-Extended
Allow: /

Sitemap: https://www.kunstform.org/sitemap.xml