Customizing robots.txt

By default, Kirby SEO generates a simple robots.txt that allows all crawlers and blocks the Panel. If you need to add your own rules, use the robots.content option.

Blocking specific bots

Some AI providers crawl websites to use the content as training data. You can block their crawlers:

<?php
// site/config/config.php

return [
  'tobimori.seo' => [
    'robots' => [
      'content' => [
        'GPTBot' => [
          'Disallow' => ['/'],
        ],
        'Google-Extended' => [
          'Disallow' => ['/'],
        ],
        'CCBot' => [
          'Disallow' => ['/'],
        ],
      ],
    ],
  ],
];

This adds rules for each bot while keeping the default rules for all other crawlers intact.

Custom rules for all crawlers

If you set rules for *, they replace the default rules entirely:

'content' => [
  '*' => [
    'Allow' => ['/'],
    'Disallow' => ['/panel', '/content', '/private'],
  ],
],

Mixing rules

You can combine rules for all crawlers with rules for specific bots:

'content' => [
  '*' => [
    'Allow' => ['/'],
    'Disallow' => ['/panel', '/content'],
  ],
  'GPTBot' => [
    'Disallow' => ['/'],
  ],
],

The Sitemap: line is added automatically if the sitemap module is active. You can override it with the robots.sitemap option.