Crawl profiles

You can create a custom crawl profile by implementing the Spatie\Crawler\CrawlProfiles\CrawlProfile interface and customizing the shouldCrawl() method for full control over what URL, domain, or subdomain should be crawled:

use Spatie\Crawler\CrawlProfiles\CrawlProfile;

class CustomCrawlProfile implements CrawlProfile
{
    public function __construct(protected string $baseUrl)
    {
    }

    public function shouldCrawl(string $url): bool
    {
        if (parse_url($url, PHP_URL_HOST) !== 'localhost') {
            return false;
        }

        return parse_url($url, PHP_URL_PATH) === '/';
    }
}

Register your CustomCrawlProfile::class in config/sitemap.php:

return [
    /*
     * The sitemap generator uses a CrawlProfile implementation to determine
     * which urls should be crawled for the sitemap.
     */
    'crawl_profile' => CustomCrawlProfile::class,
];