Crawl profiles | laravel-sitemap | Spatie

 SPATIE

  Laravel Sitemap
==================

spatie.be/open-source

  [Docs](https://spatie.be/docs)  [Laravel-sitemap](https://spatie.be/docs/laravel-sitemap/v8)  Customizing-the-crawler  Crawl profiles

 Version   v8

 Other versions for crawler [v8](https://spatie.be/docs/laravel-sitemap/v8)

- [ Introduction ](https://spatie.be/docs/laravel-sitemap/v8/introduction)
- [ Installation & setup ](https://spatie.be/docs/laravel-sitemap/v8/installation-setup)
- [ Support us ](https://spatie.be/docs/laravel-sitemap/v8/support-us)
- [ Changelog ](https://spatie.be/docs/laravel-sitemap/v8/changelog)
- [ Questions and issues ](https://spatie.be/docs/laravel-sitemap/v8/questions-issues)
- [ About us ](https://spatie.be/docs/laravel-sitemap/v8/about-us)

Creating sitemaps
-----------------

- [ Generating a sitemap ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/generating-a-sitemap)
- [ Manually creating a sitemap ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/manually-creating-a-sitemap)
- [ Creating a sitemap index ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/creating-a-sitemap-index)
- [ Splitting large sitemaps ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/splitting-large-sitemaps)
- [ Adding an XSL stylesheet ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/adding-an-xsl-stylesheet)
- [ Returning a response ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/returning-a-response)
- [ Sorting URLs ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/sorting-urls)

Customizing the crawler
-----------------------

- [ Crawl profiles ](https://spatie.be/docs/laravel-sitemap/v8/customizing-the-crawler/crawl-profiles)
- [ Filtering URLs ](https://spatie.be/docs/laravel-sitemap/v8/customizing-the-crawler/filtering-urls)
- [ Configuring the crawler ](https://spatie.be/docs/laravel-sitemap/v8/customizing-the-crawler/configuring-the-crawler)

Adding content to URLs
----------------------

- [ Alternates ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/alternates)
- [ Images ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/images)
- [ Videos ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/videos)
- [ News ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/news)
- [ Sitemapable models ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/sitemapable-models)

Advanced usage
--------------

- [ Scheduled generation ](https://spatie.be/docs/laravel-sitemap/v8/advanced-usage/scheduled-generation)
- [ Writing to disks ](https://spatie.be/docs/laravel-sitemap/v8/advanced-usage/writing-to-disks)

 Crawl profiles
==============

You can create a custom crawl profile by implementing the `Spatie\Crawler\CrawlProfiles\CrawlProfile` interface and customizing the `shouldCrawl()` method for full control over what URL, domain, or subdomain should be crawled:

```
use Spatie\Crawler\CrawlProfiles\CrawlProfile;

class CustomCrawlProfile implements CrawlProfile
{
    public function __construct(protected string $baseUrl)
    {
    }

    public function shouldCrawl(string $url): bool
    {
        if (parse_url($url, PHP_URL_HOST) !== 'localhost') {
            return false;
        }

        return parse_url($url, PHP_URL_PATH) === '/';
    }
}
```

Register your `CustomCrawlProfile::class` in `config/sitemap.php`:

```
return [
    /*
     * The sitemap generator uses a CrawlProfile implementation to determine
     * which urls should be crawled for the sitemap.
     */
    'crawl_profile' => CustomCrawlProfile::class,
];
```