Configuring the crawler | laravel-sitemap | Spatie

 SPATIE

  Laravel Sitemap
==================

spatie.be/open-source

  [Docs](https://spatie.be/docs)  [Laravel-sitemap](https://spatie.be/docs/laravel-sitemap/v8)  Customizing-the-crawler  Configuring the crawler

 Version   v8

 Other versions for crawler [v8](https://spatie.be/docs/laravel-sitemap/v8)

- [ Introduction ](https://spatie.be/docs/laravel-sitemap/v8/introduction)
- [ Installation & setup ](https://spatie.be/docs/laravel-sitemap/v8/installation-setup)
- [ Support us ](https://spatie.be/docs/laravel-sitemap/v8/support-us)
- [ Changelog ](https://spatie.be/docs/laravel-sitemap/v8/changelog)
- [ Questions and issues ](https://spatie.be/docs/laravel-sitemap/v8/questions-issues)
- [ About us ](https://spatie.be/docs/laravel-sitemap/v8/about-us)

Creating sitemaps
-----------------

- [ Generating a sitemap ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/generating-a-sitemap)
- [ Manually creating a sitemap ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/manually-creating-a-sitemap)
- [ Creating a sitemap index ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/creating-a-sitemap-index)
- [ Splitting large sitemaps ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/splitting-large-sitemaps)
- [ Adding an XSL stylesheet ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/adding-an-xsl-stylesheet)
- [ Returning a response ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/returning-a-response)
- [ Sorting URLs ](https://spatie.be/docs/laravel-sitemap/v8/creating-sitemaps/sorting-urls)

Customizing the crawler
-----------------------

- [ Crawl profiles ](https://spatie.be/docs/laravel-sitemap/v8/customizing-the-crawler/crawl-profiles)
- [ Filtering URLs ](https://spatie.be/docs/laravel-sitemap/v8/customizing-the-crawler/filtering-urls)
- [ Configuring the crawler ](https://spatie.be/docs/laravel-sitemap/v8/customizing-the-crawler/configuring-the-crawler)

Adding content to URLs
----------------------

- [ Alternates ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/alternates)
- [ Images ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/images)
- [ Videos ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/videos)
- [ News ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/news)
- [ Sitemapable models ](https://spatie.be/docs/laravel-sitemap/v8/adding-content-to-urls/sitemapable-models)

Advanced usage
--------------

- [ Scheduled generation ](https://spatie.be/docs/laravel-sitemap/v8/advanced-usage/scheduled-generation)
- [ Writing to disks ](https://spatie.be/docs/laravel-sitemap/v8/advanced-usage/writing-to-disks)

 Configuring the crawler
=======================

###  On this page

1. [ Limiting the amount of pages crawled ](#content-limiting-the-amount-of-pages-crawled)
2. [ Setting the crawl concurrency ](#content-setting-the-crawl-concurrency)
3. [ Controlling the crawl depth ](#content-controlling-the-crawl-depth)
4. [ Executing JavaScript ](#content-executing-javascript)

The crawler itself can be [configured](https://github.com/spatie/crawler#usage) to do a few different things.

You can configure the crawler used by the sitemap generator, for example, to ignore robot checks:

```
use Spatie\Crawler\Crawler;
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->configureCrawler(function (Crawler $crawler) {
        $crawler->ignoreRobots();
    })
    ->writeToFile($sitemapPath);
```

Limiting the amount of pages crawled
--------------------------------------------------------------------------------------------------------------------------------------------------------------------

You can limit the amount of pages crawled by calling `setMaximumCrawlCount`:

```
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->setMaximumCrawlCount(500)
    ->writeToFile($sitemapPath);
```

Setting the crawl concurrency
-----------------------------------------------------------------------------------------------------------------------------------------------

You can set the number of concurrent connections the crawler will use:

```
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->setConcurrency(1)
    ->writeToFile($sitemapPath);
```

The default concurrency is 10.

Controlling the crawl depth
-----------------------------------------------------------------------------------------------------------------------------------------

You can limit how deep the crawler follows links:

```
use Spatie\Crawler\Crawler;
use Spatie\Sitemap\SitemapGenerator;

SitemapGenerator::create('https://example.com')
    ->configureCrawler(function (Crawler $crawler) {
        $crawler->depth(3);
    })
    ->writeToFile($path);
```

Executing JavaScript
--------------------------------------------------------------------------------------------------------------------

The sitemap generator can execute JavaScript on each page so it will discover links that are generated by your JS scripts. You can enable this feature by setting `execute_javascript` in the config file to `true`.

Under the hood, [headless Chrome](https://github.com/spatie/browsershot) is used to execute JavaScript. You'll need to install `spatie/browsershot` separately:

```
composer require spatie/browsershot
```

Here are some pointers on [how to install it on your system](https://spatie.be/docs/browsershot/v4/requirements).

The package will make an educated guess as to where Chrome is installed on your system. You can also set the path in `config/sitemap.php`.