Preventing content from being indexed | laravel-site-search | Spatie

 SPATIE

  Laravel Site Search
======================

spatie.be/open-source

  [Docs](https://spatie.be/docs)  [Laravel-site-search](https://spatie.be/docs/laravel-site-search/v1)  Basic-usage  Preventing content from being indexed

 Version   v3   v1

 Other versions for crawler [v3](https://spatie.be/docs/laravel-site-search/v3) [v1](https://spatie.be/docs/laravel-site-search/v1)

- [ Introduction ](https://spatie.be/docs/laravel-site-search/v1/introduction)
- [ Support us ](https://spatie.be/docs/laravel-site-search/v1/support-us)
- [ Installation and setup ](https://spatie.be/docs/laravel-site-search/v1/installation-setup)
- [ About us ](https://spatie.be/docs/laravel-site-search/v1/about-us)
- [ Questions and issues ](https://spatie.be/docs/laravel-site-search/v1/questions-issues)
- [ Requirements ](https://spatie.be/docs/laravel-site-search/v1/requirements)
- [ Changelog ](https://spatie.be/docs/laravel-site-search/v1/changelog)

Basic usage
-----------

- [ High level overview ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/high-level-overview)
- [ Indexing your first site ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/indexing-your-first-site)
- [ Retrieving results ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/retrieving-results)
- [ Preventing content from being indexed ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/preventing-content-from-being-indexed)
- [ Using a search profile ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/using-a-search-profile)
- [ Authenticating requests ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/authenticating-requests)
- [ Listing indexes ](https://spatie.be/docs/laravel-site-search/v1/basic-usage/listing-indexes)

Advanced usage
--------------

- [ Creating multiple search indexes ](https://spatie.be/docs/laravel-site-search/v1/advanced-usage/creating-multiple-search-indexes)
- [ Using a custom indexer ](https://spatie.be/docs/laravel-site-search/v1/advanced-usage/using-a-custom-indexer)
- [ Indexing extra properties ](https://spatie.be/docs/laravel-site-search/v1/advanced-usage/indexing-extra-properties)
- [ Customizing Meilisearch settings ](https://spatie.be/docs/laravel-site-search/v1/advanced-usage/customizing-meilisearch-settings)
- [ Available events ](https://spatie.be/docs/laravel-site-search/v1/advanced-usage/available-events)

      You are viewing the documentation for **an older version** of this package. You can check the version you are using with the following command:

 `                                    composer show spatie/laravel-site-search                                                                                                                                                                                                                                    `

Preventing content from being indexed
=====================================

###  On this page

1. [ Using CSS selectors ](#content-using-css-selectors)
2. [ Using the config file ](#content-using-the-config-file)
3. [ Using headers ](#content-using-headers)
4. [ Using a search profile ](#content-using-a-search-profile)

Your site probably displays a lot of information that should not be indexed, such as your menu structure or footer. Or maybe entire pages do not need to be indexed.

Using CSS selectors
-----------------------------------------------------------------------------------------------------------------

In the `ignore_content_by_css_selector` key of the `site-search` config file, you can specify CSS selectors of elements that should not be indexed. By default, the content of a `nav` element will not be put in the index (but all URLs inside it will still be crawled).

Additionally, any elements with a `data-no-index` will not get indexed. In the following example, the sentence "This will not be indexed", will not be indexed.

```

        This is my page

            This is the content of the nav. It should not be indexed.

        This is the H1

        This is the content

            This will not be indexed.

```

Using the config file
-----------------------------------------------------------------------------------------------------------------------

In the `ignore_content_on_urls` key of the `site-search` config file, you may specify URLs whose content should not be indexed. All links on these URLs will still be followed and crawled.

Using headers
-----------------------------------------------------------------------------------------------

If your site's response contains a header whose name is in the `do_not_index_content_headers` of the `site-search` config file, then that page will not be indexed.

Using a search profile
--------------------------------------------------------------------------------------------------------------------------

A search profile is a class that determines what pages get crawled and what content gets indexed. Learn more about search profiles [here](/docs/laravel-site-search/v1/basic-usage/using-a-search-profile).