Crawling URLs | laravel-url-ai-transformer | Spatie

 SPATIE

  Laravel URL AI Transformer
=============================

spatie.be/open-source

  [Docs](https://spatie.be/docs)  [Laravel-url-ai-transformer](https://spatie.be/docs/laravel-url-ai-transformer/v1)  Advanced-usage  Crawling URLs

 Version   v1

 Other versions for crawler [v1](https://spatie.be/docs/laravel-url-ai-transformer/v1)

  Crawling URLs
- [ Introduction ](https://spatie.be/docs/laravel-url-ai-transformer/v1/introduction)
- [ Support us ](https://spatie.be/docs/laravel-url-ai-transformer/v1/support-us)
- [ Installation & setup ](https://spatie.be/docs/laravel-url-ai-transformer/v1/installation-setup)
- [ Questions and issues ](https://spatie.be/docs/laravel-url-ai-transformer/v1/questions-issues)
- [ Changelog ](https://spatie.be/docs/laravel-url-ai-transformer/v1/changelog)
- [ About us ](https://spatie.be/docs/laravel-url-ai-transformer/v1/about-us)

Basic usage
-----------

- [ Getting started ](https://spatie.be/docs/laravel-url-ai-transformer/v1/basic-usage/getting-started)
- [ Registering transformations ](https://spatie.be/docs/laravel-url-ai-transformer/v1/basic-usage/registering-transformations)
- [ Writing your own transformers ](https://spatie.be/docs/laravel-url-ai-transformer/v1/basic-usage/writing-your-own-transformers)

Advanced usage
--------------

- [ Testing transformers ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/testing-transformers)
- [ Conditional transformations ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/conditional-transformations)
- [ Crawling URLs ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/crawling-urls)
- [ Customizing AI models ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/customizing-ai-models)
- [ Exploring command options ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/exploring-command-options)
- [ Overriding actions ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/overriding-actions)
- [ Regenerating results ](https://spatie.be/docs/laravel-url-ai-transformer/v1/advanced-usage/regenerating-results)

 Crawling URLs
=============

Instead of manually registering URLs, you can use the [spatie/crawler](https://github.com/spatie/crawler) package to crawl (parts of) your website.

Here's an example of how to crawl all internal URLs of your website and pass them to the `Transform` class. Notice that we pass a closure to the `urls` method. By using a closure, the crawling will only happen when we actually perform the transformation, and not on each request.

```
namespace App\Providers;

use Illuminate\Support\ServiceProvider;
use Spatie\Crawler\Crawler;
use Spatie\LaravelUrlAiTransformer\Support\Transform;
use Spatie\LaravelUrlAiTransformer\Transformers\LdJsonTransformer;

class AiTransformerServiceProvider extends ServiceProvider
{
    public function register(): void
    {
        Transform::urls(
            fn() => $this->crawlAllUrls()
        )->usingTransformers(new LdJsonTransformer());
    }

    protected function crawlAllUrls(): array
    {
        return Crawler::create(url('/'))
            ->internalOnly()
            ->foundUrls();
    }
}
```

For more information on how to use the `Crawler` class, check out [the spatie/crawler docs](https://github.com/spatie/crawler).

 A good
match?
-------------

### What we do best

- All things Laravel
- Custom frontend components
- Building APIs
- AI-powered features
- Simplifying things
- Clean solutions
- Integrating services

### Not our cup of tea

- WordPress themes
- Cutting corners
- Free mockups to win a job
- "Just execute the briefing"

 In short: we'd like to be a **substantial part** of your project.

 [ Get in touch via email ](mailto:info@spatie.be?subject=A%20good%20match%21&body=Tell%20us%20as%20much%20as%20you%20can%20about%0A-%20your%20online%20project%0A-%20your%20planning%0A-%20your%20budget%0A-%20%E2%80%A6%0A%0AAnything%20that%20helps%20us%20to%20start%20straightforward%21)