Site Crawler
Unlighthouse comes with multiple methods for URL discovery in the form of crawling.
- Add the specified
site
from--site
or config - Manually providing URLs via the
--urls
flag orurls
on the provider. robotsTxt
- Reading robots.txt, if it exists. Provides sitemap URLs and disallowed paths.sitemap
- Reading sitemap.xml, if it existscrawler
- Inspecting internal links- Using provided static route definitions
Robots.txt
When a robots.txt is found, it will attempt to read the sitemap and disallowed paths.
Disabling robots
You may not want to use the robots.txt in all occasions. For example if you want to scan URLs which are disallowed.
export default {
scanner: {
// disable robots.txt scanning
robotsTxt: false
}
}
Sitemap.xml
By default, the sitemap config will be read from your /robots.txt
. Otherwise, it will fall back to using /sitemap.xml
.
Note: When a sitemap exists with over 50 paths, it will disable the crawler.
Manual sitemap paths
You may provide an array of sitemap paths to scan.
export default {
scanner: {
sitemap: [
'/sitemap.xml',
'/sitemap2.xml'
]
}
}
Disabling scan
If you know your site doesn't have a sitemap, it may make sense to disable it.
export default {
scanner: {
// disable sitemap scanning
sitemap: false
}
}
Crawler
When enabled, the crawler will inspect the HTML payload of a page and extract internal links. These internal links will be queued up and scanned if they haven't already been scanned.
Disable crawling
If you have many pages with many internal links, it may be a good idea to disable the crawling.
export default {
scanner: {
crawler: false
}
}
Manually Providing URLs
While not recommended for most use cases, you may provide relative URLs within your configuration file, or use the --urls
flag.
This will disable the crawler and sitemap scanning.
Can be provided statically.
export default {
urls: [
'/about',
'/other-page'
],
}
Or you can return a function or promise.
export default {
urls: async () => await getUrls()
}
Specify explicit relative URLs as a comma-seperated list.
unlighthouse --site https://example.com --urls /about,/other-page