Resources :: Sitemap Generator

This utility looks at a domain and generates a sitemap based on what it finds as it recursively spiders urls linked off of the home page. The sitemaps that are generated are suitable for placing on the website of the domain spidered or submitting to search engines for page submission. If you can think of ways to improve on this tool let me know.

start url	http:// (home page)
user-agent	(browser to cloak yourself as)
limit-rate	(download speed to spider server as)
limit-retry	(number of times to retry a failed connection)
timeout	(number of seconds before delayed actions timeout)
quota	(maximum website quota allowed for download)
accept files	(comma separated list of files to accept)
reject files	(comma separated list of files to reject)
accept directories	(comma separated list of directories to accept)
reject directories	(comma separated list of directories to reject)
save source	(whether to make tar collection or not)

Notes 1. URI files without extensions are downloaded by default. 2. To download all discovered URI files leave the reject list empty and put only * in the accept list.

printed @ tobymiller.com