jrobinsonc
4/25/2017 - 9:47 PM

wget

wget

wget

wget -r -l4 --spider -D example.com http://example.com

Useful options

OptionDetails
-rIndicates it’s recursive, so “follow the links”.
-lIndicates the number of levels we want to recurse. If you are on the first page and you follow a link, you are at level 1. If you follow a link on that last page, you are at level 2, etc.
–-spiderIndicates not to download anything (we just want to go through the pages, that’s all)
-DIndicates the list (separated by commas) of domains where we think it’s acceptable to “spider” (that is, if a link points to “hello.com”, we won’t follow it)