wget
wget -r -l4 --spider -D example.com http://example.com
Option | Details |
---|---|
-r | Indicates it’s recursive, so “follow the links”. |
-l | Indicates the number of levels we want to recurse. If you are on the first page and you follow a link, you are at level 1. If you follow a link on that last page, you are at level 2, etc. |
–-spider | Indicates not to download anything (we just want to go through the pages, that’s all) |
-D | Indicates the list (separated by commas) of domains where we think it’s acceptable to “spider” (that is, if a link points to “hello.com”, we won’t follow it) |