This command will return a huge listing of all the visitable url`s on the http://www.google.com.au website. Give this a shot on other websites and see how you go.
jason@jason-desktop:~$ wget --spider --force-html -r -l2 http://www.google.com.au 2>&1 | grep '^--' | awk '{ print $3 }' |
Very useful if someone wants to know what is actually on a certain website. This might take a bit of time to return the listing, but this does work very well when spidering a website. Below is sample output.
jason@jason-desktop:~$ wget --spider --force-html -r -l2 http://www.securitronlinux.com/linux/ 2>&1 | grep '^--' | awk '{ print $3 }' http://www.securitronlinux.com/linux/ http://www.securitronlinux.com/linux/ http://www.securitronlinux.com/robots.txt http://www.securitronlinux.com/linux/style.css http://www.securitronlinux.com/linux/favicon.ico http://www.securitronlinux.com/linux/maps/back.png http://www.securitronlinux.com/index.php http://www.securitronlinux.com/ http://www.securitronlinux.com/ http://www.securitronlinux.com/linux/caring_computer.php http://www.securitronlinux.com/linux/caring_computer.php http://www.securitronlinux.com/linux/fedora.php http://www.securitronlinux.com/linux/fedora.php http://www.securitronlinux.com/linux/security.php http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/ http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/ http://www.securitronlinux.com/linux/susedvd.php http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/ http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/ http://www.securitronlinux.com/linux/doom_files.php http://www.securitronlinux.com/linux/doom_files.php http://www.securitronlinux.com/linux/doom_wadfiles.php http://www.securitronlinux.com/linux/doom_wadfiles.php http://www.securitronlinux.com/linux/console_codes.php http://www.securitronlinux.com/linux/console_codes.php http://www.securitronlinux.com/linux/misc_codes.php http://www.securitronlinux.com/linux/misc_codes.php http://www.securitronlinux.com/linux/psx_doom.php http://www.securitronlinux.com/linux/psx_doom.php http://www.securitronlinux.com/linux/zdoom_acs.php http://www.securitronlinux.com/linux/windows_3.0.php http://www.securitronlinux.com/linux/windows_3.0.php http://www.securitronlinux.com/linux/ubuntu-karmic.php http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/ http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/ http://www.securitronlinux.com/linux/windows_7.php http://www.securitronlinux.com/linux/windows_7.php http://www.securitronlinux.com/linux/johnjohnrace.swf http://www.securitronlinux.com/linux/doomshots/ http://www.securitronlinux.com/linux/doomshots/ http://www.securitronlinux.com/xmlrpc.php http://www.securitronlinux.com/wp-content/themes/securitron/style.css http://www.securitronlinux.com/page/2/ http://www.securitronlinux.com/page/2/ http://www.securitronlinux.com/feed/ http://www.securitronlinux.com/comments/feed/ http://www.securitronlinux.com/wp-includes/js/jquery/jquery.js?ver=1.12.3 http://www.securitronlinux.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.0 http://www.securitronlinux.com/wp-json/ http://www.securitronlinux.com/xmlrpc.php?rsd |