Posted: . At: 8:14 AM. This was 8 years ago. Post ID: 9276
Page permalink. WordPress uses cookies, or tiny pieces of information stored on your computer, to verify who you are. There are cookies for logged in users and for commenters.
These cookies expire two weeks after they are set.

Get a listing of all the urls on a website with this simple Linux command.

This command will return a huge listing of all the visitable url`s on the http://www.google.com.au website. Give this a shot on other websites and see how you go.

jason@jason-desktop:~$ wget --spider --force-html -r -l2 http://www.google.com.au 2>&1   | grep '^--' | awk '{ print $3 }'

Very useful if someone wants to know what is actually on a certain website. This might take a bit of time to return the listing, but this does work very well when spidering a website. Below is sample output.

jason@jason-desktop:~$ wget --spider --force-html -r -l2 http://www.securitronlinux.com/linux/ 2>&1   | grep '^--' | awk '{ print $3 }'
http://www.securitronlinux.com/linux/
http://www.securitronlinux.com/linux/
http://www.securitronlinux.com/robots.txt
http://www.securitronlinux.com/linux/style.css
http://www.securitronlinux.com/linux/favicon.ico
http://www.securitronlinux.com/linux/maps/back.png
http://www.securitronlinux.com/index.php
http://www.securitronlinux.com/
http://www.securitronlinux.com/
http://www.securitronlinux.com/linux/caring_computer.php
http://www.securitronlinux.com/linux/caring_computer.php
http://www.securitronlinux.com/linux/fedora.php
http://www.securitronlinux.com/linux/fedora.php
http://www.securitronlinux.com/linux/security.php
http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/
http://www.securitronlinux.com/debian-testing/securing-your-gnulinux-system/
http://www.securitronlinux.com/linux/susedvd.php
http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/
http://www.securitronlinux.com/playing-dvd-movies-on-suse-linux-how-to-install-codecs-and-play-your-movies/
http://www.securitronlinux.com/linux/doom_files.php
http://www.securitronlinux.com/linux/doom_files.php
http://www.securitronlinux.com/linux/doom_wadfiles.php
http://www.securitronlinux.com/linux/doom_wadfiles.php
http://www.securitronlinux.com/linux/console_codes.php
http://www.securitronlinux.com/linux/console_codes.php
http://www.securitronlinux.com/linux/misc_codes.php
http://www.securitronlinux.com/linux/misc_codes.php
http://www.securitronlinux.com/linux/psx_doom.php
http://www.securitronlinux.com/linux/psx_doom.php
http://www.securitronlinux.com/linux/zdoom_acs.php
http://www.securitronlinux.com/linux/windows_3.0.php
http://www.securitronlinux.com/linux/windows_3.0.php
http://www.securitronlinux.com/linux/ubuntu-karmic.php
http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/
http://www.securitronlinux.com/more-useful-ubuntu-and-linux-mint-tips-and-tricks-for-the-desktop-user/
http://www.securitronlinux.com/linux/windows_7.php
http://www.securitronlinux.com/linux/windows_7.php
http://www.securitronlinux.com/linux/johnjohnrace.swf
http://www.securitronlinux.com/linux/doomshots/
http://www.securitronlinux.com/linux/doomshots/
http://www.securitronlinux.com/xmlrpc.php
http://www.securitronlinux.com/wp-content/themes/securitron/style.css
http://www.securitronlinux.com/page/2/
http://www.securitronlinux.com/page/2/
http://www.securitronlinux.com/feed/
http://www.securitronlinux.com/comments/feed/
http://www.securitronlinux.com/wp-includes/js/jquery/jquery.js?ver=1.12.3
http://www.securitronlinux.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.0
http://www.securitronlinux.com/wp-json/
http://www.securitronlinux.com/xmlrpc.php?rsd

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.