Posted: . At: 9:29 AM. This was 1 year ago. Post ID: 14982
Page permalink. WordPress uses cookies, or tiny pieces of information stored on your computer, to verify who you are. There are cookies for logged in users and for commenters.
These cookies expire two weeks after they are set.


Parsing the contents of the Apache web server logs is a heap of fun.

by

I was looking through my Apache server log for March today and I have found a nice way to parse the logs. This is a work in progress, but it works OK for now.

┌──[jason@192.168.1.2]─[~/Documents]
└──╼  ╼ $ sort -u | grep 'https://' securitronlinux.com-ssl_log-Mar-2021 | awk '{gsub("\"",""); print "\nURL: " $11 "\nFile: " $7 "\nTime: " $4}'

This is the output I got, this shows the URL visited and what file is actually visited by the website visitor.

jason@jason-Lenovo-H50-55:~/Desktop$ sort -u | grep 'https://' securitronlinux.com-ssl_log-Mar-2021 | awk '{if ( $11 ~ /https:\/\/*/)  print "\nServer: " $11 "\nURL: " $7 "\nTime: " $4}' | head -n 25
 
Server: "https://securitronlinux.com/bejiitaswrath/xvideo-service-thief-a-good-video-downloader-for-linux/"
URL: /arma3/servers.php
Time: [28/Feb/2021:06:15:58
 
Server: "https://securitronlinux.com/"
URL: /wp-content/uploads/2017/06/gnome-terminal-768x469.png
Time: [28/Feb/2021:06:16:03
 
Server: "https://www.securitronlinux.com/wp-cron.php?doing_wp_cron=1614514569.9472041130065917968750"
URL: /wp-cron.php?doing_wp_cron=1614514569.9472041130065917968750
Time: [28/Feb/2021:06:16:10
 
Server: "https://securitronlinux.com/"
URL: /wp-content/uploads/2017/06/gnome-terminal.png
Time: [28/Feb/2021:06:16:20
 
Server: "https://www.securitronlinux.com/wp-content/uploads/2017/06/gnome-terminal.png"
URL: /favicon.ico
Time: [28/Feb/2021:06:16:22
 
Server: "https://www.google.com/"
URL: /bejiitaswrath/crack-a-ssh-password-with-hydra-and-ways-to-avoid-this-in-future/
Time: [28/Feb/2021:06:17:15

This could be very useful for anyone who wants to get information from an Apache server log.

Another very good way to get comprehensive information about a site`s performance is the WebDruid 0.5.4 application. This can generate AWstats-styled reports about the amount of traffic a website is getting in a certain time period.

Install it like this.

┌──[jason@192.168.1.2]─[~/Desktop/Securitron]
└──╼  ╼ $ sudo apt install webdruid

Then run it against a log file in a new folder.

┌──[jason@192.168.1.2]─[~/Desktop/Securitron]
└──╼  ╼ $ webdruid securitronlinux.com-ssl_log-Mar-2021

This generates very nice reports of the traffic and users on your website for a certain time period. This is very useful if you are not using AWStats and want to see information on how your website is performing.

An example of the output that WebDruid can create.
An example of the output that WebDruid can create.

Parsing Apache logs is very useful, it is also possible to list all IP addresses and the count of each address.

╭──(john㉿DESKTOP-PF01IEE)-[/mnt/c/Users/Intel i5/Documents]
╰───────────────────────────╾┋ awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' securitronlinux.com-ssl_log

This one-liner will display a list of all IP addresses in the Apache logs and a number that is the number of times the IP address appears.

Also, count the number of times a certain keyword appears.

╭──(john㉿DESKTOP-PF01IEE)-[/mnt/c/Users/Intel i5/Documents]
╰───────────────────────────╾┋ grep -w -c "Ubuntu" securitronlinux.com-ssl_log
591

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.