Very Useful Commands For Apache Logs

Apache server records all incoming requests and all requests processed to a log file. The format of the access log is highly configurable. The location and content of the access log are controlled by the CustomLog directive. 

Default apache access log file location:

RHEL / Red Hat / CentOS / Fedora Linux Apache access file location - /var/log/httpd/access_log

Debian / Ubuntu Linux Apache access log file location - /var/log/apache2/access.log

FreeBSD Apache access log file location - /var/log/httpd-access.log
 

To find exact apache log file location, you can use grep command:

# grep CustomLog /usr/local/etc/apache22/httpd.conf

# grep CustomLog /etc/apache2/apache2.conf

# grep CustomLog /etc/httpd/conf/httpd.conf
 

Sample output:

# a CustomLog directive (see below).

# CustomLog "/var/log/httpd-access.log" common

   CustomLog "/var/log/httpd-access.log" combined
 

These assume you use a standard log format. If not you may have to adjust them to suit your format.

Most viewed pages (top ten).

Code:

awk '{print $7}' /path/to/log |sort |uniq -c |sort -rn |head -10
 

Top ten referrers:

Code:

awk '{print $11}' /path/to/log |sort |uniq -c |sort -rn |head -10
 

Search logs:

Code:

grep /path/to/log query |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]

And here is a script I wrote a long time ago that incorparates these and few others.

Code:

#!/bin/bash

# usage 

#    ./this_script search pattern log_file

#       all hosts with 5 or more matches of the given pattern will be banned

#    ./this_script ban_from_log log_file

#       all hosts that appear more than 5 times in the given log file will be banned

#    ./this_script ban_rfi log_file

#       bans all hosts that match the rfi pattern(rfi's and proxy requests)

#    ./this_script most_viewed log_file

#       shows the top ten viewed pages with the number of views

#    ./this_script statuses response log_file

#       shows the top ten viewed pages for the given response header e.g. 404, 200 ect

#    ./this_script referrers log_file 

#       shows the top ten referrers and page views for each

#

# author tomfmason

ban_file=/etc/hosts.deny
 

function ban_ip() {

  exists=`grep ${1} $ban_file`

  if [ ! "$exists" ]; then

     echo "ALL: ${1}" >> $ban_file

  fi

}
 

function search() {

  ret=`grep ${1} ${2} |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]`

  for r in $ret; do

      ban_ip $r

  done

}
 

function ban_from_log() {

  ret=`awk '{print $8}' ${1}|tail -n+5|sort|uniq|tr -d [1]`

  for r in $ret; do

     ban_ip $r

  done

}
 

function ban_rfi() {

  ret=`awk '{print $1 " " $7}' ${1} |grep -iE '(http|https|ftp)'|awk '{print $1}'|tail -n+5|sort|uniq`

  for r in $ret; do

    ban_ip $r

  done

}
 

function most_viewed() {

    awk '{print $7}' ${1} |sort |uniq -c |sort -rn |head -10

}
 

function statuses() {

   awk '{print $7 " " $9}' ${2} |grep -iE '${1}' |sort | uniq -c | sort -rn |awk '{print $1 "  " $2 }' |head -10

}
 
 

function referrers() {

    awk '{print $11}' ${1} |sort |uniq -c |sort -rn |head -10

}
 

if type "$1" | grep -qF "$1 is a function"; then "$@"; fi

Note: 

As this was written a long time ago and now I would not suggest using hosts.deny to ban an ip. Iptables would be a much better choice. Believe you can fix it yourself ;-)

Linux Tips

See Also
Examples Of Netfilter Port Forwarding

Have a Linux Problem
Linux Forum - Do you have a Linux Question?

Linux Books
Linux Certification, System Administration, Programming, Networking Books

Linux Home: Linux System Administration Hints and Tips

(c) www.gotothings.com All material on this site is Copyright.
Every effort is made to ensure the content integrity.  Information used on this site is at your own risk.
All product names are trademarks of their respective companies.
The site www.gotothings.com is in no way affiliated with or endorsed by any company listed at this site.
Any unauthorised copying or mirroring is prohibited.