How to Log Client IP AND X-Forwarded-For IP in Apache

When you use Load Balancer or any proxy in-front of you web servers, the visitor IP address is replaced with the load balancer or proxy IP. You will need to make the following changes in your apache config in order to retrieve the client IPs:

In your apache config (e.g. httpd.conf)

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
SetEnvIf X-Forwarded-For "^$" noforwarded

If you use a single log file, then change the CustomLog in httpd.conf:
CustomLog logs/access_log combined env=noforwarded
CustomLog logs/access_log proxy env=forwarded

If you are using separate log files for each domain, then you should change the CustomLog in your vhosts config:
CustomLog /log_location/domain-name.com-access_log combined env=noforwarded
CustomLog /log_location/domain-name.com-access_log proxy env=forwarded

How to find only files with specified last modified date?

To find all files in the current directory that have been modified since yesterday (24 hours ago) use:

find . -maxdepth 1 -mtime -1

How to set apache and mysql to run at server reboot

sudo chkconfig httpd on

sudo chkconfig mysqld on 

How to check if apache, mysql is configured to start at server reboot

chkconfig --list httpd
chkconfig --list mysqld

iptables

Add a rule
sudo iptables -A INPUT -p tcp -m state --state NEW,RELATED,ESTABLISHED -m tcp --dport 1234 -j ACCEPT 

Insert a rule to a specific line
sudo iptables -I INPUT 7 -p tcp -m state --state NEW,RELATED,ESTABLISHED -m tcp --dport 1234 -j ACCEPT

The above example inserts the rule before the line 7. If no line number is defined it will use the default value of 1 in which your rule is inserted at the topmost line. 

Delete a rule
-D INPUT -p tcp -m state --state NEW,RELATED,ESTABLISHED -m tcp --dport 1234 -j ACCEPT 

Saving rules
service iptables save

Restart service
service iptables restart

How to Unban IP manually in Fail2ban

fail2ban-client set [jail] unbanip [ip]

Helpful links when working with apache access logs

http://stackoverflow.com/questions/13326605/finding-blank-user-agents-and-spoofed-ua-in-access-logs

# Detect empty user-agent
$12 == "" {
  report="Empty user agent from " $1 "\n";
}

Syntax to block an IP address via iptables under Linux

CentOS / RHEL / Fedora Block An IP And Save It To Config File

# iptables -A INPUT -s IP_ADDRESS -j DROP   // (add the rule at the end)
# iptables -I INPUT -s IP_ADDRESS -j DROP   // (insert the rule at the beginning)
# service iptables save

How to make awstats process old log files

http://awstats.sourceforge.net/docs/awstats_faq.html#OLDLOG

/usr/lib/cgi-bin/awstats.pl -update -config="www.domain.com" -LogFile="/usr/local/awstats/tools/logresolvemerge.pl /var/log/httpd/access_log* |"

Light Squid - Squid log analyzer

Source: http://lightsquid.sourceforge.net/

Installation Instructions:

Required software:

1. Perl
2. http server (Apache, lighthttpd, etc)
3. Squid
4. Cron

or equivalent
in example i use Apache

1. Extract lightsquid.tgz into any folder

cd /var/www/htdocs/
mkdir lightsquid
cd lightsquid
tar -xzf lightsquid.tgz

set 'executable' flag for scrip

chmod +x *.cgi
chmod +x *.pl

change owner

chown -R apache:apache *
(where apache = httpd user)

2. Configure your Apache server

.cgi files must execute as CGI scrip

"/var/www/html/lightsquid">

AddHandler cgi-script .cgi AllowOverride All Restart apache

3. Edit lightsquid.cfg

4. If you want GROUP reports - edit group.cfg

format:

esl 01 SysAdmin
karlos 01 SysAdmin
thy 01 SysAdmin
Ivanov 02 Developer
Ivanov2 02 Developer
Petrov 03 Commerial
Petrova 03 Commerial
vasyav 04 room 312
petyava 04 room 312

5. If you want use REALNAME - edit realname.cfg

format:

esl Sergey Erokhin
karlos Super User1
thy Tech Good
Ivanov Developer numer one
Ivanov2 Developer numbe two
Petrov Good Commerial
Petrova Bad Commerial
vasyav Some User1
petyava Some User2

6. if you want graphics report

check graph report

7. run check-setup.pl

if all done try next step
else check config file.

8. Try run lightparser.pl

if everything is ok no messages will be produced
parse old statistic
./lightparser.pl access.log.1.{gz|bz2}
./lightparser.pl access.log.2.{gz|bz2}
./lightparser.pl access.log.3.{gz|bz2}
...

9. Try to use lightsquid

Using your favorite browser enter http:///lightsquid/

10. Setup crontab to run lightparser once per hour

crontab -e
This example will execute parser every 20 minutes
*/20 * * * * /var/www/htdocs/lightsquid/lightparser.pl today
if you have small log and fast machine, you may run lightparser with smaller delay
warning not set interval less 10 min

11. All done!

12. If you have problems, define $debug variable in lightsquid.cfg and rerun lightparser.

How to redirect entire domain to another domain

Add the line below in .htaccess of www.yourdomain.com/ :

Redirect 301 / http://www.anotherdomain.com/

Squid Proxy errors/warnings and solutions

"WARNING: Could not determine this machines public hostname. Please configure one or set 'visible_hostname'."  --- add the visible_hostname into the /etc/squid/squid.conf