This is another "one off" technique I've employed with success to mitigate brute-force password attacks.

If there are scads of "authentication failure" entries in /var/log/messages this technique may help. To determine the number of brute-force password attacks, run the following one-liner:

 fgrep "authentication failure" /var/log/messages* |cut -f 7 -d '=' |cut -f 1 -d ' '  |sort |uniq -c|sort -n

Output may look something like the following (truncated to the highest 10):

   1668 206.71.247.210
   1815 venice.directrouter.com
   1891 68.178.11.202
   3475 218.91.236.93
   3750 72.243.208.86
   4888 222.66.20.187
   5025 60.217.224.15
   5780 64.128.201.198
  13482 193.164.155.25
  16475 71.6.193.197

About PAM_ABL

PAM module for auto blacklisting. Provides auto blacklisting of hosts and users responsible for repeated failed authentication attempts. Generally configured so that blacklisted users still see normal login prompts but are guaranteed to fail to authenticate.

A command line tool allows to query or purge the databases used by the pam_abl module.

source: Dag Wieers :: pam_abl RPM packages for Red Hat

Installing pam_abl

1) Download the applicable RPM from Dag Wieers :: pam_abl RPM packages for Red Hat

Ex: RHEL 4

  wget http://dag.wieers.com/rpm/packages/pam_abl/pam_abl-0.2.3-1.el4.rf.i386.rpm

Test, solve any dependencies and install

 rpm -Uvh pam_abl-0.2.3-1.el4.rf.i386.rpm --test

 rpm -Uvh pam_abl-0.2.3-1.el4.rf.i386.rpm

Configuring pam_abl and adding to your pam stack

Read the Configuration documentation:

 vi /usr/share/doc/pam_abl-0.2.3/CONFIGURATION

Edit the /etc/security/pam_abl.conf file to avoid blocking root, and any $PRIMARY_USERs (replace $PRIMARY_USER with the system's "primary user" name for the target server:

 user_rule=!root|!$PRIMARY_USER:10/1h,30/2d

Honestly, blocking users is not useful, in my opinion. (This is why I don't recommend pam_talley, btw.) In practice I find myself more and more commenting out the "users" DB configuration in pam_abl.conf and just rely on hosts blocking.

You also may want to "whitelist" certain hosts. Use the host_rule line for this purpose. Since I have physical access to my servers (even if it is via a phone call) I'm less concerned about this than I should be, frankly.

Add the call to pam_abl to /etc/pam.d/system-auth:

 auth required pam_abl.so config=/etc/security/pam_abl.conf

My /etc/pam.d/system-auth looks like:

 #%PAM-1.0
 # This file is auto-generated.
 # User changes will be destroyed the next time authconfig is run.
 auth required /lib/security/$ISA/pam_env.so
 auth required pam_abl.so config=/etc/security/pam_abl.conf
 auth sufficient /lib/security/$ISA/pam_unix.so likeauth nullok
 auth sufficient pam_plesk.so try_first_pass
 auth        required      /lib/security/$ISA/pam_deny.so
 
 account     required      /lib/security/$ISA/pam_unix.so
 
 password required /lib/security/$ISA/pam_cracklib.so retry=3
 password optional pam_plesk.so try_first_pass
 password    sufficient    /lib/security/$ISA/pam_unix.so nullok use_authtok md5
 password    required      /lib/security/$ISA/pam_deny.so
 
 session     required      /lib/security/$ISA/pam_limits.so
 session     required      /lib/security/$ISA/pam_unix.so

Check the status of the module with the command:

 pam_abl

This will show stored activity and the state of each tracked username and IP.

To whitelist a user (where USER is the username to whitelist):

 pam_abl --okuser=USER

To whitelist a host (where HOST is the hostname to whitelist):

 pam_abl --okhost=HOST

This works well for several servers I've installed it on. I use this with Aggressive Spam and Zombie blocking via spamhaus.org/drop and IPTables to keep out the big bad wolves. Hope this helps!