I see a lot of ssh attempts in my auth.log. Here’s a little snippet from one of my servers:
Nov 18 21:55:49 mcremove sshd[15991]: reverse mapping checking getaddrinfo for 59.162.182.53.static.vsnl.net.in [59.162.182.53] failed - POSSIBLE BREAK-IN ATTEMPT! Nov 18 21:55:49 mcremove sshd[15991]: Invalid user business from 59.162.182.53 Nov 18 21:55:49 mcremove sshd[15991]: input_userauth_request: invalid user business [preauth] Nov 18 21:55:49 mcremove sshd[15991]: pam_unix(sshd:auth): check pass; user unknown Nov 18 21:55:49 mcremove sshd[15991]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=59.162.182.53 Nov 18 21:55:50 mcremove sshd[15989]: Failed password for invalid user kent from 125.93.51.27 port 46908 ssh2 Nov 18 21:55:50 mcremove sshd[15989]: Received disconnect from 125.93.51.27: 11: Bye Bye [preauth] Nov 18 21:55:51 mcremove sshd[15991]: Failed password for invalid user business from 59.162.182.53 port 41781 ssh2 Nov 18 21:55:51 mcremove sshd[15991]: Connection closed by 59.162.182.53 [preauth] Nov 18 21:59:22 mcremove sshd[15994]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=125.93.51.27 user=root Nov 18 21:59:25 mcremove sshd[15994]: Failed password for root from 125.93.51.27 port 40025 ssh2 Nov 18 21:59:25 mcremove sshd[15994]: Received disconnect from 125.93.51.27: 11: Bye Bye [preauth]
I haven’t totaled up how many I average, but I can assure you the attempts are constant.
Anyways, all these bots trying to get in via SSH made me curious. What passwords are they using? A little googling around led me to a pam-cifs module that logs passwords, and more specifically the libpam source code so you can compile it yourself!
This is a PAM module written by Florian Lohoff, and with a little bit of tweaking you can get it to log those password attempts.
First, I downloaded the tar.gz and extracted it. Everything is already in there all nice and ready for you to run make on it. However, you do need the PAM development files on your machine:
sudo apt-get install libpam0g-dev
I found this package from searching for debian packages: https://packages.debian.org/cgi-bin/search_contents.pl?word=pam_appl.h&searchmode=searchfiles&case=insensitive&version=stable&arch=i386
I believe if you’re on RHEL/using yum, it would be: yum install pam-devel
Anyways, this module is great and all but it needs some modifications to work for logging SSH. The exact modifications were posted on adeptus-mechanicus.com in this great post.
Here’s all the required files:
The modified pam_storepw.c
libpam-storepw_0.1.tar
After you run make, just add the new pam_storepw.so as /lib/security/pam_storepw.so
Final step, modify pam to load the new module! In the adeptus-mechanicus post, they modified /etc/pam.d/sshd. On my server though, I have an @include common-auth in my /etc/pam.d/sshd file, so I just modified my /etc/pam.d/common-auth file to look a little like this:
... ># here are the per-package modules (the "Primary" block) auth [success=2 default=ignore] pam_unix.so nullok_secure auth optional pam_storepw.so ...
Now all the failed password attempts are logged in /var/log/passwords
Hurray!
IMPORTANT!!! Change success=1 to success=2, let me explain why:
auth [success=1 default=ignore] pam_unix.so nullok_secure auth requisite pam_deny.so auth required pam_permit.so auth optional pam_cap.so
Without comments, this is what the common-auth file looks like. It’s important to explain the auth [success=1 default=ignore] pam_unix.so nullok_secure line.
This line is calling the “pam_unix” module, which checks the /etc/passwd and /etc/shadow files. I don’t have deep understanding of this, so seek information from the man pages or elsewhere online, but thats the basics of what it does.
The argument “nullok_secure” means what it says “no password is ok”, but ONLY from ttys that are listed in the /etc/securetty file. I believe that this only applies to root, but I’m not 100% sure on that. PS this doesn’t mean you can just leave the password blank for root and be able to login. From the manpage:
nullok_secure
The default action of this module is to not permit the user access
to a service if their official password is blank. The nullok_secure
argument overrides this default and allows any user with a blank
password to access the service as long as the value of PAM_TTY is
set to one of the values found in /etc/securetty.
Now we have the weird part: “[success=1 default=ignore]”.
If the module returns success, it skips “1” line. Every other return means its ignored and just moves down the PAM file like normal.
The next line is auth requisite pam_deny.so. This is important. If the previous line returns success, which means a successful login, it skips this line. If you don’t skip this line and it gets ran, this is an automatic login denial, EVEN IF THE PASSWORD WAS RIGHT.
Thus, we add our auth optional pam_storepw.so directly below the auth [success=1 default=ignore] pam_unix.so nullok_secure line, and change the value to skip 2 lines. This results on a SUCCESSFUL login skipping our storepw and the denial, and only failed login attempts getting logged and denied.
Caveat to this module, if the user doesn’t exist on your system you will get an entry similar to this:
host = 59.162.182.53 : username = css : password = INCORRECT
To actually log these, you will have to create users. I don’t need to recreate the wheel, and the solution to adding users posted on adeptus-mechanicus is pretty nice:
# cat /usr/sbin/add-honeypot useradd -c "honeypot user" -d /home/honeypot -g 2000 -m -o -s /bin/false -u 2000 $1
Change the path of the file as appropriate for your system, add a bunch of users, and watch the passwords roll in!
EDIT:
So I decided I wanted ALL the passwords, so heres a quick and easy way to add all those users in one go. First you want the usernames.
cat /var/log/auth.log | grep invalid.*preauth | cut -d\ -f 9 | sort | uniq > ~/honeyusers
You might need to tweak this a little bit according to your auth.log file.
The next thing I tried, but failed:
cat honeyusers | xargs add-honeypot
cat honeyusers | add-honeypot `awk "{ print $1 }"`
Both of those will add one user and then halt. I don’t really into bash, so here’s my python fix for it:
#!/usr/bin/python
import os,time
os.system('cat /var/log/passwords | grep -P "[\b]" | cut -d\ -f 7 | sort | uniq > /root/honeyusers')
time.sleep(2)
f = open('honeyusers').read().split('\n')
f.pop()
#The pop is to get rid of the last entry, which will just be empty
for username in f:
os.system('add-honeypot ' + username)
print('Added: ' + username)
This will cycle through and add all the users that don’t already exist. The grep -P "[\b]" identifies users that don’t exist yet, because for some reason the user line followed by INCORRECT is actually a backspace newline (0x08 0x0A).
Another little bit of code, this will clean up your /var/log/passwords and remove all the INCORRECT lines, as well as empty passwords above them:
sed -ri "/(\x08)|(INCORRECT)/d" /var/log/passwords
And finally, this line will move all the passwords you’ve collected into a file:
cat /var/log/passwords | cut -d\ -f 11 | sort | uniq > ~/collected_passwords
If you want to see the most frequently used passwords, try this:
cat /var/log/passwords | cut -d\ -f 11 | sort | uniq -c | sort -brn -o ~/collected_password_frequent
Heres a snippet after it was done:
subversion:x:2000:2000:honeypot user:/home/honeypot:/bin/false sun1:x:2000:2000:honeypot user:/home/honeypot:/bin/false sunos:x:2000:2000:honeypot user:/home/honeypot:/bin/false superior:x:2000:2000:honeypot user:/home/honeypot:/bin/false superman:x:2000:2000:honeypot user:/home/honeypot:/bin/false supermbox:x:2000:2000:honeypot user:/home/honeypot:/bin/false support:x:2000:2000:honeypot user:/home/honeypot:/bin/false support123:x:2000:2000:honeypot user:/home/honeypot:/bin/false suresh:x:2000:2000:honeypot user:/home/honeypot:/bin/false suzuki:x:2000:2000:honeypot user:/home/honeypot:/bin/false sven:x:2000:2000:honeypot user:/home/honeypot:/bin/false svenb:x:2000:2000:honeypot user:/home/honeypot:/bin/false svn:x:2000:2000:honeypot user:/home/honeypot:/bin/false svukovic:x:2000:2000:honeypot user:/home/honeypot:/bin/false swsoft:x:2000:2000:honeypot user:/home/honeypot:/bin/false sybase:x:2000:2000:honeypot user:/home/honeypot:/bin/false sylvie:x:2000:2000:honeypot user:/home/honeypot:/bin/false sysadmin:x:2000:2000:honeypot user:/home/honeypot:/bin/false sysop:x:2000:2000:honeypot user:/home/honeypot:/bin/false
I guess todo in the future would be to recompile the .so to place these logs into a /var/log/passwords/passwords folder, and add a rotate job into the weekly cron so I don’t wind up with huge files.