Welcome to my Linux security crash course designed for Liberty University’s GenCyber Summer Camp. It explains how to get a linux system secure enough to survive a day of Red Team attacks.
No human could possibly tell you all there is to know about linux. I don’t even remember half of what I learn about it, and I only know probably 5% of what there is to know about it.
man command will give you a very technical user’s manual for a given command. This manual will be fairly technical, and if you haven’t used linux very long, will make very little sense. But you will be able to catch bits and pieces, and the more you read them, the more useful they become. Searching using the / key is very helpful with man pages. And remember to quit with q.When securing a system, you need to identify two things: the value of the thing you are securing, and the threats against which you are securing it.
Security is an economics game.
Any time you attempt to achieve something, you want to have a structured process that can get you to your objective. For us, we want to start at the base level of our system, and work outwards. We establish a secure zone, and then we expand the borders of that secure zone. Below is the outline of that process:
Before you try to secure an operating system you will need to remove any malware that could track or thwart your ability to do anything on your server. Finding and removing malware is an entire industry on its own. However, in our case, it appears that only two of the linux servers have a back door (according to the OpenVAS scan).
In order to find and remove these backdoors from the inside:
netstat -tulpn | grep 1337tcp 0 0 0.0.0.0:1337 0.0.0.0:* LISTEN 3652/ncps -eFH lists processes running on the system in a tree form – showing which processes own which other processes.ps -eFH | grep -B5 <the process id of nc> will show us the line of the nc process that is actually listening on the port, as well as 5 lines before it, enabling us to see the tree leading to that process./usr/sbin/backdoor, which was ultimately run by /USR/SBIN/CRON
root 16142 2520 0 14614 1556 0 20:44 ? 00:00:00 /USR/SBIN/CRON
rose 16143 16142 0 1047 588 0 20:44 ? 00:00:00 /bin/sh -c /usr/sbin/backdoor
rose 16144 16143 0 2690 1388 0 20:44 ? 00:00:00 /bin/bash /usr/sbin/backdoor
rose 16149 16144 0 1549 724 0 20:44 ? 00:00:00 nc -l -p 1337 -e /bin/bash
rose, we can guess that it is rose’s crontab that is scheduling the backdoor (If you want to learn more about cron or scripting, look at the resources section).crontab -u rose -e* * * * * /usr/sbin/backdoor tells the scheduler to run the program /usr/sbin/backdoor every minute.crontab -u rose -lkill -s SIGKILL <the process id goes here>rm /usr/sbin/backdoornetstat -tulpn | grep 1337 ( should give no output )cat /usr/sbin/backdoor (should say “no file or directory”)As a fun aside, here’s an explanation of the backdoor:
## It is a script:
#!/bin/bash
## This is the command it uses to check that the script is running:
isOn=`netstat -tln | grep ':1337'| wc -l`
## The if statement checks the result of the isOn command -- if it is zero, then
if [ $isOn == "0" ]
then
## It starts a netcat command that enables data sent to the 1337 port to give access to bash (the command line)
nc -l -p 1337 -e /bin/bash
fi
Another note: with this backdoor, we could simply remove the offending file from /usr/sbin and killed the process. Then the scheduled task would simply stop working because it wouldn’t have a program to execute. However, other backdoors could be more complicated (for example, if another scheduled task regenerated the backdoor in the location where you deleted it). So backdoors have to be treated with a lot of investigation and care.
In a corporate environment, you would possibly have a good chat with Rose. You may even give her a kind invitation to no longer work in your organization. Or perhaps, invite some helpful Law Enforcement Officers to have a conversation with her. ;)
Once we’ve cleaned up the operating system, we can move on to the server itself.
This will be the MOST IMPORTANT thing you do to secure your servers. Make sure you do it well. Most of the teams in last year’s camp got hacked very quickly by red team because they hadn’t changed all of their operating system user account passwords. With Red Team’s skill, it only takes one poorly secured user account for them to get in and hose the server.
I’ve developed a strategy for generating secure passwords that are easy to remember. The ideas come from this xkcd cartoon and a fellow LU student who mentioned using the Oxford Dictionary website to generate unusual words. A full explanation of what makes this strategy effective is beyond the scope of this lesson, but the idea is that long passwords are hard to guess, and passwords made up of combinations of unusual words (unlikely to be found in normal hacker dictionaries).
My strategy is roughly like this:
Wikipedia works well for this strategy because it has articles about many diverse subjects with strange words. Biological and scientific terms, place names or terms from foreign languages, and technical terms from esoteric subjects, all combine to make randomized wikipedia a rich ecosystem of linguistic entropy.
However, any site that generates random long words, or even a site that generates passwords made up of multiple words (like the xkcd password generator), can produce a similar effect. You may not get the diversity of wikipedia, but really, the key is to make things long without making them excessively complex.
I have a script that helps me get just the titles from Wikipedia’s random page repeatedly. To use this script, do the following:
wget https://raw.githubusercontent.com/slackboxster/slackboxster.github.io/master/gencyber2017/scripts/wikipwgen.shchmod +x wikipwgen.sh./wikipwgen.shCtrl+C to stop the script./etc/passwd
cat /etc/passwd/etc/passwd – if that is less than 1000, it is a system account./etc/login.defs file – look for UID_MIN 1000).toor account is very easily overlooked. It is an alias to the superuser account root, so it is incredibly important to make sure it is secured.The toor user is unnecessary and poses significant security risks.
deluser toor should remove the toor user. However, the toor user is running a process - proftpd specifically. Thus we will address removing the toor user when we work on securing ftp.
This is the most important step. If you forget to change a password, it is likely that Red Team will be logged into that account within 5 minutes. Maybe an hour because of how many servers they are attacking. In other words, one unchanged password and you will not stand a chance.
Use the printouts to set the passwords for your system. Ordinarily we would not use the same password twice ever. However, we will strategically reuse passwords to save time and simplify things. But we won’t use all the same passwords so that Red Team doesn’t get a whole lot of benefit from discovering one password.
Your root user password should not be reused anywhere. Reuse the same password for all your normal user accounts. And then use a different password for anything else on the server.
passwdpasswd <username>If you do the password change, this is a less significant vulnerability, but it could still have an impact if an attacker gets into a different service on the machine.
members sudo to find which users are in the sudo group
sudo apt-get install membersdeluser user group – replace user with the username, group with the group (in this case sudo)deluser user – that would delete the user.rose from the sudo group:
deluser rose sudoOnce you have secured your user accounts, the next thing that Red Team will want to attack is vulnerabilities that give access without having a valid user account. They will use a database of known vulnerabilities, so the best way to secure against this is to update your software. The moment a Red Team member detects out of date software, they are only two clicks away from hosing your server.
Updating software in Debian is usually really simple:
sudo apt-get update; sudo apt-get upgradeThere are a number of services that don’t need to be running on your server. You do want to keep the scored services running, and you want to keep services that support the scored services running. Also keep ssh, otherwise you can’t log in.
netstat -tulpnFor each service, remove it by uninstalling the associated package with apt-get remove. Removing the package should also stop the service.
Also, you could specify multiple packages in the remove command, for example: apt-get remove cups samba nfs-common rpcbind
To test if a service is needed, stop it and check your nagios: service <service> stop.
Remove at least the following services:
apt-get remove cups (this actually disables cups, even though it isn’t installed using cups.service cups start and check netstat.rm /etc/init.d/cups, then remove the package.update-inetd --disable swat (verify with cat /etc/inetd.conf)apt-get remove sambaapt-get remove nfs-commonapt-get remove rpcbindapt-get remove avahi-daemonapt-get remove exim4 exim4-base (note that there are two packages to remove – exim4-base is the one that removes the service)apt-get remove minissdpdOnce you’ve removed those, run netstat -tulpn again. Here is an example output:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 3050/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3706/sshd
tcp6 0 0 :::80 :::* LISTEN 2411/apache2
tcp6 0 0 :::21 :::* LISTEN 3778/proftpd: (acce
tcp6 0 0 :::22 :::* LISTEN 3706/sshd
udp 0 0 0.0.0.0:68 0.0.0.0:* 2365/dhclient
udp 0 0 0.0.0.0:64836 0.0.0.0:* 2365/dhclient
udp6 0 0 :::54657 :::* 2365/dhclient
When you are done:
On Jenkins, before doing autoremove, prevent java-common from being autoremoved: apt-get install java-common
apt-get autoremove to remove remaining unnecessary packages.
Interesting tidbit: I got most of my information for this section from googling “How to Secure
We need to disable anonymous access. But it also appears that the toor user has manually installed and configured a less secure ftp server, so we’ll have to remove that first.
nmap -p 21 -v -oN results.txt --open --script ftp-anon 192.168.210.54 (make sure to change the IP Address!!) PORT STATE SERVICE
21/tcp open ftp
| ftp-anon: Anonymous FTP login allowed (FTP code 230)
| drwxr-xr-x 2 root root 4096 May 18 11:35 backups
| drwxr-xr-x 16 root root 4096 Jun 27 02:42 cache
| -rw-r--r-- 1 root root 36 May 16 15:01 checkfile
| drwxr-xr-x 2 root root 4096 May 15 19:09 games
| drwxr-xr-x 55 root root 4096 Jun 26 16:27 lib
| drwxrwsr-x 2 root staff 4096 May 30 2016 local
| lrwxrwxrwx 1 root root 9 May 15 18:57 lock -> /run/lock
| drwxr-xr-x 16 root root 4096 Jun 27 02:42 log
| drwxrwsr-x 2 root mail 4096 Jun 27 01:50 mail
| drwxr-xr-x 2 root root 4096 May 15 18:57 opt
| lrwxrwxrwx 1 root root 4 May 15 18:57 run -> /run
| drwxr-xr-x 6 root root 4096 Jun 26 16:27 spool
| drwxrwxrwt 2 root root 4096 May 16 15:35 tmp [NSE: writeable]
|_drwxrwxrwx 10 root root 4096 May 18 00:02 www [NSE: writeable]
PORT STATE SERVICE
21/tcp open ftp
apt-get remove proftpd-basic, and after stopping the service: service proftpd stopnetstat -tulpn | grep proftpdps -eFH | grep -B5 <process id>
toor 3795 1 0 6495 1200 0 Jun25 ? 00:00:00 proftpd: (accepting connections) unfortunately, all this tells us is that it was run by toor. (who really should be named tool).lsof -p <process id> gives at least something useful:
proftpd 3795 toor txt REG 8,1 531568 2894102 /usr/local/sbin/proftpd/usr/local/sbin/proftpdcat /usr/local/sbin/proftpd will fill your screen with junk, because it is a binary file, not a text file).rm /usr/local/sbin/proftpdrm /etc/init.d/proftpdkill -s SIGKILL 3795netstat -tulpn | grep proftpd ( should give no output )toor user:
deluser toor no longer gives an error message about a running process.cat /etc/vsftpd.conf – you’ll notice a lot of “insecure” options enabled.apt-get purge vsftpd. This will also insure we can install clean vsftpd with new config files.
apt-get purge cupsapt-get install vsftpdnano /etc/vsftpd.conf
anonymous_enable=YES to anonymous_enable=NO#local_enable=YES – make it local_enable=YES (otherwise scorebot will have trouble…)service vsftpd restartnetstat -tulpn | grep vsftpdnmap -p 21 -v -oN results.txt --open --script ftp-anon 192.168.210.54 (make sure to change the IP Address!!)If you are having trouble installing or removing vsftp:
rm /var/lib/dpkg/info/vsftpd.*apt-get cleanapt-get purge vsftpdapt-get purge cupsapt-get install vsftpdrm /var/www/phpinfo.phprm -rf /var/www/backups (on some servers… contains a copy of the passwd file!!!)/var/www/checksetup.plls -al /var/www to see the permissionschown -R www-data:www-data /var/www – change the web files to be owned by the apache user.find /var/www -type f -exec chmod 640 {} \; – give all files the right permissionsfind /var/www -type d -exec chmod 750 {} \; – give all directories the right permissionscd /etc/apache2ls sites-enabled – should only be 000-default. If not, let me know.nano /etc/apache2/sites-enabled/000-default ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>
<Directory /> to:
<Directory />
Options None
Order deny,allow
Deny from all
</Directory>
<Directory /var/www> to:
<Directory /var/www>
Options -Indexes
AllowOverride None
Order allow,deny
allow from all
</Directory>
service apache2 restarthttp://192.168.210.54/~root/.bashrc (replace IP address…) in a browser – you should get a “forbidden” error.http://192.168.210.54/ (replace the IP of course).cd /etc/apache2ls sites-enabled – should only be 000-default. If not, let me know.nano /etc/apache2/sites-enabled/000-default ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>
<Directory /> to:
<Directory />
Options None
Order deny,allow
Deny from all
</Directory>
<Directory /var/www> to:
<Directory /var/www>
AddHandler cgi-script .cgi
Options +ExecCGI +FollowSymLinks
DirectoryIndex index.cgi index.html
AllowOverride All
Allow from all
</Directory>
service apache2 restarthttp://192.168.210.54/~root/.bashrc (replace IP address…) in a browser – you should get a “forbidden” error.http://192.168.210.54/ (replace the IP of course).Verify it’s running as www-data – you should not need to change anything for this step.
This is a little confusing, because in /etc/apache2/apache2.conf we would expect to see some usernames, but instead we get this wierd junk.
# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
cat /etc/apache2/envvars | grep APACHE_RUN
export APACHE_RUN_USER=www-data
export APACHE_RUN_GROUP=www-data
export APACHE_RUN_DIR=/var/run/apache2$SUFFIX
ps -eF | grep apache and you should see something like:
root 20292 1 0 47285 10396 0 00:09 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20299 20292 0 47295 6584 0 00:09 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20300 20292 0 47295 6584 0 00:09 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20301 20292 0 47591 12224 0 00:09 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20302 20292 0 47295 6584 0 00:09 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20303 20292 0 47295 6584 0 00:09 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20330 20292 0 50350 23220 0 00:10 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20333 20292 0 47341 7052 0 00:10 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20334 20292 0 47295 6584 0 00:10 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20335 20292 0 47295 6584 0 00:10 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 20336 20292 0 47341 7012 0 00:10 ? 00:00:00 /usr/sbin/apache2 -k start
root 20425 18845 0 1960 872 0 00:18 pts/0 00:00:00 grep apache
nano /etc/apache2/conf.d/securityServerSignature On to ServerSignature OffServerTokens OS to ServerTokens Prodservice apache2 restartapt-get install libapache2-modsecurity (should enable the module and restart apache automatically)If you are having problems with a web application, make sure to change the database file permissions:
- verify things are bad:
ls -al /var/lib/mysql– you will see files owned by root.chown -R mysql:mysql /var/lib/mysql- verify things are now good:
ls -al /var/lib/mysql– the files should be owned by mysql
netstat -tulpn | grep mysql
nano /etc/mysql/my.cnfbind-address = 0.0.0.0 to bind-address = 127.0.0.1local-infile=0user = root to user = mysqlchown -R mysql:msyql /var/lib/mysqlservice mysql restartnetstat -tulpn | grep mysql
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 24388/mysqld – this indicates we are listening only on localhost.nmap 192.168.210.54This is necessary on at least drupal. I recommend not removing it since it listens locally by default, and can cause issues if you remove it.
Note: if you are having trouble restarting tomcat, you will need to reinstall java:
- install the Java 7 runtime environment:
apt-get install openjdk-7-jre- reconfigure tomcat to use the newly installed java:
- edit this config file:
nano /etc/default/tomcat7- change this line
#JAVA_HOME=/usr/lib/jvm/openjdk-6-jdkto thisJAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
echo $CATALINA_HOME
/opt/tomcat7cd /opt/tomcat7cd webappspwd – should give: /opt/tomcat7/webappsrm -rf docs examples ROOTnano /opt/tomcat7/conf/tomcat-users.xml <role rolename="manager-gui"/>
<user username="tomcat" password="tomcat" roles="manager-gui"/>
password="tomcat" to password="badwolf" changes the password to badwolf – but use a service account password.service tomcat7 restartnano /opt/tomcat7/conf/server.xml<Server port="8005" shutdown="SHUTDOWN"> to <Server port="-1" shutdown="SHUTDOWN">service tomcat7 restartnano /opt/tomcat7/conf/server.xml<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />service tomcat7 restartSecure SHell allows operating system users to log in. Apart from changing operating system user passwords and updating packages, there isn’t much to do here.
There is the scan warning about weak encryption algorithms. However, as advanced as our Red Team is, they aren’t the NSA, and they don’t have a week to attack us, so the weak encryption algorithms should not cause significant problems, and therefore are not worth spending time on (yet). Also, we aren’t using outdated ssh clients, so we won’t be exposing this vulnerability.
If you have time, try installing fail2ban as a countermeasure.
You can also remove ssh key-based authentication as a pivot prevention measure:
nano /etc/ssh/sshd_config
PubkeyAuthentication yes to PubkeyAuthentication noservice ssh restartOnce you have your operating system and services secure, it’s time to secure the applications running on your system, which can often provide a lot of access for attackers to your computer.
Your applications are:
For each, you will need to address the same broad categories:
I have been in charge of two servers running WordPress that got hacked. One was because WordPress was out of date, another was because a WordPress account had a bad password.
Mr Frankenfield will cover this section in more detail.
Check for anonymous ftp on all your linux servers:
* From Kali nmap -p 21 -v -oN results.txt --open --script ftp-anon 192.168.210.0/24 (make sure to change the 210 to your subnet!)
Mr Frankenfield will cover firewalls in his networking section, but I’ll quickly cover them in less detail.
The Securing Debian Manual has a lot of in-depth detail on the process of securing a Debian Linux computer.
Here are some videos to help introduce concepts that may be foreign to you:
Learn the basics from the learning the shell tutorial.
If you already know how to use the command line, you should learn scripting. Scripting unleashes the true power of linux and its command line. You can learn more about that from a different tutorial on the site.
You can make your scripts even more useful with scheduling. Here’s an article on cron
For the adventurous: I love using vim. If you are bored and want a really cool challenge, learn vim. Try this vim tutorial to get started.
User management can get much more complicated, and in a normal production setup we would disable root login, give every team member a user account on the server, and do a lot more in-depth configuration of the user accounts on the server. Our objective is to secure against Red Team, so we will do the absolute minimum that achieves that goal, so that we have time to do all the things that secure against Red Team, instead of spending the whole week doing enterprise-grade user management.
See this ubuntu article for more info on managing users (our servers are debian, but ubuntu is based on debian, and its article is much more friendly).
You can get more information from this article and this q&a.
FTP Apache Apache Wiki Permissions Article Apache Permissions Article Securing Mysql Running Mysql as non-root Securing Tomcat Scanning for malware and rootkits Disabling SSH Public Key Authentication