Corelan Cybersecurity Research

Introduction

After spending a few hours fighting a battle against Snorby and Apache2 + Passenger, I finally managed to get it to run properly on my Ubunty 10.x box (32bit). Looking back, I figured I might not be the only one who is having issues with this.

So I decided to publish the notes I took while setting everything up, and as a little bonus, explain how to install and configure Suricata as well (configured in combination with barnyard2 which will pick up local logs and send them to the remote MySQL server).

There are the components that will be installed :

• Snorby 2.x (latest revision from git)
• MySql 5
• Ruby 1.9.2p0
• Apache2
• Passenger 3
• Barnyard 2
• Suricata 1.1beta1 with emerging-threat ruleset

 

Install dependencies / prerequisites for Snorby

Packages

First, make sure your system is up to date :

aptitude update
apt-get update
apt-get upgrade
apt-get dist-upgrade

Then install new packages :

apt-get install gcc g++ build-essential libssl-dev libreadline5-dev \
     zlib1g-dev linux-headers-generic libsqlite3-dev libxslt-dev libxml2-dev \
     imagemagick git-core libmysqlclient-dev mysql-server libmagickwand-dev \
     default-jre

wkhtmlpdf with QT patch

cd /tmp
wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.10.0_rc2-static-i386.tar.bz2
bunzip2 wkhtmltopdf-0.10.0_rc2-static-i386.tar.bz2
tar xvf wkhtmltopdf-0.10.0_rc2-static-i386.tar
cp wkhtmltopdf-i386 /usr/bin/wkhtmltopdf

Ruby 1.9.2p0

cd /tmp
wget http://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p0.tar.gz
tar -xvzf ruby-1.9.2-p0.tar.gz
cd ruby-1.9.2-p0
./configure
make && make install
ln -s /usr/local/ruby/bin/bundle /usr/bin

Run "ruby – v" and verify that it returns the correct version :

ruby 1.9.2p0 (2010-08-18 revision 29036) [i686-linux]

(If this shows a different version, then verify that /usr/local/ruby/bin/ruby -v is version 1.9.2p0)

gems

gem install thor i18n bundler
gem install tzinfo builder memcache-client rack rack-test erubis mail text-format
gem install rack-mount --version=0.4.0
gem install rails sqlite3-ruby

 

Installing Snorby

git clone http://github.com/Snorby/snorby.git /var/www/snorby

Edit configuration files :

Edit /var/www/snorby/config/database.yml : look for the "snorby" entry and enter the mysql root username & password here :

snorby: &snorby
  adapter: mysql
  username: root
  password: <enter the mysql root password here>
  host: localhost

(don’t worry, we’ll get rid of the root username/password later on)

Edit /var/www/snorby/config/snorby_config.yml : set the correct path to wkhtmltopdf

development:
  domain: localhost:3000
  wkhtmltopdf: /usr/bin/wkhtmltopdf

test:
  domain: localhost:3000
  wkhtmltopdf: /usr/bin/wkhtmltopdf

production:
  domain: localhost:3000
  wkhtmltopdf: /usr/bin/wkhtmltopdf

 

 

Run Snorby setup :

cd /var/www/snorby
rake snorby:setup

It is very likely that you will get the following error :

(in /var/www/snorby)
You have requested:
  activesupport = 3.0.3

The bundle currently has activesupport locked at 3.0.4.
Try running `bundle update activesupport`
Try running `bundle install`.

Fix : run the following commands in the /var/www/snorby folder :

bundle update activesupport railties rails
gem install arel
gem install ezprint
bundle install

Run the setup again :

cd /var/www/snorby
rake snorby:setup

If all goes well, the snorby database should get created/populated now. Since we used the mysql root username/password in the database.yml configuration file, the necessary database and tables should be created successfully.

root@server:/var/www/snorby# rake snorby:setup
(in /var/www/snorby)
<...long key....>
[datamapper] Created database 'snorby'
[datamapper] Finished auto_upgrade! for :default repository 'snorby'

If you get an error about ezprint:

(in /var/www/snorby)
rake aborted!
http://github.com/mephux/ezprint.git (at rails3) is not checked out. \ 
   Please run `bundle install`
/var/www/snorby/Rakefile:4
(See full trace by running task with --trace)

Solution : run this from /var/www/snorby

bundle pack
bundle install --path vender/cache

then run bundle install the rake snorby:setup command again

 

 

Configure mysql

We used the root user / password to allow snorby to create the necessary tables.  If you prefer to use a mysql user account that has less privileges, then you can add a new mysql user, grant privileges, and edit the snorby configuration again :

mysql -u root -p

creat user 'snorbyuser'@'localhost' IDENTIFIED BY 'some_pass';
grant all privileges on snorby.* to 'snorbyuser'@'localhost' with grant option;
flush privileges;

Now edit /var/www/snorby/config/database.yml again and replace the username and password with the newly created user

snorby: &snorby
  adapter: mysql
  username: snorbyuser
  password: some_pass
  host: localhost

You will need to create a useraccount for your (remote) suricata/snorby sensors too.  The procedure is exactly the same as indicated above, but you will have to replace ‘localhost’ with the IP address of the remote sensor.  If the sensor is local, you can use the snorbyuser@localhost mysql user account as well.

By default, the mysql server listens on localhost only.  Edit /etc/mysql/my.cnf to change the default behaviour :

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 127.0.0.1

Comment the bind-address statement (add a # in front of the line) and restart mysql

service mysql restart

Verify that the server is now listening on all ip addresses :

root@server:/# lsof -i | grep mysqld
mysqld  21309    mysql   10u  IPv4 16405476      0t0  TCP *:mysql (LISTEN)

TCP *:mysql => listening on all interfaces

 

Apache2 & Passenger

Install packages & dependencies

apt-get install apache2 apache2-prefork-dev libapr1-dev libaprutil1-dev libopenssl-ruby
apt-get install libcurl4-openssl-dev

Start apache2 and make sure the default webpage loads

service apache2 start

Install passenger

gem install --no-ri --no-rdoc --version 3.0.3 passenger

Install passenger module for apache2

/usr/local/ruby/lib/ruby/gems/1.9.1/gems/passenger-3.0.3/bin/passenger-install-apache2-module -a

 

Edit /etc/apache2/mods-available/passenger.load (or create if it does not exits) :

LoadModule passenger_module /usr/local/ruby/lib/ruby/gems/1.9.1/gems/passenger-3.0.3/ext/apache2/mod_passenger.so

Edit /etc/apache2/mods-available/passenger.conf :

   PassengerRoot /usr/local/ruby/lib/ruby/gems/1.9.1/gems/passenger-3.0.3
   PassengerRuby /usr/local/ruby/bin/ruby

Enable the module (and some other modules you might need) :

a2enmod passenger
a2enmod rewrite
a2enmod ssl

Set file/folder permissions on the snorby folder :

chown www-data:www-data /var/www/snorby -R

 

Integrate Snorby with Apache2

Suppose we want the snorby frontend to be reachable using virtualhost snorby.corelan.be :

Create a file "snorby" under /etc/apache2/sites-available :

        ServerAdmin webmaster@localhost
        ServerName snorby.corelan.be
        DocumentRoot /var/www/snorby/public

        /var/www/snorby/public">
                AllowOverride all
                Order deny,allow
                Allow from all
                Options -MultiViews
        

Enable the new website :

ln -s /etc/apache2/sites-available/snorby /etc/apache2/sites-enabled/snorby

Restart apache2 :

service apache2 restart

Make sure snorby.corelan.be points at your local apache2 server, and navigate to that website :

(log in with user snorby@snorby.org and password snorby)

If you get an error page instead of the login page :

-> complaining about ezprint.git not being installed, then go to the /var/www/snorby folder and run the following 2 commands :

bundle pack
bundle install --path vender/cache

Wait until the process has finished.

Restart apache2, and then try to access the website again, you should now be able to log on.

 

 

If you get a message about the "worker" not being started :

Solution : click "Administation", Click "Worker Options" Administration menu and select "Start worker".

Now click on "Worker Options" and start the 2 jobs

If you go back to the main page now, you may see "Currently caching" for a brief moment (depending on the number of events already in the database) :

Tip : if, at any given time, the dashboard continues to show 0 events (or an incorrect number of events in general), but the Events view shows that all entries are inside the database, then you may have to clear the caches and rebuild it from scratch :

mysql -u root -p

use snorby;
truncate table caches;
exit

Now remove the 2 worker jobs (use the little trash can icon next to each worker job to remove the job)

Recreate the jobs via Worker Options, and the main dashboard should eventually get populated again.

 

Okay, the server is now ready to receive data from local/remote sensors (Snort, Suricata, …).

 

Updating Snorby

Updating snorby is as easy as running the following commands :

cd /var/www/snorby
git pull origin master
rake snorby:update

 

 

Installing Suricata & Barnyard2

Dependencies

apt-get install libpcre3 libpcre3-dbg libpcre3-dev \
            build-essential autoconf automake libtool \
            libpcap-dev libnet1-dev mysql-client libmysqlclient16-dev

Set up yaml :

yaml :
cd /tmp
wget http://pyyaml.org/download/libyaml/yaml-0.1.3.tar.gz
tar xvfz yaml-0.1.3.tar.gz
cd yaml-0.1.3
./configure && make && make install

Install barnyard2 :

cd /tmp
wget http://www.securixlive.com/download/barnyard2/barnyard2-1.9.tar.gz
tar xvfz barnyard2-1.9.tar.gz
cd barnyard2-1.9
./configure --with-mysql && make && make install

Do NOT delete the /tmp/barnyard2-1.9 folder yet.

Install suricata:

cd /tmp
wget http://www.openinfosecfoundation.org/download/suricata-1.1beta1.tar.gz
tar xvfz suricata-1.1beta1.tar.gz
cd suricata-1.1beta1
mkdir /var/log/suricata
./configure && make && make install

Do NOT remove the /tmp/suricata-1.1beta1 folder yet, we need some files from this folder later on.

Try to run suricata :

suricata

 

If you get the following message :

suricata: error while loading shared libraries: libhtp-0.2.so.1: cannot open shared object file: No such file or directory

then add "/usr/local/lib" to /etc/ld.so.conf and run ldconfig.

root@server:/# cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf
/usr/local/lib
root@server:/# ldconfig

Run "suricata" again :

suricata
[14005] 27/2/2011 -- 22:08:28 - (suricata.c:440)  (main) -- This is Suricata version 1.1beta1
[14005] 27/2/2011 -- 22:08:28 - (util-cpu.c:171)  (UtilCpuPrintSummary) -- CPUs/cores online: 2
[14005] 27/2/2011 -- 22:08:28 - (suricata.c:765)  (main) -- [ERRCODE: SC_ERR_OPENING_FILE(40)]
    - Configuration file has not been provided

Suricata 1.1beta1
USAGE: suricata

        -c                     : path to configuration file
        -i                : run in pcap live mode
        -r                     : run in pcap file/offline mode
        -s                     : path to signature file (optional)
        -l                      : default log directory
        -D                           : run as daemon
        --engine-analysis            : print reports on analysis of different sections in the engine and exit.
                                       Please have a look at the conf parameter engine-analysis on what reports
                                       can be printed
        --pidfile              : write pid to this file (only for daemon mode)
        --init-errors-fatal          : enable fatal failure on signature init error
        --dump-config                : show the running configuration
        --pcap-buffer-size           : size of the pcap buffer value from 0 - 2147483647
        --user                 : run suricata as this user after init
        --group               : run suricata as this group after init
        --erf-in               : process an ERF file

To run the engine with default configuration on interface eth0
with signature file "signatures.rules", run the command as:

suricata -c suricata.yaml -s signatures.rules -i eth0

Get suricata rules (emerging-threats)

mkdir /etc/suricata
cd /etc/suricata
wget http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz
tar xvfz emerging.rules.tar.gz

Configure suricata :

cd /tmp/suricata-1.1beta1
cp suricata.yaml /etc/suricata/
cp classification.config /etc/suricata/
cp reference.config /etc/suricata/

(note : After copying those files, you can remove the installation folder from /tmp)

Edit /etc/suricata/suricata.yaml

Make sure alert output for barnyard2 is enabled (it is enabled by default) :

  # alert output for use with Barnyard2
  - unified2-alert:
      enabled: yes
      filename: unified2.alert

      # Limit in MB.
      #limit: 32

Scroll down until you reach "default-rule-path:" and enable/put the emerging-threat rules files that are relevant to your system under "rule-files:". (You can find the list with rules under /etc/suricata/rules).  Example :

default-rule-path: /etc/suricata/rules/
rule-files:
 - emerging-attack_response.rules
 - emerging-dos.rules
 - emerging-exploit.rules
 - emerging-games.rules
 - emerging-inappropriate.rules
 - emerging-malware.rules
 - emerging-p2p.rules
 - emerging-policy.rules
 - emerging-scada.rules
 - emerging-smtp.rules
 - emerging-virus.rules
 - emerging-voip.rules
 - emerging-web_client.rules
 - emerging-web_server.rules
 - emerging-web_specific_apps.rules
 - emerging-worm.rules
 - emerging-user_agents.rules
 - emerging-current_events.rules

Next, edit the HOME_NET variable and set it to your local IP or IP subnet

Example :

HOME_NET: "[192.168.0.0/24]"

That’s the basic config.

Keeping suricata up to date

You can use this optional simple script to grab a copy of the git master and update the suricata binaries :

#!/bin/bash
cd /tmp
rm -rf /tmp/suricata
mkdir suricata
cd suricata
/usr/bin/git clone git://phalanx.openinfosecfoundation.org/oisf.git
cd oisf
./autogen.sh
./configure && make && make install

 

Configure barnyard2 :

Get the sample config file from the installation folder :

cp /tmp/barnyard2-1.9/etc/barnyard2.conf /etc/suricata/

(note : After copying the file, you can remove the installation folder from /tmp)

Edit the conf file and set the following parameters :

(we’ll assume you are installing suricata on the same box as the snorby engine)

config reference_file:      /etc/suricata/reference.config
config classification_file: /etc/suricata/classification.config
config gen_file:            /etc/suricata/rules/gen-msg.map
config sid_file:            /etc/suricata/rules/sid-msg.map

output database: log, mysql, user=snorbyuser password=some_pass /
   dbname=snorby host=localhost sensor_name=sensor1

(obviously the output database configuration must be placed on one line, remove the / between the password and dbname.)

If you are installing remote suricate sensors (remote from the mysql server / snorby engine point of view), then you will have to configure mysql and grant access to the remote mysqluser, from the IP of the sensor.  The "host" entry in the barnyard2.conf file needs to point at the remote mysql server.

Finally, create the log folder for barnyard2 :

mkdir /var/log/barnyard2

Run barnyard2 :

barnyard2 -c /etc/suricata/barnyard2.conf -d /var/log/suricata -f unified2.alert -w /var/log/suricata/suricata.waldo -D

This will run barnyard2 in daemon mode. If barnyard2 does not appear to be working, omit the -D parameter and you will be able to see any errors that might prevent barnyard2 from running.

When barnyard2 is running, you should see a new sensor in Snorby. If you don’t like the display name of the sensor, you can change the name via Administration Menu – Sensors

When barnyard2 is running, you can launch suricata too :

Run suricata :

suricata -c /etc/suricata/suricata.yaml -i eth0 -D

(change interface accordingly.  -D will make suricate run in daemon mode)

 

As soon as suricata starts generating alerts, barnyard2 should pick them up, and use the mysql connector to write them into the events table of the snorby database. You should be able to see these new events in the "events" view of Snorby.

In the background (every 30 mins), the snorby worker jobs will pick up the events, process them, add them to the caches table, and show them on the dashboard too.

 

Test IDS

If you want to test your setup, then run :

lynx www.testmyids.com

(if lynx was not installed, run apt-get install lynx and try again)

Watch the /var/log/suricata folder. You should see something similar to this :

root@server:/var/log/suricata# ls -al
total 88
drwxr-xr-x  2 root root  4096 2011-02-28 05:38 .
drwxr-xr-x 18 root root  4096 2011-02-28 05:30 ..
-rw-r-----  1 root root   194 2011-02-28 05:37 fast.log
-rw-r-----  1 root root     0 2011-02-28 05:35 http.log
-rw-r--r--  1 root root 66873 2011-02-28 05:39 stats.log
-rw-------  1 root root  2056 2011-02-28 05:38 suricata.waldo
-rw-r-----  1 root root     0 2011-02-28 05:34 unified2.alert.1298867650
-rw-r-----  1 root root    60 2011-02-28 05:37 unified2.alert.1298867720

If the fast.log file, suricata.waldo and unified2.alert files are growing, then the IDS is picking up the test alerts from www.testmyids.com

 

Addendum (march 2nd 2011)

After running this setup for a few days, I noticed that Snorby only appears to be seeing "low severity" events, no matter how hard I try.

Something must be wrong.

I did some tests and it appears that suricata 1.1 might not be able to properly classify events.

I tried with snort, and that seems to work well.

Update : I filed a bug report with suricata and it looks like they have fixed the issue. You can use the suricata "update" script to grab the latest version.

 

Snort

Follow the installation guide at http://www.snort.org/assets/158/snortinstallguide2904.pdf with the exception of  creating a "snort" database. Make sure to set up barnyard2 logging and tell it to use "snorby" instead of the "snort" database.

In short, the easiest way to get snort to run on ubuntu 10.x is to run apt-get install snort   (or apt-get install snort-mysql if you want to have snort log events directly into the MySql database. This is not the recommended configuration and barnyard2 will be able to handle MySQL downtime)

 

 

---

© 2011 – 2021, Peter Van Eeckhoutte (corelanc0d3r). All rights reserved.