cxtracker, daemonlogger, Debian, forensics, Linux Distributions, OpenSourceSoftware, PADS, Security, Sguil, Snort, Suricata, Ubuntu

Ubuntu repo for sguil

I have spent the last week setting up a Ubuntu Launchpad PPA for my packages I used to hoste here on my blog.

The URL to my PPA is : https://launchpad.net/~ebf0/+archive/gamelinux

I pack the packages mainly for Lucid Lynx 10.04.
To try them out, you can add the following in /etc/apt/sources.list:
deb http://ppa.launchpad.net/ebf0/gamelinux/ubuntu lucid main
deb-src http://ppa.launchpad.net/ebf0/gamelinux/ubuntu lucid main

To add my key to you Ubuntu installation:
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4B04D050

Then you should be able to apt-get update, and then apt-get install my packages 🙂

Please try them out and give me feedback!
You will find my howto on how to configure them here.

Happy F8’ing!

Standard
Information, Linux Distributions, OpenSourceSoftware, Security, Sguil, Ubuntu

Ubuntu 10.04 and my sguil-client .deb package

Finally moving on to Ubuntu 10.04 LTS (lucid) and installing my sguil-client_0.7.0-3_all.deb package, I had to run into some problems…

$ ./sguil.tk
ERROR: Cannot fine the Iwidgets extension.
The iwidgets package is part of the incr tcl extension and is
available as a port/package most systems.
See http://www.tcltk.com/iwidgets/ for more info.

Read here if you want to know more.

Quick and dirty, this is how I fixed it after installing the sguil-client:

$ sudo apt-get remove tcl8.5

Install itk3 and itcl3 from here and here.
Then:

$ sudo apt-get install iwidgets4

Install the sguil-client_0.7.0-3_all.deb again, and Bob is your uncle!

I also pinned the packages, so that an upgrade would not b0rk things.
In /etc/apt/preferences.d/00Sguil:

Package: itcl3
Pin: release a=hardy
Pin-Priority: 900

Package: itcl3
Pin: release a=lucid
Pin-Priority: -10

Package: itk3
Pin: release a=hardy
Pin-Priority: 900

Package: itk3
Pin: release a=lucid
Pin-Priority: -10

Not sure if this is 100% correct, as I don’t have hardy in my sources.list, but it seems to work 🙂
For aptitude, use:

$ sudo aptitude hold itcl3 itk3

Enjoy!

Standard
Information, Linux Distributions, OpenSourceSoftware, PADS, Security, Sguil

Bumped version on PADS

Small PADS info:

I bumped the version of pads to 1.2.1 (My version) after applying a patch that fixes many issues as follow:
PADS did not enable warnings during compilation. Enabling that revealed
that it did not actually include header files declaring the functions it
used. Fixing this revealed a multitude of little bugs of varying
severity, including:
- Uninitialized variables
- Unused variables
- Using close() instead of fclose()
- Using a bstring as a string, rather then using bdata()
- Useless statements
- Return without argument, even though function must return something
- Assuming time_t is int
- Passing pointers to arrays instead of the array itself

Many thanks to Erwin Paternotte for submitting this patch in the work of getting pads to work on Hardened Gentoo 64bit.

Standard
Debian, Information, Linux Distributions, OpenSourceSoftware, Security, Sguil, Ubuntu

Sguil on Ubuntu 10.04 LTS (Lucid Lynx)

As Ubuntu 10.04 (Lucid Lynx) is the next LTS (Long Time Support) version of Ubuntu that is coming out soon (April 29, 2010), I have started to look at how sguil and my dot deb packages will work.

I installed Lucid Lynx yesterday and installed my server and sensor debs on it.

Some first notes:

* MySQL is not eating the create_sguildb.sql (Just remove the comments)
* Lucid (and Karmic) does not ship with tclx8.3 😦 (Installing the Hardy version worked just fine)

(I filled a bug report to Ubuntu, hoping to get tclx8.3 into the final release…)

So, from my first tests, it seems to work fine!

I have yet to test the sguil-client on Lucid, and also I did not get to test with extensive amount of traffic and operations on the Lucid test server.

So, looking promising 🙂

Standard
Information, Linux Distributions, OpenSourceSoftware, PADS, PRADS, Security, Sguil, Ubuntu

My version of pads-1.2-sguil-mods

Saturday 18 Jun 2005 Matthew J. Shelton released PADS. PADS is a great tool, and the security industry really needs a good open source passive asset tool. But since 2005, PADS development has stopped, and there are no place to send new signature or patches/bugs too, and hope that they will get added/fixed. Also, logical, there are no new features being added…

I have used PADS in my Sguil setup, but have seen that it lacks stuff that I wanted to have there, and also, there has been some problems running PADS on newer operation systems. I have a copy of the pads-1.2-sguil-mods.tar.gz, and I added it to github yesterday, fixed some issues when writing data to the FIFO file for Sguil, added some patches, among vorants vlan patch. I compiled it on Ubuntu Hardy and Jaunty (x86_64), and it has been running fine for 12+ hours.

If you try out my version of PADS and have issues, I will try to solve them. I see there are some, in stuff that I don’t use, and if I one day find the urge, I’ll fix them on my own.

I should probably also mention, shamelessly again, that there is a project that takes PADS to the next level and then some more….
You can read about PRADS here and what more it can do for you.

Standard
Information, Linux Distributions, OpenSourceSoftware, Security, Sguil, Snort, Sourcefire, Suricata

Some notes on “making Snort go fast under Linux”

These are general pointers too things you want to dig into when you need to optimize Snort. If you are one of those who believe that Snort can’t go beyond 100Mbit/s and still not drop packets, you should read on. Comments/feedback/new tips/corrections on how to tune a Snort system is very welcome.

–[ Optimize the hardware ]–
This is always a moving target… And you need to keep yourself updated on the topic and pay attention when you buy your hardware. If someone in the community is maintaining a updated list of such hardware, give me a note!

Intel Network Interface Controllers(NIC) are the off the shelf choice of network adapters, 825NNXX PCI Express series with minimum TCP segmentation offload, TCP, UDP, IPv4 checksum offload, interrupt moderation, and maybe Bypass if you use inline mode/IPS.

If you want to pay someone that already has researched a bit (pure speculation from my side), then maybe Endace could be a choice. But if you first go there, then why not just go straight to Sourcefire (The makers of Snort).

(Matt Jonkman states that you can increase your Snort throughput up to a 16-fold increase if you introduce Endace platform’s acceleration features. Matt is the founder of Emerging Threats, and also deep into the OISF and the Suricata project)

At one time (early 2009), a discussion on IRC (Freenode) summed up in something like this:
“IICH8 southbridge, and 975G north bridge performing at 1066MHz, 8GB of 1333MHz DDR2 ram on a Intel quad core 3.2Ghz 8MB L2 cache processor running at 1333 MHz FSB and Intel 825NNXX PCI Express Gigabit Ethernet Controller.” – for a high end sniffer at that time.

Your whole system would benefit great from fast hard drives, as I/O too hard drives generally sucks juice, and locks up the system.

To sum it up:
Fast CPUs, fast RAM, fast buses, fast hard drives and a good network adapter.

–[ Optimize the Linux kernel ]–
In the file /etc/sysctl.conf – you should consider options like these:

# Just sniffing:
net.core.netdev_max_backlog = 10000
net.core.r mem_default = 16777216
net.core.rmem_max = 33554432
net.ipv4.tcp_mem = 194688 259584 389376
net.ipv4.tcp_rmem = 1048576 4194304 33554432
net.ipv4.tcp_no_metrics_save = 1
# IF also in Inline mode:
net.core.wmem_default = 16777216
net.core.wmem_max = 33554432
net.ipv4.tcp_wmem = 1048576 4194304 16777216
# Memory handling – not that important
vm.overcommit_memory=2
vm.overcommit_ratio = 50

–[ Optimize your network interface card ]–
Change the RX and TX parameters for the interfaces. The following command will display the current settings and the maximum settings you can bump them up to.

# ethtool -g ethX

To change settings, the command is something like this:

# Just sniffing
ethtool -G ethX rx
# and for inline mode, also add
ethtool -G ethX tx

Adding the command to /etc/rc.d/rc.local so that they are execute automatically when you boot would be a good idea.

–[ Optimize Snort ]–
Snorts performance is based on several factors.
1 – YOUR network!
2 – How snort is compiled
3 – Preprocessors enabled
4 – Rules
5 – Snort in general and snort.conf

–[ 1. YOUR network! ]–
Your network is a variable that is most likely not like any other networks. The amount of concurrent connections, packets and packet size flowing through, is most likely unique. Also, depending on the payload in your packets, Snort will perform differently. Also, if your $HOME_NET is one single host, compared to complex list of “networks” and “!networks”, Snort will spend more time figuring out what to do.

–[ 2. How snort is compiled ]–
First, I recommend only to compile Snort with the options that you need. I used to compile Snort in two different ways, one including options among “–enable-ppm and –enable-perfprofiling” and one without. But as my sensors are not suffering enough at the moment, I include them both by default, for easy access to preprocessor and rule performance data if I need too.

Also, I have not confirmed this, because its out of my budged reach, but the rumors are that Snort performs up to 30% better if it is compiled with an Intel C compiler (and probably run on pure Intel hardware).

If you use Phil Wood mmap libpcap and compile Snort with that, you will get some better performance in the packetcapture, giving you less dropped packets. I nice writeup/howto is found here.

–[ 3 – Preprocessors enabled ]–
How many and which preprocessors you have enabled is also playing a role on the total performance of your system. So if you can, you need to reduce the numbers of preprocessor to a minimum. Also you need to read the Snort documentation, and figure out the best settings that you can live with for each preprocessors that takes configuration options. The flow_depth parameter in the http_inspect preprocessor is a good example.

Here are two settings/views I switch between when profiling preprocessors:

config profile_preprocs: print 20, sort avg_ticks, filename /tmp/preprocs_20-avg_stats.log append
# And
config profile_preprocs: print all, sort total_ticks, filename /tmp/preprocs_All-total_stats.log append

You should now review the *stats.log files and make changes based on your interpretation, and profile again to see if things get better or worse.

–[ 4 – Rules ]–
The amount of rules also affects the performance of Snort. So tuning your rules to just enable the ones that you need is essential when aiming for performance.
Also, how a rule is performing on your network, might defer from how it performs in my network… That said, you need to profile your set off rules, and tweak or disable them so your system uses less overall “ticks”.

Here are two settings/views I switch between when profiling rules:

config profile_rules: print 20, sort avg_ticks, filename /tmp/rules_20-avg_stats.log append
# And
config profile_rules: print all, sort total_ticks, filename /tmp/rules_All-total_stats.log append

You will get a fairly good view of rules that needs/should/would benefit from tuning/disabling.

–[ 5 – snort in general and snort.conf ]–
* search-method
You should look into which search-method snort is using. The default search method is AC-BNFA (Aho-Corasick NFA – low memory, high performance). This is probably the best overall search method, but if you have the RAM for it, AC (Aho-Corasick Full – high memory, best performance) would be a better choice. Snort 2.8.6 added a new pattern matcher named AC-SPLIT. The new pattern matcher is optimized to use less memory and perform at AC speed. This would probably the choice for the future? Need to test right away 🙂
To enable it, add something like:

config detection: search-method ac-split, max-pattern-len 20,
search-optimize

* Latency-Based Packet Handling
If you have a problem with dropped packets, I would say over 1% on an average, I would recommend enabling Latency-Based Packet Handling. You should run some tests in your environment to find a value that works for you, but the general situation is like this:
If your Snort “Packet Performance Summary” is telling you that your “avg pkt time is 10 usecs” then Snort can inspect about 1000 packets in 10000 usecs. If a packet for some reason is using 10000 usec to get through Snort, you may have dropped/sacrificed 1000 other packets in that time frame, just to inspect this packet. So if you configure max-pkt-time to be 1000, Snort will stop inspecting packets that take more time than 1000 usec, and in this basic example leaving you with 100 dropped packets instead of 1000. You choose! (The example is not technical correct, as a packet can take over 10000 usec with out Snort dropping any packets at all (Imagine if there is only one packet going through snort that day…), but in my tests, this is more or less the real world outcome of enabling Latency-Based Packet Handling).
Example:

config ppm: max-pkt-time 10000, fastpath-expensive-packets, pkt-log

Other keywords you should be aware off in the Snort config, that I don’t want to go into details about, as I don’t have enough Snort-Fu about to stand firm, and the doc is rather lacking! I have a personal understanding of what they do, and how it effects performance etc. but if anyone has some nice writeup of the topics, please point me to it!! :
* Event Queue Configuration
* Latency-Based Rule Handling

–[ Additional notes ]–
Obviously, if you need to go as fast as possible, your system should not be used for lots of other different stuff. So keep your running processes/services too a minimum.

Snort is also, as far as I can tell, single threaded when it comes too packet inspection. There is a pdf here from Intel, explaining how Sensory Networks Software Acceleration Solutions boost performance of Snort and things alike, making them Multi-core enabled/aware.

That said, Snort benefits from sticking to one CPU, so using schedtool in a proper way, might help snort perform overall better. If you are running multiple instances of Snort on one multi-CPU server, you should use schedtool to stick each Snort process to its own physical CPU etc. Example:

$ man schedtool # and read about “AFFINITY MASK” and understand the difference between cpu-cores and hyper-threading etc.
$ schedtool <pid of snort> # Displays current settings
$ schedtool -a 0x01 <pid of snort> # Pin the snort process to one CPU (The first)
$ schedtool -M 2 -p 10 # Change the policy to SCHED_RR and set priority to 10 (0 highest, 100 lowest)
$ schedtool <pid of snort> # to verify your changes

Always when optimizing a system, you should have some sort of measuring system. I use Munin. I wrote some basic Munin plugins for Snort which monitors the most important stuff.

And as always,
“Measure, don’t speculate” — Unknown
“Premature optimization is the root of all evil” — Tony Hoare

Standard