daemonlogger, forensics, fpcgui, Information, Linux Distributions, Metasploit, OpenFPC, OpenSourceSoftware, Security, Ubuntu


In early June, Leon Ward and I teamed up in Oslo chatting about his OpenFPC and my FPCGUI project. I met Leon for the first time in April 2009 at Sourcefires offices in Wokingham, UK, and I have chatted with him now and then on IRC etc. since then.

I started using Sourcefire 3D in December 2008, and the first thing I was missing was the lack of pcaps from the events that I got. The second was the real-time view that you get in Sguil (I can live without that though).

So I needed a second host that did full packet capture along side my new IPS/IDS. Just running tcpdump/daemonlogger/sancp is OK for a small installation, but carving out the sessions manually was taking time. I needed to script something that would take an easy interface, so I could quickly get a pcap from the whole sessions I was getting events from. So I was thinking of an API and a easy way to add this to the Sourcefire 3D WebGUI.

My PoC was FPCGUI (Full Packet Capture Graphical User Interface). It can take a query in the URL, search the flow data from its database and give you the sessions details if it exists. If you click on the session, you will get the pcap served straight in you face, and I choose to open my pcaps with wireshark. With a little grease monkey magic, this would have been an OK solution for satisfying my pcap needs working with SF3D.

I made my thoughts public in a blog post in September 2009 and started coding right away. I also discussed FPCGUI with Leon the day after I posted the blog. The first release that worked good enough for me was in January 2010. Leon released his project in May 2010, and I quickly saw that we where doing more or less the same. He had implemented the distributed node part, which I had not started to even draft, and I had the WebGUI and flowdata which gives more meaning and is more user friendly to the analyst.

So, instead of working on two separate projects (aiming for the same goal), we decided to join forces and merge the two projects. And as I thought that OpenFPC is a better name than FPCGUI, OpenFPC it is 🙂

I have merged my parts slowly into OpenFPC during the summer, with vacation time and changing job, I did not have much time for coding on the side. We also re-factored much of the code, file names etc., so getting thing to a working condition has been the main task.

As of the last weeks, I can now install OpenFPC and use it in the way that I want it again, like I did with FPCGUI. The plus is that I now have a command line interface, a distributed architecture (Not WebGUI friendly yet), and a way to automagically extract pcaps and files in it, for automatic analysis 🙂

To test my dream of automatic analysis, I used a setup similar to this earlier blog post where I more or less did the same. I carve the pcap with openfpc-client (which will come from an event from an IDS or nftracker), extract files with tcpxtract (or simular tools), scan files with ClamAV and also test md5/sha sums towards shadowserver, virustotal or wepawet. I tried some different infected and non infected PDF files. All files I had to test with was detected with ClamAV, even my home grown metasploit PDF. All known bad PDF files was detected with the md5/sha sum of the files towards the different services (shadow/VT/wepawet), but again, only ClamAV detected my home made metasploit PDF.
evil.pdf: Heuristics.PDF.ObfuscatedNameObject FOUND

So, now I will have more events to live with 🙂

Information, Linux Distributions, OpenSourceSoftware, Security, Sguil, Ubuntu

Ubuntu 10.04 and my sguil-client .deb package

Finally moving on to Ubuntu 10.04 LTS (lucid) and installing my sguil-client_0.7.0-3_all.deb package, I had to run into some problems…

$ ./sguil.tk
ERROR: Cannot fine the Iwidgets extension.
The iwidgets package is part of the incr tcl extension and is
available as a port/package most systems.
See http://www.tcltk.com/iwidgets/ for more info.

Read here if you want to know more.

Quick and dirty, this is how I fixed it after installing the sguil-client:

$ sudo apt-get remove tcl8.5

Install itk3 and itcl3 from here and here.

$ sudo apt-get install iwidgets4

Install the sguil-client_0.7.0-3_all.deb again, and Bob is your uncle!

I also pinned the packages, so that an upgrade would not b0rk things.
In /etc/apt/preferences.d/00Sguil:

Package: itcl3
Pin: release a=hardy
Pin-Priority: 900

Package: itcl3
Pin: release a=lucid
Pin-Priority: -10

Package: itk3
Pin: release a=hardy
Pin-Priority: 900

Package: itk3
Pin: release a=lucid
Pin-Priority: -10

Not sure if this is 100% correct, as I don’t have hardy in my sources.list, but it seems to work 🙂
For aptitude, use:

$ sudo aptitude hold itcl3 itk3


Information, OpenSourceSoftware, Security, Snort

Virtual splitting networks in Snort

I haven’t seen any HOWTOs yet on how to use the feature of dividing up your network into multiple snort configuration files (Virtual Networks?). I have tried this on my sensors and it works great.

Before, one would solve the problem by firing up multiple instances of snort, each with their own sets of options/arguments. Now we only start one instance of snort with a default snort config, and including config files for each IP, IP-range or VLAN that one would like to monitor. The default snort config file is used as a fall-back if the traffic is not matched in one of the virtual configs.


config binding: /etc/snort/vips/snort-0.conf net
config binding: /etc/snort/vips/snort-1.conf net
config binding: /etc/snort/vips/snort-2.conf net
config binding: /etc/snort/vips/snort-3.conf vlan 1337

So, you have a default /etc/snort/snort.conf and configure that as a fall-back configuration (Catch all traffic not handled by your virtual configs) and then add the statements above. You can then configure snort-0.conf, snort-1.conf, snort-2.conf and snort-3.conf to handle their respective traffic (Variables, rules, preprocessors etc).

In this case, if you have: on eth1 on eth2 on eth3
vlan 1337 on eth4

you would need to bond them together and have snort listen on the bonded interface.

My gut feelings are that there are some performance and memory benefits firing up one instance of snort configured with virtual-networks, then firing up X instances of snort, but I have not done any tests.

Read more in the README.multipleconfigs in the doc/ directory of the Snort Tarball.

*I would like to hear thoughts from other playing with this feature*

Information, Linux Distributions, OpenSourceSoftware, PADS, Security, Sguil

Bumped version on PADS

Small PADS info:

I bumped the version of pads to 1.2.1 (My version) after applying a patch that fixes many issues as follow:
PADS did not enable warnings during compilation. Enabling that revealed
that it did not actually include header files declaring the functions it
used. Fixing this revealed a multitude of little bugs of varying
severity, including:
- Uninitialized variables
- Unused variables
- Using close() instead of fclose()
- Using a bstring as a string, rather then using bdata()
- Useless statements
- Return without argument, even though function must return something
- Assuming time_t is int
- Passing pointers to arrays instead of the array itself

Many thanks to Erwin Paternotte for submitting this patch in the work of getting pads to work on Hardened Gentoo 64bit.

Information, OpenSourceSoftware, Security

Phony security advisory from Tim Brown (www.nth-dimension.org.uk) about Varnish

I just had to comment on this…

Read the advisory here.

My short comment; If you install any type of Software, or use any kind of mechanical devices, or do anything in life, be sure too know what you are doing.

If you buy a car, and the car door is not locked when you are handed the keys, do still lock the doors if you don’t want people to come into your car!

Snippets from the advisory:
"In order to completely protect against the vulnerability (in the short term), Nth Dimension recommend turning off the server and replacing it with another reverse proxy such as Squid."

That would be like stepping out of a Ferrari and crawling into twelve old Tractors… I don’t think people will do that Mr. Brown…

"Should this not be possible, Nth Dimension would strongly recommend that users confirm that the master process is not listening on an external network interface."

This is so much easier to do than migrate to Squid or alike? And the right thing to do if you are not in a trusted environment. Again, do lock your car door.

"In the latter case, users should confirm that only trusted users have SSH access to the system."

As a rule of thumb: You should NEVER have untrusted users on your systems if you value your data on it or the data accessible from it.

There are tons of information on how to harden a Operating System (OS). One of the first and most common step is to make sure the system does not listen on network ports that you don’t want it too. I feel that the advisory is bogus because it is a feature of Varnish.

The advisory should have been aimed at the distributions that have packages that don’t implement “non-clue friendly defaults”.

That said, there is nothing stopping me from sending out my passwords via email once I have installed a browser and I manage to log into my gmail account…
Subject: “Medium security hole in Mozilla Firefox”
Body: “I’ve identified a couple of security flaws affecting Mozilla Firefox (All versions) which may allow privilege escalation….”

“Grumpy day”

Debian, Information, Linux Distributions, OpenSourceSoftware, Security, Sguil, Ubuntu

Sguil on Ubuntu 10.04 LTS (Lucid Lynx)

As Ubuntu 10.04 (Lucid Lynx) is the next LTS (Long Time Support) version of Ubuntu that is coming out soon (April 29, 2010), I have started to look at how sguil and my dot deb packages will work.

I installed Lucid Lynx yesterday and installed my server and sensor debs on it.

Some first notes:

* MySQL is not eating the create_sguildb.sql (Just remove the comments)
* Lucid (and Karmic) does not ship with tclx8.3 😦 (Installing the Hardy version worked just fine)

(I filled a bug report to Ubuntu, hoping to get tclx8.3 into the final release…)

So, from my first tests, it seems to work fine!

I have yet to test the sguil-client on Lucid, and also I did not get to test with extensive amount of traffic and operations on the Lucid test server.

So, looking promising 🙂

Information, OpenSourceSoftware, PRADS, Security, Snort

Populating Snorts host attribute tables with PRADS

It has been a long journey, but after about two years, I finally got a way to populate Snorts host attribute table, automagically(tm)!

When I started this, my first option was to use nmap to scan the network to populate the information. This was scratched, as my goal was to be non intrusive and always up to date (The minute a new host popped up, I want to know). Scanning 65535 ports times two for each of the hosts Im monitoring is not an options also… I started to look at the Open Source tools out there, on which to use to get the information from. As I was familiar with p0f and PADS, I saw that they could do the job, but they needed some band-aid to work together, and they where lacking active development… p0f has a DB patch/version, and I already had PADS hooked up in Sguil, so I had the info in a DB, but not in the way I wanted it. So I started out on a journey to merge the two projects, enhance them, and try to speed things up a bit.

The project is still in development, but the main parts are done. It is useful in the way that it will print out information about detected hosts, like this in verbose mode (And yes, it also does IPv6):

2a02:c0:1002:100:21d:72ff:fe92:728,[syn:S4:64:1:40:M1440,S,T,N,W7:Z],[Linux:2.6 (newer, 7) IPv6],[link:IPv6/IPIP],[uptime:2hrs],[distance:0]
2a02:c0:1002:10::2,[synack:5712:63:1:40:M1440,S,T,N,W7:ZAT],[Linux:2.6 (newer, 7) IPv6],[link:IPv6/IPIP],[uptime:4069hrs],[distance:1]
2a02:c0:1002:10::2,[service:OpenSSH 5.1p1 (Protocol 2.0):22:6],[distance:1]
2a02:c0:1002:100:21d:72ff:fe92:728,[client:OpenSSH 5.1p1 (Protocol 2.0):22:6],[distance:0]

At the moment, it also makes a file in your /tmp/ dir, /tmp/prads-asset.log, which presents the info in the following way:

2a02:c0:1002:100:21d:72ff:fe92:728,0,56268,6,SYN,[S4:64:1:40:M1440,S,T,N,W7:Z:Linux:2.6 (newer, 7) IPv6:link:IPv6/IPIP:uptime:2hrs],0,1269420770
2a02:c0:1002:10::2,0,22,6,SYNACK,[5712:63:1:40:M1440,S,T,N,W7:ZAT:Linux:2.6 (newer, 7) IPv6:link:IPv6/IPIP:uptime:4069hrs],1,1269420770
2a02:c0:1002:10::2,0,22,6,SERVER,[ssh:OpenSSH 5.1p1 (Protocol 2.0)],1,1269420770
2a02:c0:1002:100:21d:72ff:fe92:728,0,22,6,CLIENT,[ssh:OpenSSH 5.1p1 (Protocol 2.0)],0,1269420770

Input from the community on how to present the information/output for a best possible way for integration with other applications are welcome.

To try it out, this is what I believe is necessary to install on my Ubuntu machine to run it:

$ sudo aptitude install build-essential git-core libpcre3-dev libpcap0.8-dev
$ git clone http://github.com/gamelinux/prads.git
$ cd prads/src/ && make
$ # then to start it
$ sudo ./prads -i ethX -v

For populating the Snort host attribute table, there is a script in the tools dir, prads2snort.pl, which takes the prads-asset.log file and processes it.

$ perl prads2snort.pl -i prads-asset.log -o hosts_attribute.xml -v -f

The -v (verbose) mode prints out some details, which can be good to check to see if stuff seems to be detected correctly.

Snort supports reloading of the attribute table if you give it the signal 30. (kill -30 <snort-pid>). This means, that if you discover a difference in your host attribute table (Say you got a new http service some where, or a host has changed OS), you can swap out the attribute file with a new updated one, and tell snort to reload its attribute file without restarting snort! Darn cool if you ask me 🙂

You can read more about Snort and its host attribute table here, and you can read about another tool called Hogger here. Also, you should read the Snort documentation section 2.7 (around page 104/105) “Host Attribute Table”.

I would once again like to thank Michal Zalewski and Matt Shelton for their work on p0f and pads. I would also like to thank Martin Roesch & The Snort Team (And all the contributers) for a great application and all the effort they have put into Snort and its surroundings. (Virtually giving you the price for best Open Source security application 2000 – 2010!).

Attribute Table Loaded with 980 hosts

Attribute Table Reload Thread Starting…
Attribute Table Reload Thread Started, thread 363022672 (15333)

$ /bin/kill -30 15333

Swapping Attribute Tables.

$ /bin/kill 15333

Attribute Table Stats:
Number Entries: 980
Table Reloaded: 1

Information, Linux Distributions, OpenSourceSoftware, PADS, PRADS, Security, Sguil, Ubuntu

My version of pads-1.2-sguil-mods

Saturday 18 Jun 2005 Matthew J. Shelton released PADS. PADS is a great tool, and the security industry really needs a good open source passive asset tool. But since 2005, PADS development has stopped, and there are no place to send new signature or patches/bugs too, and hope that they will get added/fixed. Also, logical, there are no new features being added…

I have used PADS in my Sguil setup, but have seen that it lacks stuff that I wanted to have there, and also, there has been some problems running PADS on newer operation systems. I have a copy of the pads-1.2-sguil-mods.tar.gz, and I added it to github yesterday, fixed some issues when writing data to the FIFO file for Sguil, added some patches, among vorants vlan patch. I compiled it on Ubuntu Hardy and Jaunty (x86_64), and it has been running fine for 12+ hours.

If you try out my version of PADS and have issues, I will try to solve them. I see there are some, in stuff that I don’t use, and if I one day find the urge, I’ll fix them on my own.

I should probably also mention, shamelessly again, that there is a project that takes PADS to the next level and then some more….
You can read about PRADS here and what more it can do for you.