cxtracker updates (0.9.7 beta)

Thanks to Ian Firns that has implemented custom output formating (sancp like), pcap indexing and pcap capturing (daemonlogger-style)…!

Starting from commit 6b32fb24db, cxtracker can now, additional to writing flowdata, also do packet-capturing and outputting indexing data about where in the pcap(s) the flow starts and ends. This should potentially bring down the time needed to carve a session out of a big pcap. Right now, all this is just in beta, but the functionality is there, and there is also an example perl-script to carve out a session based on index-data.

Output fields of interest:
%spf pcap file containing start packet in session
%spo pcap file offset of start packet in session
%epf pcap file containing last packet in session
%epo pcap file offset of last packet in session

Example on a indexed pcap output, using: “%spf|%spo|%epf|%epo”
“/tmp/test1.pcap.1321821603|10115|/tmp/test1.pcap.1321821809|62704”

So, basically, if you have a 1 GB pcap file, normally you could use tcpdump with a BPF filter to care out the session you where looking for, reading and searching the whole 1 GB pcap file.

With this addition to cxtracker, you would now be able to spool right to the start-byte off the session and start carving from there until the end-byte of the session. So if the session is placed say 450 MB into the pcap, and ends at 550 MB into the pcap, you basically only have to read and carve in 100 MB of pcap data. In the example perl script (cxt2pcap.pl), the file handle for the file is opened, it would seek to the right place in the pcap (_not_ reading 450 MB of data from your disk) and start reading 100 MB data from your disk and carving+filtering and then close the file handle.

We would love to have some feedbacks here, and to have people test it. Again, it is still beta, so be aware 🙂

Note: Idexing pcap files is nothing new, the sancp project did add alike features, but was never properly released.

Advertisement

Suricata 1.0.2 Debian/Ubuntu package

In stead of compiling Suricata over and over again on different hosts I have… I just made a debian package for my Ubuntu Lucid 10.04 systems.

Its a simple build, and Ill hopefully update it with time to incorporate different usage and install help etc.
Right now its just aimed at being a simple IDS using libpcap.

You can find suricata and other cool NSM stuff at my gamelinux PPA found here.

apt-get install suricata
cd /etc/suricata/ && wget http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz
vim /etc/default/suricata
vim /etc/suricata/suricata.yaml
/etc/init.d/suricata start

Feedback and thoughts are welcome and needed 🙂 !

FPCGUI -> OpenFPC

In early June, Leon Ward and I teamed up in Oslo chatting about his OpenFPC and my FPCGUI project. I met Leon for the first time in April 2009 at Sourcefires offices in Wokingham, UK, and I have chatted with him now and then on IRC etc. since then.

I started using Sourcefire 3D in December 2008, and the first thing I was missing was the lack of pcaps from the events that I got. The second was the real-time view that you get in Sguil (I can live without that though).

So I needed a second host that did full packet capture along side my new IPS/IDS. Just running tcpdump/daemonlogger/sancp is OK for a small installation, but carving out the sessions manually was taking time. I needed to script something that would take an easy interface, so I could quickly get a pcap from the whole sessions I was getting events from. So I was thinking of an API and a easy way to add this to the Sourcefire 3D WebGUI.

My PoC was FPCGUI (Full Packet Capture Graphical User Interface). It can take a query in the URL, search the flow data from its database and give you the sessions details if it exists. If you click on the session, you will get the pcap served straight in you face, and I choose to open my pcaps with wireshark. With a little grease monkey magic, this would have been an OK solution for satisfying my pcap needs working with SF3D.

I made my thoughts public in a blog post in September 2009 and started coding right away. I also discussed FPCGUI with Leon the day after I posted the blog. The first release that worked good enough for me was in January 2010. Leon released his project in May 2010, and I quickly saw that we where doing more or less the same. He had implemented the distributed node part, which I had not started to even draft, and I had the WebGUI and flowdata which gives more meaning and is more user friendly to the analyst.

So, instead of working on two separate projects (aiming for the same goal), we decided to join forces and merge the two projects. And as I thought that OpenFPC is a better name than FPCGUI, OpenFPC it is 🙂

I have merged my parts slowly into OpenFPC during the summer, with vacation time and changing job, I did not have much time for coding on the side. We also re-factored much of the code, file names etc., so getting thing to a working condition has been the main task.

As of the last weeks, I can now install OpenFPC and use it in the way that I want it again, like I did with FPCGUI. The plus is that I now have a command line interface, a distributed architecture (Not WebGUI friendly yet), and a way to automagically extract pcaps and files in it, for automatic analysis 🙂

To test my dream of automatic analysis, I used a setup similar to this earlier blog post where I more or less did the same. I carve the pcap with openfpc-client (which will come from an event from an IDS or nftracker), extract files with tcpxtract (or simular tools), scan files with ClamAV and also test md5/sha sums towards shadowserver, virustotal or wepawet. I tried some different infected and non infected PDF files. All files I had to test with was detected with ClamAV, even my home grown metasploit PDF. All known bad PDF files was detected with the md5/sha sum of the files towards the different services (shadow/VT/wepawet), but again, only ClamAV detected my home made metasploit PDF.
evil.pdf: Heuristics.PDF.ObfuscatedNameObject FOUND

So, now I will have more events to live with 🙂

Ubuntu repo for sguil

I have spent the last week setting up a Ubuntu Launchpad PPA for my packages I used to hoste here on my blog.

The URL to my PPA is : https://launchpad.net/~ebf0/+archive/gamelinux

I pack the packages mainly for Lucid Lynx 10.04.
To try them out, you can add the following in /etc/apt/sources.list:
deb http://ppa.launchpad.net/ebf0/gamelinux/ubuntu lucid main
deb-src http://ppa.launchpad.net/ebf0/gamelinux/ubuntu lucid main

To add my key to you Ubuntu installation:
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4B04D050


Then you should be able to apt-get update, and then apt-get install my packages 🙂

Please try them out and give me feedback!
You will find my howto on how to configure them here.

Happy F8’ing!

Full Packet Capture GUI (FPCGUI)

I started a little project of mine that I have been thinking about since the summer of 2008 (Also see this post). I saw that it was a problem finding vendors selling a cheap setup for a full packet capture solution. The recommendation was to set up a Linux server on your own, run tcpdump and spool pcaps to disk. Well, once you have all that data, you need some way to manage it. I thought about using sancp to index the connections, and tools like tcpxtract, foremost, dsniff, chaosreader, tcptrace and combine features from xplico to add some extra value and possibilities on top.

So I started my project back in september 09, calling it FPCGUI (Full Packet Capture Graphical User Interface). It is currently supporting daemonlogger/tcpdump/sancp for spooling pcaps with a wrapper script that puts pcaps in directories based on “year-month-date”. cxtracker/sancp can be used for connection profiling/tracking, writing session data to disk, where I have written fpc-session-loader.pl which picks up the session data files and inserts them to a mysql database. If I now have an interest in seeing all the traffic from one host, I can do a search in my webgui and get the data. I can do rather interesting queries on all the data from cxtracker/sancp, and get interesting results.

I use cxtracker in my setup, as it collects meta data on both IPv4 and IPv6 connections. I have also managed to store IPv4 and IPv6 addresses in the mysql database in a reasonable and usable way.

I have just finished writing a PHP webgui, where I can enter a search term, and get a list (or just a single session if I’m specific enough), click on the session of choice, and up pops a download dialog, where I can choose to open the pcap straight away in wireshark! The pcap of the specific session is carved out from the pcaps for the relevant period (days) when the session took place. More or less the same functionality you find in a Sguil stack setup. I wrote the php-gui in such a way, that it can take search terms via an URL, like “?srcip=10.10.10.10&srcport=80” and so on, making it easier to integrate with other applications.

Example screenshot of what happens when you click on an event:

I have associated the pcap files with: ‘Content-Type: application/pcap-capture’ and set firefox to spawn wireshark for those files automatic 🙂

So now I’m one step closer to having Full Packet Capture with my Sourcefire 3D system! Just need to find out what part of the 3D webgui code to hack, to add my custom <click here to get the pcap of the session that triggered the event> 🙂 Of course I can enter the data manually, but I’m lazy, and I like to hack stuff 🙂

The project i hosted here. Any thoughts are more than welcome.

cxtracker update

I started out writing cxtracker and PRADS in perl, as my C fu was kinda rusty, and perl programmers claimed that perl was fast enough… It turned out, that perl was not fast enough for my purpose, but it added to my perl skillz at least.

So I wanted to rewrite cxtracker and PRADS in C, and the last three weeks, I have re-learned C and rewritten cxtracker. It has been many late hours and small bugs have been annoying me way too much. But I have had the chance to play more with debuggers and C which I find very interesting and pleasing.

cxtracker can now be used instead of sancp in a sguil setup. cxtracker is also meant to be used in another little project I have going, FPCGUI (Full Packet Capture GUI), but thats another blog entry soon to come.
cxtracker also logs IPv6 traffic, something that sancp does not. Sguil does not eat IPv6 yet, so to use cxtracker with sguil, a bpf filer for not inspecting IPv6 traffic should be used.

On my sensors, cxtracker seems to use about 20-30% lesser CPU time than sancp, which makes me really happy! The memory footprint is the same as sancp.

The reason I started with cxtracker as my first C project, is that it is a corner stone in my other projects, FPCGUI and PRADS. In FPCGUI, cxtracker is used to track IP sessions and storing them to a DB. In PRADS, it also tracks sessions, so one can limit the amount of data to check for service signatures in (Just check xx first packets, or xxx bytes of data after initial connection).

cxtracker is hosted on github : http://github.com/gamelinux/cxtracker

git clone git://github.com/gamelinux/cxtracker.git

It uses libpcap and I have yet only tried to compile it on Ubuntu and Debian machines (x86 and x86_64).

To test it:

# libpcap and a build environment is needed.

$ git clone git://github.com/gamelinux/cxtracker.git
$ cd cxtracker/src/
$ make
$ ./cxtracker -h

USAGE:
$ cxtracker [options]

OPTIONS:

-i : network device (default: eth0)
-b : berkeley packet filter
-d : directory to dump sessions files in
-u : user
-g : group
-D : enables daemon mode
-h : this help message
-v : verbose

$ ./cxtracker -i eth0 -D -d /nsm_data/sensor-hostname/sancp/ -u nsm -g nsm -b ‘ip’

If you try out cxtracker, feedback is more than welcome!

Binaries for Incident Response/Forensic analysis

I did a forensics job in December 08, running upon the problem that my static compiled binaries did not work on my target system. I made static bins that I needed for the job, but it took some time, so I noted a better way to do it from a sans blogpost. I made a quick and dirty script to prepare my forensic media, and have used it like that since.

A fellow GCFA posted his version yesterday of a script, and I merged it with mine and I plan to make more .iso’s and you will find them under: http://download.gamelinux.org/InsidentResponse/

The iso’s has the format:
Forensics-dynamic-bins-<Distro>-<Version>-<Arch>.iso

PTK – an alternative advanced interface for `The Sleuth Kit`

I have always used Autopsy and The Sleuth Kit when diving into hard drives, usb sticks and memory cards ect. It works ok for me, and gets my job done fairly well. But the Autopsy web GUI gives one a feel of being at the end of the 90’is… I’m exited to know, that PTK now has been released in 1.0 version which seems to add new life to TSK, and free forensics… The only drawback is.. that PTK is distributed under “PTK License”, claiming to be free software, which its clearly not!.. It should have been GPL or BSD licensed…

One day in the future, when I got the time… I would love to check it out more in dept though…