daemonlogger, forensics, fpcgui, Information, Linux Distributions, Metasploit, OpenFPC, OpenSourceSoftware, Security, Ubuntu

FPCGUI -> OpenFPC

In early June, Leon Ward and I teamed up in Oslo chatting about his OpenFPC and my FPCGUI project. I met Leon for the first time in April 2009 at Sourcefires offices in Wokingham, UK, and I have chatted with him now and then on IRC etc. since then.

I started using Sourcefire 3D in December 2008, and the first thing I was missing was the lack of pcaps from the events that I got. The second was the real-time view that you get in Sguil (I can live without that though).

So I needed a second host that did full packet capture along side my new IPS/IDS. Just running tcpdump/daemonlogger/sancp is OK for a small installation, but carving out the sessions manually was taking time. I needed to script something that would take an easy interface, so I could quickly get a pcap from the whole sessions I was getting events from. So I was thinking of an API and a easy way to add this to the Sourcefire 3D WebGUI.

My PoC was FPCGUI (Full Packet Capture Graphical User Interface). It can take a query in the URL, search the flow data from its database and give you the sessions details if it exists. If you click on the session, you will get the pcap served straight in you face, and I choose to open my pcaps with wireshark. With a little grease monkey magic, this would have been an OK solution for satisfying my pcap needs working with SF3D.

I made my thoughts public in a blog post in September 2009 and started coding right away. I also discussed FPCGUI with Leon the day after I posted the blog. The first release that worked good enough for me was in January 2010. Leon released his project in May 2010, and I quickly saw that we where doing more or less the same. He had implemented the distributed node part, which I had not started to even draft, and I had the WebGUI and flowdata which gives more meaning and is more user friendly to the analyst.

So, instead of working on two separate projects (aiming for the same goal), we decided to join forces and merge the two projects. And as I thought that OpenFPC is a better name than FPCGUI, OpenFPC it is 🙂

I have merged my parts slowly into OpenFPC during the summer, with vacation time and changing job, I did not have much time for coding on the side. We also re-factored much of the code, file names etc., so getting thing to a working condition has been the main task.

As of the last weeks, I can now install OpenFPC and use it in the way that I want it again, like I did with FPCGUI. The plus is that I now have a command line interface, a distributed architecture (Not WebGUI friendly yet), and a way to automagically extract pcaps and files in it, for automatic analysis 🙂

To test my dream of automatic analysis, I used a setup similar to this earlier blog post where I more or less did the same. I carve the pcap with openfpc-client (which will come from an event from an IDS or nftracker), extract files with tcpxtract (or simular tools), scan files with ClamAV and also test md5/sha sums towards shadowserver, virustotal or wepawet. I tried some different infected and non infected PDF files. All files I had to test with was detected with ClamAV, even my home grown metasploit PDF. All known bad PDF files was detected with the md5/sha sum of the files towards the different services (shadow/VT/wepawet), but again, only ClamAV detected my home made metasploit PDF.
evil.pdf: Heuristics.PDF.ObfuscatedNameObject FOUND

So, now I will have more events to live with 🙂

Standard
cxtracker, daemonlogger, forensics, fpcgui, Information, OpenSourceSoftware, Security, Sguil, Snort, Suricata

Full Packet Capture GUI (FPCGUI)

I started a little project of mine that I have been thinking about since the summer of 2008 (Also see this post). I saw that it was a problem finding vendors selling a cheap setup for a full packet capture solution. The recommendation was to set up a Linux server on your own, run tcpdump and spool pcaps to disk. Well, once you have all that data, you need some way to manage it. I thought about using sancp to index the connections, and tools like tcpxtract, foremost, dsniff, chaosreader, tcptrace and combine features from xplico to add some extra value and possibilities on top.

So I started my project back in september 09, calling it FPCGUI (Full Packet Capture Graphical User Interface). It is currently supporting daemonlogger/tcpdump/sancp for spooling pcaps with a wrapper script that puts pcaps in directories based on “year-month-date”. cxtracker/sancp can be used for connection profiling/tracking, writing session data to disk, where I have written fpc-session-loader.pl which picks up the session data files and inserts them to a mysql database. If I now have an interest in seeing all the traffic from one host, I can do a search in my webgui and get the data. I can do rather interesting queries on all the data from cxtracker/sancp, and get interesting results.

freebsd search

I use cxtracker in my setup, as it collects meta data on both IPv4 and IPv6 connections. I have also managed to store IPv4 and IPv6 addresses in the mysql database in a reasonable and usable way.

IPv6 search

I have just finished writing a PHP webgui, where I can enter a search term, and get a list (or just a single session if I’m specific enough), click on the session of choice, and up pops a download dialog, where I can choose to open the pcap straight away in wireshark! The pcap of the specific session is carved out from the pcaps for the relevant period (days) when the session took place. More or less the same functionality you find in a Sguil stack setup. I wrote the php-gui in such a way, that it can take search terms via an URL, like “?srcip=10.10.10.10&srcport=80” and so on, making it easier to integrate with other applications.

search1

Example screenshot of what happens when you click on an event:
search1
I have associated the pcap files with: ‘Content-Type: application/pcap-capture’ and set firefox to spawn wireshark for those files automatic 🙂

So now I’m one step closer to having Full Packet Capture with my Sourcefire 3D system! Just need to find out what part of the 3D webgui code to hack, to add my custom <click here to get the pcap of the session that triggered the event> 🙂 Of course I can enter the data manually, but I’m lazy, and I like to hack stuff 🙂

The project i hosted here. Any thoughts are more than welcome.

Standard
daemonlogger, fpcgui, Information, OpenSourceSoftware, Security, Sguil, Snort

Full packet capture…

I was on a seminar today, where one of the key focus was full packet capture of network traffic.

It was rather strange to me, that it seem to be presented as something new, exiting and “must have”…

IDS/IPS without full packet capture – is time consuming if you try to investigate an incident. All analysts knows that, and there is nothing new about that. Richard Bejtlich has preached this for years ( Read Tao of Network Security Monitoring, Beyond Intrusion Detection ).

As a happy Sguil user, I always have full packet capture of my network traffic, and can drill down in all the network data from an event. Meaning that I save tons of time investigating events, and can better tune down my false positives also. Most commercial vendors don’t integrate any “full packet capture appliances”, and don’t even support 3rd parties packet capture services. In my earlier days, I brought this to among IBM and Juniper, where they just look strangely at me and replied more or less the same – “Full packet capture is just to much data to handle… you need big disk and lots of CPU/RAM… We are not sure how to integrate this…”

Well, there is a free and open source way to implement such a device. A standard Linux host with daemonlogger is one example. (There are other tools that also does packet capture, but daemonlogger really aim just at packet capture, and nothing more, and does it in a way that I want it.)

Now that you can get 67 terabyte of storage for about $7,800 USD, there should not be a problem storing your data 🙂

You can split up sguil to run different services on different hardware, so if you have a Network Tap that can mirror traffic to more than one devices, you can run IDS on one server, pcap on another, network statistics on a third and asset detection on a forth example vis. Basic overview of Sguil with all services running on one sensor:
DUAL HDD NAS
If you want to, or need more juice from your snort sensor etc. you can split it up, so that one sensor takes the traffic from X most used services, and the other sensor take the rest. Or even split it up more!

Since I started using Sourcefire 3D system, I have planed to make a way for me to easily integrate my package capture server with the Defense Center. My thoughts are on using Firefox with Greasemonkey and some perl-cgi on the pcap server to carve out the the right portion of the pcaps. Capture has some nice ideas and I might reuse some code from there. If Sourcefire don’t beat me to it, I might have something of my own in a near future…

If you don’t capture packets today, you should look into a way of doing it. It saves you time, and it saves you lots of work. I would not be without mine 🙂

Standard