Information, OpenSourceSoftware, PADS, PRADS, Security

PRADS, and how it compares to pads and p0fv2 and p0fv3

The question was brought up to me late last night on IRC, as p0fv3 RC was recently announced. This is a short answer to that question:

“People that find the PRADS page and already know p0f or pads may be interested in a comparison or essentially arguments why you would use one over the other.”

First off, its exiting to see Michal Zalewski back with p0fv3 🙂 I quickly read through his code yesterday and tested it out, and its rather interesting how he solves things. The fingerprint database at the moment is limited, but expect that to grow in the near future. I also love his non formal output in his applications 🙂

[PRADS vs PADS]
So, back to the questions. First off, pads “Passive Asset Detection System” uses regexp syntax to look for common bytes in payload to identify server application. So if the server says ” Server: Apache/2.2.3 (Linux/SUSE)” that is collected as what service is running on the server port where this was detected. The “rules” can be written more specifically for each server software, but are rather general and small today. Some pads “rules” look for ASCII strings, and some for different bytes in hex etc. to identify stuff like SSL/TLS. Pads is no longer actively developed by the original author, but I do maintain a fork of the last version with enhancements added.

PRADS extended the way pads does asset detection. We have build in IPv6 support in PRADS, so it also detects asset listening on IPv6 addresses. We also have build in connection tracking, so that we can cut off detection in a stream after an amount of packets or bytes seen from the client or server. This to drop trying to look for server/client assets in connections that transfers big files or are encrypted etc. Most “banners/identifiers” are in the first packet etc. so limiting for how many packets in a stream to do detection on helps on performance etc.

To extend pads a bit, we also added detection for client applications using the same method as for detecting server applications.

My future thoughts on enhancing the pads/PRADS asset rules are to make them more like the Snort/Suricata rule language and use fast pattern matching before invoking the pcre method etc. Pads does no OS fingerprinting per say btw.

[PRADS vs p0f]
PRADS tcp fingerprinting was based on the p0fv2 way as p0f had the fingerprint DB and we thought that reusing the fingerprints would make it easier for people to migrate if they wanted, instead of recollecting and adding fingerprints. PRADS also added some touches of its own (for IPv6 etc) and the way we match the fingerprints (and fuzzing). We have thought about extending the fingerprints and re-write them, but thats in the future. Right now they are doing a good job. We also added all the p0fv2 ways of fingerprinting to the whole tcp session, from the syn to the rst/fin. p0fv2 could just use one method at a time, depending on how you started p0fv2. PRADS outputs all the info it gathers, and leaves the final correlation to the end user/program etc. A good example on that is prads-asset-report and prads2snort, which ads wight to each type of fingerprints, ranging the syn and syn+ack higher than stray-ack, rst and fin etc. You can also base the final guess on client or server applications to, say if the User-Agent contains: “Linux” or “Windows NT 6.1” or “Macintosh; Intel Mac OS X 10.7” etc.
or if the Server string of the web server is: “Microsoft-IIS 6.0” or “Apache 2.2.15 (FreeBSD)” or “Apache 2.2.3 (Red Hat)” etc.

The p0fv3 tcp fingerprints are new in the way they are written. A new fingerprint file format, that makes it easy to add different types of fingerprints into one and same file (TCP/HTTP/SMTP etc). The most significant enhancement in the TCP fingerprints that I see is the MSS and MTU multiplier field. p0fv3 also detects new quirks not measured in p0fv2. The rules are now also more human readable, Example:

# RULE
label = s:unix:Linux:2.6.x
sig = *:64:0:*:mss*4,6:mss,sok,ts,nop,ws:df,id+:0

# Will match:
.-[ X.X.X.X/58435 -> Y.Y.Y.Y/22 (syn) ]-
|
| client = X.X.X.X/58435
| os = Linux 2.6.x
| dist = 9
| params = none
| raw_sig = 4:55+9:0:1460:mss*4,6:mss,sok,ts,nop,ws:df,id+:0
|
`—-

The way the tcp fingerprints are matched are also changed a bit, and I believe Michal Zalewski has done this for good reasons and that it will enhance the detection.

Beside the new tcp fingerprint changes, p0fv3 also has application layer detection added. I looked at the HTTP stuff, and p0fv3 matches also on the HTTP header order and dont blindly trust the User-Agent, as we do in PRADS. We have thought about extending the “rule/signature” in PRADS to be more Snort/Suricata like, so you can have more content matches etc, but more accuracy can be achieved today using the pcre language, to verify header order etc, before blindly trusting the UA, but pcre is way too expensive used alone I think, so organizing the signatures/rules better internally and having something like a fast_pattern matcher would help alot. Quick pcre example for a User-Agent with specific HTTP header order:

# Detects Firefox/3.6.X with HTTP header order to add confidence in the match.
# PRADS rule:
http,v/MFF 3.6.X/$1//,rnHost: .*rnUser-Agent: Mozilla/5.0 (.*Firefox/3.6..*)rnAccept: .*rnAccept-Language: .*rnAccept-Encoding: .*rnAccept-Charset:

Running it in PRADS on an old pcap gives me:

# Client IPs deducted just to be kind
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.:80:6],[distance:8]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-GB; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox:80:6],[distance:11]
[client:MFF 3.6.X (Windows; U; Windows NT 5.1; de; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.:80:6],[distance:10]
[client:MFF 3.6.X (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR :80:6],[distance:14]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Linux Mint/10 (Julia) Firefox/3:80:6],[distance:15]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox:80:6],[distance:9]
[client:MFF 3.6.X (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12:80:6],[distance:6]
[client:MFF 3.6.X (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6:80:6],[distance:12]
[client:MFF 3.6.X (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12:80:6],[distance:14]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.10) Gecko/20101005 Fedora/3.6.10-1.fc14 Firefox/3.:80:6],[distance:8]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.:80:6],[distance:12]

Not the whole User-Agent is grabbed, and we need to extend that in the future. But the pcre language makes it possible to match on as much content as you want, to have the confidence you need in your signatures/rules for detecting assets. PRADS looks for client and server applications on all ports and both UDP and TCP and for IPv4 and IPv6.

[PRADS vs The World]
Right now we are working on adding the DHCP OS fingerprinting and ICMP OS fingerprinting. DHCP is pushed to the git master on github but is not fully integrated into the PRADS core yet, but printing and matching is working, so you can help add fingerprints if you want :). The ICMP part is tricky as I want to fingerprint on the protocol layer, and also the payload, so I kind of have to combine the p0f way with the pads way of detecting and matching.

PRADS has also lots of other stuff, like connection tracking/Flow gathering with output compatible with cxtracker and sancp. I have also been working on my passivedns project, and I tend to port the relevant function over to PRADS, so we can have domain names mapped with assets to.

p0fv3 has an API so you can talk to it, to fetch relevant info about the IPs it knows about. I see p0fv3 with this functionality aimed at mail and web servers etc, to determine if this is spam or ham stuff coming its way, but you can use it in lots of cool ways.
I know PRADS is used in much the same way from people I have talked too. An example that Kacper put up can be found on http://prads.delta9.pl/. On the road map for upcoming PRADS releases, we have access to assets via shared memory. That will make it easier for extracting info from the running PRADS process that is current. PRADS also ships with prads2db.pl which parses a prads asset log-file and inserts the info to a DB so you can query it for info.

PRADS philosophy is something like: “If it can be detect passively, PRADS should probably do it.”

So if you are comparing for deciding which application to go for, I would say use them all, and correlate the the knowledge that each tool gives you. You can even add the output from the active fingerprinting tool nmap into the mix.

That said, much of my view on PRADS comes from that I use it in my Network Security Monitoring setup and from my wish to “know as much as possible about my assets”. If you have any wishes or suggestions, god or bad etc, feel free to contact us.

E

Advertisement
Standard
Information, OpenSourceSoftware, PRADS, Security, Snort

Populating Snorts host attribute tables with PRADS

It has been a long journey, but after about two years, I finally got a way to populate Snorts host attribute table, automagically(tm)!

When I started this, my first option was to use nmap to scan the network to populate the information. This was scratched, as my goal was to be non intrusive and always up to date (The minute a new host popped up, I want to know). Scanning 65535 ports times two for each of the hosts Im monitoring is not an options also… I started to look at the Open Source tools out there, on which to use to get the information from. As I was familiar with p0f and PADS, I saw that they could do the job, but they needed some band-aid to work together, and they where lacking active development… p0f has a DB patch/version, and I already had PADS hooked up in Sguil, so I had the info in a DB, but not in the way I wanted it. So I started out on a journey to merge the two projects, enhance them, and try to speed things up a bit.

The project is still in development, but the main parts are done. It is useful in the way that it will print out information about detected hosts, like this in verbose mode (And yes, it also does IPv6):

2a02:c0:1002:100:21d:72ff:fe92:728,[syn:S4:64:1:40:M1440,S,T,N,W7:Z],[Linux:2.6 (newer, 7) IPv6],[link:IPv6/IPIP],[uptime:2hrs],[distance:0]
2a02:c0:1002:10::2,[synack:5712:63:1:40:M1440,S,T,N,W7:ZAT],[Linux:2.6 (newer, 7) IPv6],[link:IPv6/IPIP],[uptime:4069hrs],[distance:1]
2a02:c0:1002:100:21d:72ff:fe92:728,[ack:45:64:1:*:N,N,T:ZAT],[Linux:2.6],[uptime:2hrs],[distance:0]
2a02:c0:1002:10::2,[service:OpenSSH 5.1p1 (Protocol 2.0):22:6],[distance:1]
2a02:c0:1002:10::2,[ack:45:63:1:*:N,N,T:ZAT],[Linux:2.6],[uptime:4069hrs],[distance:1]
2a02:c0:1002:100:21d:72ff:fe92:728,[client:OpenSSH 5.1p1 (Protocol 2.0):22:6],[distance:0]

At the moment, it also makes a file in your /tmp/ dir, /tmp/prads-asset.log, which presents the info in the following way:

2a02:c0:1002:100:21d:72ff:fe92:728,0,56268,6,SYN,[S4:64:1:40:M1440,S,T,N,W7:Z:Linux:2.6 (newer, 7) IPv6:link:IPv6/IPIP:uptime:2hrs],0,1269420770
2a02:c0:1002:10::2,0,22,6,SYNACK,[5712:63:1:40:M1440,S,T,N,W7:ZAT:Linux:2.6 (newer, 7) IPv6:link:IPv6/IPIP:uptime:4069hrs],1,1269420770
2a02:c0:1002:100:21d:72ff:fe92:728,0,56268,6,ACK,[45:64:1:*:N,N,T:ZAT:Linux:2.6:uptime:2hrs],0,1269420770
2a02:c0:1002:10::2,0,22,6,SERVER,[ssh:OpenSSH 5.1p1 (Protocol 2.0)],1,1269420770
2a02:c0:1002:10::2,0,22,6,ACK,[45:63:1:*:N,N,T:ZAT:Linux:2.6:uptime:4069hrs],1,1269420770
2a02:c0:1002:100:21d:72ff:fe92:728,0,22,6,CLIENT,[ssh:OpenSSH 5.1p1 (Protocol 2.0)],0,1269420770

Input from the community on how to present the information/output for a best possible way for integration with other applications are welcome.

To try it out, this is what I believe is necessary to install on my Ubuntu machine to run it:

$ sudo aptitude install build-essential git-core libpcre3-dev libpcap0.8-dev
$ git clone http://github.com/gamelinux/prads.git
$ cd prads/src/ && make
$ # then to start it
$ sudo ./prads -i ethX -v

For populating the Snort host attribute table, there is a script in the tools dir, prads2snort.pl, which takes the prads-asset.log file and processes it.
Example:

$ perl prads2snort.pl -i prads-asset.log -o hosts_attribute.xml -v -f

The -v (verbose) mode prints out some details, which can be good to check to see if stuff seems to be detected correctly.

Snort supports reloading of the attribute table if you give it the signal 30. (kill -30 <snort-pid>). This means, that if you discover a difference in your host attribute table (Say you got a new http service some where, or a host has changed OS), you can swap out the attribute file with a new updated one, and tell snort to reload its attribute file without restarting snort! Darn cool if you ask me 🙂

You can read more about Snort and its host attribute table here, and you can read about another tool called Hogger here. Also, you should read the Snort documentation section 2.7 (around page 104/105) “Host Attribute Table”.

I would once again like to thank Michal Zalewski and Matt Shelton for their work on p0f and pads. I would also like to thank Martin Roesch & The Snort Team (And all the contributers) for a great application and all the effort they have put into Snort and its surroundings. (Virtually giving you the price for best Open Source security application 2000 – 2010!).

Attribute Table Loaded with 980 hosts

Attribute Table Reload Thread Starting…
Attribute Table Reload Thread Started, thread 363022672 (15333)

$ /bin/kill -30 15333

Swapping Attribute Tables.

$ /bin/kill 15333

===========================================
Attribute Table Stats:
Number Entries: 980
Table Reloaded: 1
===========================================

Standard
Information, Linux Distributions, OpenSourceSoftware, PADS, PRADS, Security, Sguil, Ubuntu

My version of pads-1.2-sguil-mods

Saturday 18 Jun 2005 Matthew J. Shelton released PADS. PADS is a great tool, and the security industry really needs a good open source passive asset tool. But since 2005, PADS development has stopped, and there are no place to send new signature or patches/bugs too, and hope that they will get added/fixed. Also, logical, there are no new features being added…

I have used PADS in my Sguil setup, but have seen that it lacks stuff that I wanted to have there, and also, there has been some problems running PADS on newer operation systems. I have a copy of the pads-1.2-sguil-mods.tar.gz, and I added it to github yesterday, fixed some issues when writing data to the FIFO file for Sguil, added some patches, among vorants vlan patch. I compiled it on Ubuntu Hardy and Jaunty (x86_64), and it has been running fine for 12+ hours.

If you try out my version of PADS and have issues, I will try to solve them. I see there are some, in stuff that I don’t use, and if I one day find the urge, I’ll fix them on my own.

I should probably also mention, shamelessly again, that there is a project that takes PADS to the next level and then some more….
You can read about PRADS here and what more it can do for you.

Standard
OpenSourceSoftware, PRADS, Security

PRADS status update…

PRADS is finally getting somewhere!

Since last time, Kacper Wysocki has joined the project, working mostly on the OS SYN/SYN-ACK fingerprinting and the DBI implementation and being a great resource in all aspects of the project.

So, at this stage, PRADS now has implemented:

* OS fingerprinting, both SYN and SYN+ACK (IP/TCP) (Compatible with p0f fingerprints)
* TCP service fingerprinting (Signatures from/compatible with pads)
* TCP discovery of hosts (SYN and SYN+ACK)
* UDP discovery of hosts (Work to be done on UDP service detection)
* UDP OS fingerprinting
* ARP discovery of hosts
* MAC vendor fingerprinting (from ARP data)
* ICMP discovery of host
* ICMP OS fingerprinting
* perl DBI support (sqlite (default), MySQL, PostgreSQL, Oracle, MSSQL…)
* Daemon mode
* Some packet statistics (received,dropped,drop-rate and dropped by interface)

All this is done passively.

ICMP and UDP fingerprinting is not 100% accurate, but it gives an indication and therefor added for building up a higher confidence level for a “total” OS detection/fingerprinting.

One of my main goals, was to use PRADS with my Sguil setup. PRADS would be replacing PADS and also giving OS info to the Sguil console. Also I would like PRADS to populate the attribute_table for Snort automatically. See http://snort.org/docs/ for more info on the attribute_table and why you should need it.

PRADS was also intended to work separately, and now that we have support for different databases through perl DBI, it should be ready for a GUI (WebGUI maybe..). The GUI could extract the info, and maybe draw a “real-time” map of your network… First off I believe, would be the need for grouping stuff by OS, Services and IP/mask etc.

Some near future TODOs:
* TCP Service discovery/matching needs polishing for performance
* UDP Service detection (To the extent it is possible of doing so)
* Client application detection/fingerprinting
* Performance optimization all over
* Code clean up
* WebGUI or some GUI

We would also very much like help on verifying/adding fingerprints (SYN, SYN-ACK, ICMP and UDP). Testing and feedback on ideas/thoughts would also be much appreciated 😀

“Know your assets!”

Standard
Information, OpenSourceSoftware, PRADS, Security

PRADS – Passive Real-time Asset Detection System

In February 2008, I drafted a document (specs) on a program that I called PPADS back then (Perl Passive Asset Detection System). The program was thought to be a Perl implementation of PADS (Passive Asset Detection System) which is a program that listens to a network and attempts to provide an up-to-date look at the hosts and services running on the network.

My main goal was to use it with Sguil instead of PADS.

In August 2008 I implemented a proof of concept, and then in September Jan Henning Thorsen polished the code, and implemented the part that loads the PADS original service signatures.

The Output was now:
(MAC address of the asset that sent the IP packet) source_ip:port -> (MAC address of the asset that is receiving the IP packet) destination_ip:port -> Service identification string

(001f3caedaa9) 87.238.45.168:80 -> (001f3b938df8) 10.10.10.123:42753 -> Apache 2.2.3 Debian

But my initial draft of my document had more… If it says Debian.. and maybe even Etch… in the service string… It might even be a Linux server 🙂 So my draft also includes Operating System guessing and fingerprinting like p0f does.

Also, one can look at strings in client communication etc.:
“Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.6) Gecko/2009020911 Ubuntu/8.10 (intrepid) Firefox/3.0.6 Ubiquity/0.1.5”
“Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6”
So client fingerprinting is also on the road map.

I had some names in my head on what to call the project… It ended up with PRADS – Passive Real-time Asset Detection System.

At the time of writing, PRADS has implemented Service Detection, and a PoC of OS fingerprinting.
I am currently working on loading the original p0f syn-fingerprints and basing the detection on that.

Why I am I doing this ?
PADS is really cool…
p0f is really cool 🙂

But they can both be more, and I don’t see a good reason for not implementing such a great thing in one combined program. (And I believe that each time you fire up another “libpcap sniffer” on your sensor, it suck juice out of your sensor, so if you use the sensor to do IPS/IDS, your total capability lowers). Also, they seem to lack active development and signature/fingerprints updates.

I am no hardcore C/C++ hacker. Perl is easy and fast 🙂 And I want one agent to “rule them all”… meaning that I want one agent that will give me (and others) insight into what is going on in the network. Not 2,3,4…10…

The project is hosted on github, and people who might be interested in joining the project, are welcome 🙂

My dream right now, is to have a GUI that shows you an updated view of what clients(OS and Clients programs) and servers(OS and services) are running/beeing used on your network *right now*

PRADS in Action

Visibility is gold.

Standard