PassiveDNS version 1.0

Im happy to announce that my PassiveDNS has reach version 1.0 (stable)!

For those of you who has played with earlier versions, the biggest changes in the last tags is the log output format:

Old:
1341819126||1.2.3.4||8.8.8.8||IN||www.google.com.||A||173.194.32.7||300

New:
1341819126.845527||1.2.3.4||8.8.8.8||IN||www.google.com.||A||173.194.32.7||300||17

I added microseconds to the unix timestamps, and also added a count field (the last field). The count field outputs how many times it has seen a query answer since it last printed it as PassiveDNS if you use caching. If you run PassiveDNS with -P 0 (No caching), it should always output 1.

Running PassiveDNS with default options, it will look something like this for a domain:

1341500304.265705||1.2.3.4||8.8.8.8||IN||www.facebook.com.||A||69.171.247.21||45||1
…
1341779965.656576||1.2.3.4||8.8.8.8||IN||www.facebook.com.||A||69.171.247.21||107||11

This means that in the time PassiveDNS was running, a query for http://www.facebook.com. returned 69.171.247.21 12 times in total. 11 of the entries happened between the configured “print time”. ( -P Seconds between printing duplicate DNS info (default 86400). )

So if you have any custom tools for parsing the output, you probably need to update it, before you upgrade to v1.0. pdns2db.pl which you will find in the tools/ dir has patched to handle the change.

Now that v1.0 is out, I will work with releasing new versions of PassiveDNS. In versions to come, I will make it so that you can customize the output fields via the command line.

BTW, I have also added a bit more statistics when passivedns 1.0 ends. It looks something like this:

— Total DNS records allocated : 15726
— Total DNS assets allocated : 23259
— Total DNS packets over IPv4/TCP : 0
— Total DNS packets over IPv6/TCP : 0
— Total DNS packets over TCP decoded : 0
— Total DNS packets over TCP failed : 0
— Total DNS packets over IPv4/UDP : 222139
— Total DNS packets over IPv6/UDP : 0
— Total DNS packets over UDP decoded : 222133
— Total DNS packets over UDP failed : 6
— Total packets received from libpcap : 463374
— Total Ethernet packets received : 463374
— Total VLAN packets received : 0

You can download the 1.0 release in tar.gz or in zip.

Or you can find the project on github.

Version 1.0 has been tested extensively and should be considered stable and production ready. But if you find any issues, please don’t hesitate to report your findings here.

Hacky New Year by the way!

Advertisement

pdns-ui – by Philipp Hunold

A great thing about open source software, is that you can make something that works for you, and someone else might add stuff that works for them, and combined, you might have something all in all more powerful…

pdns.ui – A Minimalistic WebUI for PassiveDNS

phunold (Philipp Hunold) has made a webgui for my PassiveDNS 🙂 I cloned it on git and have it up and running here at home. I’m not a webcoder, so seeing that someone made a GUI for my PassiveDNS makes me happy! (As I would have spent too much time on doing it than it would be worth). I’ve emailed with Philipp and I know that pdns-ui is in an early stage, but I would like to let other people know about the UI so that they can use it instead of making their own and maybe come with suggestions on how to improve it, come with patches etc.

So for people who wants a web-frontend to their PassiveDNS DB, try it out and give the feedback to Philipp!

Big thanks Philipp 🙂

PassiveDNS 0.5.0

I have pushed PassiveDNS version 0.5.0.

According to the roadmap, I have been at 0.5.0 for a while, and even started to implement stuff for the 1.5.0 version. But my real aim is the 1.0.0 release, and I have started all the activities for the 1.0.0 release, but I lack the statistics that I set in the roadmap when PassiveDNS ends. I have played it against pcaps with DNS attacks, Im fuzzing pcaps being read by PassiveDNS etc. so a 1.0.0 is hopefully not that far away 🙂

Some of the changes since my last blog post (v0.2.9):

* Logging of NXDOMAINs (-Xx -L nxdomain.log)
* DNS over UDP/TCP on IPv4 and IPv6 (Used to be just IPv4+UDP)
* Logging to stdout (-L – / -l -), both for NXDOMAINS and other DNS records.
* Implemented some hardening, including checking that client TID match server TID etc.
* Other small optimization and fixing a small memleak etc.

The way I implemented NXDOMAINS in PassiveDNS for now, makes it compete with the memory pool from “normal” domains/records. So if you have a fastflux or someone just querying for generated b0gus domains on your network, you might push out valid domains from the cache in favor for a NXDOMAIN. The reason I did this, is that it was faster than implementing an own memory pool for the NXDOMAINS and it give the possibility to log NXDOMAINS in current version with out to much hassle. If this way of implementing NXDOMAINS turns out to fight for memory more aggressively than one would like, one can always start two instances of PassiveDNS, one just looking for NXDOMAINS, and the other one looking for the regular domains. As I gain more experience with NXDOMAINS in PassiveDNS and get more feedback, Ill reconsider the implementation if needed 🙂

One note, the current logfile format will be stable until the 1.5.0 release (that is my intention at least), After that, my plan is to implement a customizable log format, and also more fields of interest will be available. If you have any additional data that you want to output and thoughts about how the output for those data should be, don’t hesitate to let me know 🙂

I ran into a security related bug on my Ubuntu 10.04 which might be triggered running PassiveDNS. I have emailed the Debian package maintainer and reported the bug to security@ubuntu.com and also filed a bug report. The bug is fixed upstream in ldns long time ago, so hopefully it will be fixed soon in Ubuntu 10.04 too 🙂

For reporting issues or making feature request, please do so here.

Happy DNS Archiving 🙂

PassiveDNS 0.2.9

I added some features and changes to PassiveDNS. The most important change is that the output now contains the TTL value, so you need to use the current tools/* (if you use them) as they are also changed to work with this new output format (or update your own tools).

I also added the ability to specify the DNS record types that you want to log from the command line and I added support for more record types. PassiveDNS now should be able to track: A, AAAA, CNAME, DNAME, NAPTR, SOA, PTR, RP, SRV, TXT, MX and NS.

Support for chroot and dropping privileges are also added.

I also added some features to tools/pdns2db.pl while I was at it:
1) You can now process a passivedns.log file in “batch” mode, exiting when finished.
2) You can now specify a file with a list of domains or IPs to skip insertion to the DB.
3) You can now specify a file with a list of PCRE (Perl Compatible Regular Expressions) of “domains/IPs” to skip insertion to the DB.
4) You can now specify a file with a list of domains or IPs to alert on!
5) You can now specify a file with a list of PCRE of “domains/IPs” to alert on!
6) You can now specify a file with a list of domains to whitelist and not alert on.
7) You can now specify a file with a list of PCRE of “domains/IPs” to whitelist and not alert on.

The skiplists will be checked first, and if the domain/IP is found/matched there, whitelist and blacklist will be ignored and insertion to DB will be ignored.

Next the whitelists will be checked, and if a domain/IP is found there or match a PCRE that you have defined it will not be checked by the blacklist.

Last the blacklists is checked, and if a domain/IP is found there or match a PCRE that you have defined, it will write the PassiveDNS record to the alert file that you specify (Default: /var/log/passivedns-alert.log).

There are different sources for getting lists of known bad domains. Here is one if you want to test the blacklist functionality: http://isc.sans.edu/feeds/suspiciousdomains_High.txt

Im pretty far as what it comes to planed features at this stage. Please try out PassiveDNS and beat the crap out of it 🙂 I will probably “up” the version to 0.5.0 soon and from there on, it is just testing and testing and more testing before it will be a “one dot O” release.

If you have any issues with PassiveDNS, please submit them here.

PassiveDNS update (v0.2.4)

It has been some while since I had time to code on my C projects. But the last week I got some time and used it to get PassiveDNS into a state where Im more relaxed about it. Previous version (V0.1.1) used to spit out all DNS data it saw. The latest version caches DNS data internally in memory and only prints out a DNS record when it sees if for the first time, or if it is a active domain, it prints it out again after 24 hours and so on (once a day). The previous version would give me Gigabytes of DNS data daily in my test setup, while this version gives me about 2 Megabytes. This version also just gives you A, AAAA, PTR and CNAME records at the moment. I’m open for suggestions for more (use-cases would be great too!).

In my tests and in feedback from people who has tried it, PassiveDNS is very resource friendly when it comes to CPU usage (more or less idling). In current version (v0.2.4) there is not implemented any limitation on memory usage, so if your network sees a lot of DNS traffic, you might end up using some hundreds of Megabytes RAM for the internal cache. The most I’ve seen is around 100 MB at the moment. My plan is to implement some sort of “soft-limit” on memory usage, so that you can specify how much memory PassiveDNS should maximum use. The “downside” of this though, is that PassiveDNS would have to expire domains from its cache faster. That might end up in bigger log files with duplicate entries. When I say “downside”, its not a real downside as I see it. From my tests with the example scripts pdns2db.pl and search-pdns.pl, it is not much of a problem keeping up with insertions to the DB (MySQL) and your last seen timestamp will be a bit more accurate. I guess this kind of data though, is better suited for a NoSQL solution, if you are collecting lots of it.

If you have read this, and you are into Network Security Monitoring, and you don’t use passive DNS in your work, I recommend you too Google it and read a bit about it.

Passive DNS and PassiveDNS/PRADS

For those of you not familiar with the concept of Passive DNS, there are lots of stuff on it on the intertubes…

Just some of the links:
Some use cases: http://conferences.npl.co.uk/satin/presentations/satin2011slides-Rasmussen.pdf
A public passive dns db: http://www.bfk.de/bfk_dnslogger.html?query=sans.org#result
Or just click here: http://lmgtfy.com/?q=passivedns

I have not found any good tools yet that lets you build your own passive DNS DB, so I have started to walk down that path…
First off, I have coded a DNS sniffer (passivedns) I have ported the same functionality over into PRADS. All code is in beta at the moment.

I announce this release, so if anyone is interested, I will take input on the output format 🙂
My first tests shows that the passive DNS data collected on a small network is too much… My plan is to implement a in memory “state” so that it don’t prints out the same record more than X times over a time interval (say, if a record is the same, just print it once a day, but if it changes, print it immediate). When that is done, Ill write a parser to feed it into a DB and a query tool to fetch passive DNS records on request.

Feedback is always welcome!