Twenty months ago, I wanted to try something out, and made a proof of concept in just 3 hours (thats including the time it took to buy the domain networktotal.com and point it to the IP!).
The concept was to have a place where one could upload a pcap, and have Snort and Suricata (and other tools), with all the different rule sets (VRT-Registered, VRT-Subscription, ET Open and ET PRO), parse through a pcap and display the result. Other engines like Bro could also be added etc…
It would be what VirusTotal is for binaries, just for pcaps…. I did not announce my project anywhere, and I just kept it private, but some of my friends knew about the place. So during the the first year, I counted around 250 pcaps uploaded from different places around the world (not bad for not announcing it!).
My personal usage of networktotal.com was more for taking random pcap at different places and uploading it to look for infections easy.
First off, I liked the NetworkTotal idea, but keeping the IDS engines up2date and the rulesets up2date took some time. Also, the way Suricata and Snort used to work, made it so that it took like 5 to 10 minutes to process a pcap on the old hardware that NT is running on (You needed to start and stop the engines for each pcap, and just starting them up takes some time). So when Suricata went stable with their unix-socket support, meant that it would take me around a second to process a pcap with Suricata on my old hardware (Still 5 minutes with Snort). My subscription for VRT-Subscriber rules also ran out, so I dropped Snort for this setup, and I might add it back one day, but I would need more powerful hardware and a new VRT-S license…
But I started to develop the NetworkTotal backend code more on a host I set up at home. There where many goals with this, among learning about different NoSQL/Document stores which I find necessary in today’s world of data. I stared out with Cassandra, but needed fulltext, so I tried ElasticSearch (ES), but had some trouble using the Ruby Tire to do what I wanted. I ended up doing a MongoDB setup along side the ES setup, and kept that for about 8 months. But now I got ES to behave like I want to, but I still got lots to learn about ES.
Anyways, I have now ported NetworkTotal.com to only use Suricata with the Emerging Threats PRO rules. Processing of a pcap should normally take around a second and the events are stored in ElasticSearch. For all the malware I process at home with my cuckoo setup, I send the pcap in the form of <md5sum of the malware>.pcap to NetworkTotal.com, so you can search for events generated by a malware on NetworkTotal.com. As I don’t have storage to save all the pcaps, I do save some metadata in ElasticSearch. Mainly because I want to search up malware I upload based on events that fired, IPs that it talked to, domains it resolved, User-agent or URIs. This also uses disk space, so I wounder how long I can keep doing that too. Luckily, ElasticSearch scales linearly, so I’m hoping that will save me somehow…
Virustotal has also began to run pcaps through Snort and Suricata, but its not always that a malware produces a pcap, so there will be times that there are no events etc…
Here is an example from VT: https://www.virustotal.com/en/file/a7a526177c4475d47cdd49ea1b3ead2b/analysis/ where there is no snort or suricata events from the network traffic. You can search for the same md5sum on NetworkTotal.com where there is a rule that tags this as SpyEye fires : http://www.networktotal.com/search.php?q=a7a526177c4475d47cdd49ea1b3ead2b
Example of the search bash-script output:
$ nt-search.sh a7a526177c4475d47cdd49ea1b3ead2b
[i] Result URL: http://networktotal.com/search.php?q=a7a526177c4475d47cdd49ea1b3ead2b&json=1
“[1:2012686:4] ET TROJAN SpyEye Checkin version 1.3.25 or later“,
“[1:2010706:8] ET USER_AGENTS Internet Explorer 6 in use – Significant Security Risk”,
“[1:2406376:303] ET RBN Known Russian Business Network IP (189)”
I still have some more stuff to fix in the backend code, but having started to write this blog post for about 18 months ago, I figured Ill just blog it, and write updates as they come 🙂
I’ll see if I can make my cuckoo module more generic and share that, ( it uploads the <malware-md5sum>.pcap to NT and gets back the events that fired).
Again, I hope this is useful for anyone out there. Suggestions on improvements are also appreciated 🙂 Im also open to discuss what more data could be displayed publicly without showing off information from the uploaded pcaps that can be somehow sensitive (like Microsoft registration keys encoded in the URI of a malware checkin, or the name of a company etc.).