tcpdump and
wireshark are the tools that usually come to mind when you have to capture network traffic. But in some situations where you have to record a large amount of data and you want to avoid losing packets, tcpdump has some limitations. When I was hit myself by the tcpdump packet loss problem, I quickly found out that I was not alone and that a number of people had already researched the topic and/or provided alternatives.*
In particular, I found two different tools to perform the task: Corey Satten's gulp (
http://corey.elsewhere.org/gulp/) and lindump from HP Labs (
http://tesla.hpl.hp.com/opensource/)
I also found two interesting papers about capturing high volumes of traffic:
http://www.usenix.org/events/fast09/tech/full_papers/anderson/anderson_html/ and
http://docs.di.fc.ul.pt/jspui/bitstream/10455/3299/1/thesis-nhenriqu.pdf (the second quotes the first one among others, and also contains useful info to optimally spread the load among different cores)
After some tests I quickly became a happy gulp user, and thanks to the software being open source I was able to add features to it that I missed from the latest tcpdump versions:
-n - allows to change the default filename template
-t - allows to add a timestamp to the filename
-G - rotate pcap file every n seconds
-F - allows to skip the check for an ethernet interface
-Z - allows to specify a command to post-process each capture file
I've sent a patch to Corey Satten, who intends to setup a repository to hold the various contributions he gets for gulp. In the meanwhile, you can find my changes in the attached file (
02-gulp-ntGFZ.patch.gz). For your convenience and for completeness, I also provide here
the patch from Guy Harris that fixes issues on 64 bit systems (see
http://seclists.org/wireshark/2009/Oct/105, apply that one first).
Updates:
2012-08-22: new version of my patch to fix the issue reported by SgtMalicious
2017-02-03: long-standing bug fixed
Downloads:
01-gulp-amd64.patch.gz: fix issues with 64-bit systems
02-gulp-ntGFZ.patch.gz: additional functionality as described above
gulp-1.58-crox.tgz: source with both patches applied
* other people have reported a performance drop with libpcap version 1.0 compared to previous builds, see
http://thread.gmane.org/gmane.network.tcpdump.devel/4629 or
http://seclists.org/tcpdump/2010/q3/index.html#11