gulp - tcpdump alternative for lossless capture on Linux
tcpdump and wireshark are the tools that usually come to mind when you have to capture network traffic. But in some situations where you have to record a large amount of data and you want to avoid losing packets, tcpdump has some limitations. When I was hit myself by the tcpdump packet loss problem, I quickly found out that I was not alone and that a number of people had already researched the topic and/or provided alternatives.*
In particular, I found two different tools to perform the task: Corey Satten's gulp (http://corey.elsewhere.org/gulp/) and lindump from HP Labs (http://tesla.hpl.hp.com/opensource/)
I also found two interesting papers about capturing high volumes of traffic: http://www.usenix.org/events/fast09/tech/full_papers/anderson/anderson_html/ and http://docs.di.fc.ul.pt/jspui/bitstream/10455/3299/1/thesis-nhenriqu.pdf (the second quotes the first one among others, and also contains useful info to optimally spread the load among different cores)
After some tests I quickly became a happy gulp user, and thanks to the software being open source I was able to add features to it that I missed from the latest tcpdump versions:
-n - allows to change the default filename template
-t - allows to add a timestamp to the filename
-G - rotate pcap file every n seconds
-F - allows to skip the check for an ethernet interface
-Z - allows to specify a command to post-process each capture file
I've sent a patch to Corey Satten, who intends to setup a repository to hold the various contributions he gets for gulp. In the meanwhile, you can find my changes in the attached file (02-gulp-ntGFZ.patch.gz). For your convenience and for completeness, I also provide here the patch from Guy Harris that fixes issues on 64 bit systems (see http://seclists.org/wireshark/2009/Oct/105, apply that one first).
Updates:
2012-08-22: new version of my patch to fix the issue reported by SgtMalicious
2017-02-03: long-standing bug fixed
Downloads:
01-gulp-amd64.patch.gz: fix issues with 64-bit systems
02-gulp-ntGFZ.patch.gz: additional functionality as described above
gulp-1.58-crox.tgz: source with both patches applied
* other people have reported a performance drop with libpcap version 1.0 compared to previous builds, see http://thread.gmane.org/gmane.network.tcpdump.devel/4629 or http://seclists.org/tcpdump/2010/q3/index.html#11
In particular, I found two different tools to perform the task: Corey Satten's gulp (http://corey.elsewhere.org/gulp/) and lindump from HP Labs (http://tesla.hpl.hp.com/opensource/)
I also found two interesting papers about capturing high volumes of traffic: http://www.usenix.org/events/fast09/tech/full_papers/anderson/anderson_html/ and http://docs.di.fc.ul.pt/jspui/bitstream/10455/3299/1/thesis-nhenriqu.pdf (the second quotes the first one among others, and also contains useful info to optimally spread the load among different cores)
After some tests I quickly became a happy gulp user, and thanks to the software being open source I was able to add features to it that I missed from the latest tcpdump versions:
-n - allows to change the default filename template
-t - allows to add a timestamp to the filename
-G - rotate pcap file every n seconds
-F - allows to skip the check for an ethernet interface
-Z - allows to specify a command to post-process each capture file
I've sent a patch to Corey Satten, who intends to setup a repository to hold the various contributions he gets for gulp. In the meanwhile, you can find my changes in the attached file (02-gulp-ntGFZ.patch.gz). For your convenience and for completeness, I also provide here the patch from Guy Harris that fixes issues on 64 bit systems (see http://seclists.org/wireshark/2009/Oct/105, apply that one first).
Updates:
2012-08-22: new version of my patch to fix the issue reported by SgtMalicious
2017-02-03: long-standing bug fixed
Downloads:
01-gulp-amd64.patch.gz: fix issues with 64-bit systems
02-gulp-ntGFZ.patch.gz: additional functionality as described above
gulp-1.58-crox.tgz: source with both patches applied
* other people have reported a performance drop with libpcap version 1.0 compared to previous builds, see http://thread.gmane.org/gmane.network.tcpdump.devel/4629 or http://seclists.org/tcpdump/2010/q3/index.html#11
