Skip to content

gulp - tcpdump alternative for lossless capture on Linux

tcpdump and wireshark are the tools that usually come to mind when you have to capture network traffic. But in some situations where you have to record a large amount of data and you want to avoid losing packets, tcpdump has some limitations. When I was hit myself by the tcpdump packet loss problem, I quickly found out that I was not alone and that a number of people had already researched the topic and/or provided alternatives.*

In particular, I found two different tools to perform the task: Corey Satten's gulp (http://corey.elsewhere.org/gulp/) and lindump from HP Labs (http://tesla.hpl.hp.com/opensource/)

I also found two interesting papers about capturing high volumes of traffic: http://www.usenix.org/events/fast09/tech/full_papers/anderson/anderson_html/ and http://docs.di.fc.ul.pt/jspui/bitstream/10455/3299/1/thesis-nhenriqu.pdf (the second quotes the first one among others, and also contains useful info to optimally spread the load among different cores)

After some tests I quickly became a happy gulp user, and thanks to the software being open source I was able to add features to it that I missed from the latest tcpdump versions:

-n - allows to change the default filename template
-t - allows to add a timestamp to the filename
-G - rotate pcap file every n seconds
-F - allows to skip the check for an ethernet interface
-Z - allows to specify a command to post-process each capture file

I've sent a patch to Corey Satten, who intends to setup a repository to hold the various contributions he gets for gulp. In the meanwhile, you can find my changes in the attached file (02-gulp-ntGFZ.patch.gz). For your convenience and for completeness, I also provide here the patch from Guy Harris that fixes issues on 64 bit systems (see http://seclists.org/wireshark/2009/Oct/105, apply that one first).

Updates:
2012-08-22: new version of my patch to fix the issue reported by SgtMalicious
2017-02-03: long-standing bug fixed

Downloads:
01-gulp-amd64.patch.gz: fix issues with 64-bit systems
02-gulp-ntGFZ.patch.gz: additional functionality as described above
gulp-1.58-crox.tgz: source with both patches applied

* other people have reported a performance drop with libpcap version 1.0 compared to previous builds, see http://thread.gmane.org/gmane.network.tcpdump.devel/4629 or http://seclists.org/tcpdump/2010/q3/index.html#11
  • Twitter
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at del.icio.us
  • Facebook
  • Google Bookmarks
  • FriendFeed
  • Digg gulp - tcpdump alternative for lossless capture on Linux
  • Mixx gulp - tcpdump alternative for lossless capture on Linux
  • Bloglines gulp - tcpdump alternative for lossless capture on Linux
  • Technorati gulp - tcpdump alternative for lossless capture on Linux
  • Fark this: gulp - tcpdump alternative for lossless capture on Linux
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at YahooMyWeb
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at Furl.net
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at reddit.com
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at blinklist.com
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at Spurl.net
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at NewsVine
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at Simpy.com
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux at blogmarks
  • Bookmark gulp - tcpdump alternative for lossless capture on Linux with wists
  • wong it!
  • Bookmark using any bookmark manager!
  • Stumble It!
  • Print this article!
  • E-mail this story to a friend!
  • Identi.ca

Trackbacks

No Trackbacks

Comments

Display comments as Linear | Threaded

Don on :

Hi can you let me know how you are using this? I am performing the following:

gulp -i eth1 > pcapfile

which appears to be capturing and writing to a file called pcapfile but when I attempt to read:

tcpdump -r pcapfile
reading from file pcapfile, link-type EN10MB (Ethernet)
tcpdump: pcap_loop: bogus savefile header

Any ideas of what I could be doing wrong?

Thanks

crox on :

If I remember correctly, that's the error you get without the 64-bit patch: http://blog.crox.net/uploads/01-gulp-amd64.patch.gz

Don on :

Thanks crox, all good now. Your patch was awesome we now have tcpdump functionality which was what we were looking for.

SgtMalicious on :

There's no check in the patch to test against zcmd being null and passing this along to execlp which will cause segmentation faults. You should check that zcmd != NULL.

crox on :

You are right, I've uploaded a new version of the patch that fixes this issue. Thanks for reporting it.

Khi on :

"-G - rotate pcap file every n seconds" does not work because it produces error message like: (pcap: File has 538976288-byte packet, bigger than maximum of 65535)

Can you please send me your working version? Thanks.

crox on :

I can't remember having seen this issue - and I'm using "-G" a lot.

Which version of libpcap are you using?

This is the original source from Corey with patches applied: http://blog.crox.net/uploads/gulp-1.58-crox.tgz

ahat on :

I see some drops even with gulp. Any ideas why?

root@moat:/tmp# ./gulp -i eth0 -r 1024 > /var/log/magi/local.pcap
^C
422051 packets captured
422106 packets received by filter
55 packets dropped by kernel
ring buffer use: 0.0% of 1024 MB

crox on :

To begin with, I'd try using -o to write to disk instead of shell redirection, to check whether it makes a difference.

ghaleb on :

hallo
sorry for this question but how to run patch file??

crox on :

Hello ghaleb, see http://en.wikipedia.org/wiki/Patch_%28Unix%29

You can also download the full source with patches applied, see comment #2.1.1.1.

roetherb on :

FYI crox - your tarball spawns a new instance of gulp every 1s ad infinitum... I've tried everything I have time to try, but I can't offer you any explanation - sorry :-/

crox on :

Can you provide some details about the system you are using, like OS and libpcap version?

crox on :

[What appears to be the same issue should be fixed in the latest version of the patch.]

kk on :

i am new to Linux.
how can i install gulp?
I applied 64bit patch by "patch -i 01-gulp-amd64.patch" then "make" but getting error.
ubuntu:~/gulp$make
cc -g -0 gulp.c -o gulp -lpthread -lpcap
gulp.c:79:18 fatal error: pcap.h: no such file or directory
#include
^
complilation terminated.
make ***[gulp} Error 1

crox on :

You need to install the libpcap-dev package, eg like this:
$ sudo apt-get install libpcap-dev

tpz on :

Is there any way to change it from using epoch time to utc or local time when using the -t option?

crox on :

You could prepare a string containing the date/time in your favorite format with strftime() , and modify this line to include it in the file name:
snprintf(ofile, sizeof(ofile), "%s/%s%lld.%03d", dir, oname, (long long int)time(NULL), num);

tpz on :

Is there any way to make it so that it doesn't execute -Z when the program initially starts up?

crox on :

That's indeed a bug that should be fixed sometime. I work around it by calling a shell script with -Z where I check the arguments before executing the actual post-processing from the shell script.

tpz on :

Is there any way to make it so that for instance when I type -G 3600 that it will rotate the file every hour on the hour instead of just every hour from whenever I start the program? or if I type -G 900 it will rotate it every 15 minutes on the 15 minute mark instead of every 15 minutes from when I start the program?

crox on :

That would require changes to the code. If you need one pcap file per hour you could post-process the captured data with tcpslice.

ogd on :

i love the addition to the time, however i am not seeing the files rotate with -C 1000 when using -t. I actually modified it to do max_files = 10 however it doesn't rotate if I use the -t tag. When I take out the time(NULL) function it seems to work. was wondering if you could provide any help? Thanks!

Steveno on :

I'm running gulp on a linux server with the nohup and the "&" options to run a rotating packet capture for a specific subnet. I took a look at the server this morning to see how things were going and for the most part everything looked kosher however I'm seeing multiple instances of the process running. Is this normal? See my output down below.

This is how I launched gulp to run continuously:

nohup ./gulp -i bond0 -f "net 10.30.205.0/24" -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t &


ps -ef | grep gulp
root 1871 1 1 Oct18 ? 01:07:55 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 1874 1871 0 Oct18 ? 00:14:53 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3282 1871 0 Oct19 ? 00:05:24 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3762 1871 0 09:37 ? 00:01:23 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3778 1871 0 10:19 ? 00:01:14 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3789 1871 0 10:50 ? 00:01:06 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3802 1871 0 11:20 ? 00:00:59 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3825 1871 0 12:35 ? 00:00:41 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3841 1871 0 13:19 ? 00:00:30 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3852 1871 0 13:46 ? 00:00:25 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3861 1871 0 14:08 ? 00:00:19 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3874 1871 0 14:32 ? 00:00:13 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 3883 1871 0 15:03 ? 00:00:06 ./gulp -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
n3tus3r 3996 3974 0 15:27 pts/0 00:00:00 grep --color=auto gulp

Steveno on :

Hi,


I'm seeing odd behavior when running gulp with rotating packet captures in that once in awhile a file is created with minimal traffic but is saved as a 500MB file per my following command execution. For instance, when it works right, the pcap files contain millions of packets but when it flakes out it has between 200 - 300 packets. Has anyone experienced this and is there a fix? I'm running the patched gulp version on ubuntu 14.0.x. Any help would be very much appreciated.

Here's how I execute the gulp process. You'll notice I'm using nohup and the "&" to run my captures while away from my computer in order to investigate a problem we're experiencing with VoIP.

nohup ./gulp -x -i bond0 -f "net 10.30.205.0/24" -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t &


Not sure if it's related but when I check on the gulp process running, I see multiple instances.

Example:
ps -ef | grep gulp
root 7458 1 2 Oct22 ? 00:27:20 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 7461 7458 0 Oct22 ? 00:05:37 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 8270 7458 0 08:28 ? 00:00:58 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 8285 7458 0 09:15 ? 00:00:47 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 8302 7458 0 10:03 ? 00:00:36 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 8317 7458 0 10:36 ? 00:00:28 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 8330 7458 0 11:14 ? 00:00:19 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t
root 8348 7458 0 12:03 ? 00:00:08 ./gulp -x -i bond0 -f net 10.30.205.0/24 -r 500 -C 1 -W 500 -o captures/ -n 200LS- -t

crox on :

I have no experience running gulp with nohup. I usually use screen [1] to achieve the same result.

[1] https://www.gnu.org/software/screen/

crox on :

[What appears to be the same issue should be fixed in the latest version of the patch.]

Jishnu on :

Hi
I'm tryng to install GULP.I have installed libpcap:1386 but still make gives me the following error:
cc -O -m32 check64bit.c -o check64bit
./check64bit; rm -f check64bit
cc -g -O -m32 gulp.c -o gulp -lpthread -lpcap
gulp.c: In function ‘Reader’:
gulp.c:400:22: warning: ignoring return value of ‘read’, declared with attribute warn_unused_result [-Wunused-result]
if (tmpfd >= 0) read(tmpfd, (char )&fh, sizeof(fh));
^
gulp.c:436:24: warning: ignoring return value of ‘fscanf’, declared with attribute warn_unused_result [-Wunused-result]
if (procf) {fscanf(procf, "%d", &rmem_def); fclose(procf);}
^
gulp.c:438:24: warning: ignoring return value of ‘fscanf’, declared with attribute warn_unused_result [-Wunused-result]
if (procf) {fscanf(procf, "%d", &rmem_max); fclose(procf);}
^
gulp.c: In function ‘newoutfile’:
gulp.c:493:11: warning: ignoring return value of ‘fchown’, declared with attribute warn_unused_result [-Wunused-result]
fchown(tmpfd, getuid(), -1); /
in case running setuid /
^
gulp.c: In function ‘Writer’:
gulp.c:542:9: warning: ignoring return value of ‘seteuid’, declared with attribute warn_unused_result [-Wunused-result]
seteuid(getuid()); /
drop setuid privilege /
^
/usr/bin/ld: cannot find -lpcap
collect2: error: ld returned 1 exit status
make: **
[gulp] Error 1

Could you please help me

crox on :

Hello,

I assume you are using a 64-bit distribution? You should not need 32-bit libraries in that case. But you may need to install libpcap-dev or the equivalent package for your distribution.

Try to unpack the patched source from comment #2.1.1.1 and then just run "make".

Jishnu on :

Hi
It gave me the same error with the patch. But when I manually complie using gcc (with pcap) i get a gulp executable.
I hope that will work for me. COuld you also please help me to use gulp to save 100 pcap files with 20MB size with 1GB as buffer size. It seems gulp wont let me use a smaller file size with huge buffer size

regards

crox on :

You can probably pipe to a second gulp instance (or to tcpdump) to achieve this goal. Something like:

gulp -i eth0 -r 1024 | gulp -i - -r 20 -C 1 -o /path/

Jishnu on :

Hi
Thanks for reply, I used your code but it give me
Gulp : fatal write error: Broken pipe

I also have a doubt, my intention is to use Gulp to capture at 200Mbps, but as the buffer cannot be increased above 1024MB how will it clear the buffer and not report "Buffer full"

crox on :

The ring buffer is only used when gulp cannot write as fast as it reads. If you have a slow disk but enough RAM, you could pipe several gulp processes to increase the total buffer size.

This works for me:

root@laptop:~# mkdir /tmp/gulp
root@laptop:~# gulp -i wlan0 -r 1024 | gulp -i - -r 1024 | gulp -i - -r 20 -C 1 -o /tmp/gulp/

It creates 20 MB files named pcap000, pcap001 etc. in /tmp/gulp

qwerty on :

i got this error after running make for the patched file in #2.1.1.1

cc -O check64bit.c -o check64bit
./check64bit; rm -f check64bit
If output from Gulp is not compatible with tcpdump or wireshark,
Please see: http://staff.washington.edu/corey/gulp/gulp.html#64bit
cc -g -O gulp.c -o gulp -lpthread -lpcap

Please help

crox on :

The 64-bit part in the original distribution is based on a wrong assumption, see http://seclists.org/wireshark/2009/Oct/105

I recommend you compile from the source with both patches applied: http://blog.crox.net/uploads/gulp-1.58-crox.tgz

qwerty on :

how do i check if gulp is installed or not?
could you please give a detailed explanation of how to install gulp from the patch file in #2.1.1.1.I did not understand the installation

crox on :

I recommend you compile from the source with both patches applied: http://blog.crox.net/uploads/gulp-1.58-crox.tgz

Once you've successfully compiled it, just copy gulp to e.g. /usr/local/bin/ or /usr/local/sbin/

qwerty on :

can gulp capture packets at line rate of 10Gbps?

crox on :

I guess that depends on your hardware, and filters you would apply.

qwerty on :

What do you mean by ring buffer in gulp?what does setting -r do?

crox on :

There are two threads, one that captures packets and puts them in the buffer, and a separate one that writes the captured packets from the buffer to the disk. The benefit of this architecture is that writing to disk doesn't block the capture unlike what can happen with other software. See the author's page for details: http://corey.elsewhere.org/gulp/

-r sets the size of the ring buffer in MB (default is 100 MB) - check the man page (or gulpman.pdf) for details.

Chris on :

Have installed on Fedora 26 using libpcap-devel 14:1.8.1-3.fc26. When starting first capture it gives error Reader could not set cpu affinity: Invalid argument. Haven't tried to load up system yet, was wondering if anyone else has seen this error. Still testing configurations and am not dead set on Fedora.

Chris on :

Previous error about cannot set processor affinity was caused by only having one core assigned to VM. Runs fine with 2 cores assigned.

Carsten on :

Hi Crox,

your collection solves my issues with the initial version of gulp on a current 64bit Linux. Thereby I want to share my feedback with the current version.

In process_savefile() a final exit() is missing. It's necessary if the zflag is not set or execlp() fails. In both cases the child process would unexpectedly return to the main loop and probably run in an infinite loop.

The second change is triggered by the error message "gzip: No such file or directory":

$ ./gulp -i eth0 -o . -Z gzip
gzip: No such file or directory
^C (Ctrl-C pressed after afew seconds)
3405 packets captured
3409 packets received by filter
0 packets dropped by kernel
ring buffer use: 0.1% of 100 MB

This message from gzip is caused by the first call of process_savefile() with an empty filename. The message itself is just ugly and confusing with gzip, but may cause serious trouble with other post processing programs.

This is the call stack for the situation if the error message is triggered by an empty filename:
Breakpoint 1, process_savefile (filename=0x60a400 "") at /home/carsten/gulp-1.58-crox/gulp.c:491
491 if (fork())
(gdb) bt
#0 process_savefile (filename=0x60a400 "") at /home/carsten/gulp-1.58-crox/gulp.c:491
#1 0x0000000000403686 in newoutfile (dir=0x7fffffffec92 ".", num=0) at /home/carsten/gulp-1.58-crox/gulp.c:550
#2 0x00000000004038ad in Writer (arg=0x0) at /home/carsten/gulp-1.58-crox/gulp.c:604
#3 0x00007ffff7bc6e25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007ffff76b334d in clone () from /lib64/libc.so.6

May you execute "execlp(zcmd,...)" only the filename is set ("strlen(filename) > 0").

Thank you for your great extension of gulp.

Regards,
Carsten

Jernej on :

I've added this code to Github so anybody can improve it. Currently the repository is under my username - https://github.com/jmakov/gulp. I'd ask the author to create his own repo so I can delete my and send pull requests to his. You can notify me over e.g. email if you decide so.

crox on :

I've meant to do this for a long time :-)

Chris Glover on :

Thanks!

Gulp works very nicely for capturing SMPTE 2022-6 video streams at 1.5Gbps from a 100G NIC. Set the ringbuffer to maximum and had no dropped packets.

Makes my life so much easier!

Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Form options