Sending queued packets with NFQUEUE? - c

Question

I'm using libnetfilter_queue and iptables with the NFQUEUE target to store incoming packets in three different queues with --queue-num x.
I successfully create the three queues with libnetfilter_queuefunctions, bind them, listen to them and read from them as follows:
/* given 'h' as a handler of one of my three queues */
fd = nfq_fd(h);
while ((rv = recv(fd, buf, sizeof(buf), 0)) && rv >= 0) {
nfq_handle_packet(h, buf, rv);
}
The callback function, triggered with nfq_handle_packet, has the nfq_set_verdict(qh, id, NF_ACCEPT, 0, NULL); command where it sends the packet as soon it has been processed.
The problem is: I don't want every packet to be sent right away, since I need to store them in a custom struct (written below).
So I came across a potential solution: I may call NF_DROP verdict instead of NF_ACCEPT on every packet I want to queue (so it won't be immediately sent away), store it in my custom struct and then (sooner or later) re-inject it at my need.
Sounds great, but the situation is: I don't know how to re-inject my queued packets at my pleasure from my userspace application. Is correct to use nfq_set_verdict again at a same point of my code, but with NF_ACCEPT verdict? Or should I open a socket (maybe a raw one)?
This is my custom struct
struct List {
int queue;
int pktsize;
unsigned char *buffer;
struct nfq_q_handle *qh;
struct nfqnl_msg_packet_hdr *hdr;
struct List *next;
};
representing a packet caught with the rule above.
These are my queues where to store packets.
struct List *List0 = NULL; // low priority
struct List *List1 = NULL; // medium priority
struct List *List2 = NULL; // high priority
I have Ubuntu 14.04 3.13.0-57-generic.
Any suggestions would be appreciated.

Answer 1

Your idea makes sense. In fact I've seen a very similar scheme implemented in a commercial product I worked on. It had to process individual packets at high rates, so it would always copy the incoming packet and immediately set an NF_DROP verdict. It would then perform the processing, and if it decided that the packet should be forwarded, it would send the copy to the outbound interface. So you're not alone.
As far as I know, nfq_set_verdict can be called only once per packet. Once the verdict is set, NFQUEUE sends the packet to the destination (which is packet heaven in your case). It doesn't keep an extra copy of the packet just in case you change your mind. So to send the packet back to the network you'll have to store a copy of it and send it using your own socket. And yes, if you want to send the received packet as-is (including headers) the outbound socket would have to be raw.

Answer 2

I don't know if this will fit with your application model, but Frottle just holds the packets in limbo until it decides whether to accept them or drop them. The "novelty" of this approach relies on the fact that you aren't required to call nfq_set_verdict during the NFQUEUE callback function itself; you can call it later and outside the netfilter loop proper. It will use more kernel memory, but the alternative would be just to use more usermode memory so it isn't much of a loss.
Hope this helps!