drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: PCAP Issues
Date Tue, 09 Jan 2018 14:45:16 GMT
I've been following this and and also now playing with them more.  A couple
of tidbits that I think are worth re-mentioning:

1. pcapng the ability to read this too will be handy.

2. The flags are really cool, and the work here is great.

The other thing I'd like to consider for this is with recent work on the
Kafka plugin, and I am guessing a similar plugin with MapR Streams, what if
we were able to have a tool like daemonlogger, or tcpdump read from a
network interface and write directly to a Kafka/MapR Stream topic?

My thought is this, you write to the topic, you partition based on a hash
of known data components (like the 5-tuple of protocol, sport, dport,
saddr, and daddr). From here, you could have things subscribe to the topic
and process, you could etl, you could query.  There would be some concern
at high scale, but even that has methods to address.  One of the challenges
we have infosec is getting processing close enough to capture.... OR
capture to file, move them around, have creates process to read and process
files.

It would be cool, to combine the pcap plugin directly with the kafka/stream
plugin in drill and be able to read live pcaps!

One big challenge is the pcap format has a header, but it seems to be a
simple header that we "may" be able to forge upon dropping the needle on a
stream of packet records.  I am just brainstorming aloud right now, (I
should consider waiting until at least 3 cups of coffee before I make this
inner dialog public)

Curious on thoughts here.




On Fri, Jan 5, 2018 at 10:01 AM, Charles Givre <cgivre@gmail.com> wrote:

> I assumed (incorrectly as it turned out) that I was not reading the
> offsets correctly or something.
>
>
> > On Jan 5, 2018, at 10:59, Ted Dunning <ted.dunning@gmail.com> wrote:
> >
> > Yeah... I got the same result.
> >
> > I will push my latest shortly (without the boolean stuff)
> >
> > On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <cgivre@gmail.com> wrote:
> >
> >> Ted,
> >> I’m wondering if we’ve uncovered an unsupported, undocumented feature in
> >> Drill with respect to the BitWriter.  I made the following changes:
> >>
> >>
> >> To PcapRecordReader.java, I added:
> >>
> >>
> >> case "ack_flag":
> >>  if (packet.isTcpPacket()) {
> >>    int flag = (packet.getAckFlag()) ? 1 : 0;
> >>    setBooleanColumnValue(flag, pci, count);
> >>  }
> >>  break;
> >> ...
> >>
> >> private void setBooleanColumnValue(final int data, final
> >> ProjectedColumnInfo pci, final int count) {
> >>  ((NullableBitVector.Mutator) pci.vv.getMutator())
> >>    .setSafe(count, data);
> >> }
> >>
> >> Then I added this to PcapFormatUtils
> >>
> >> public static int getBit(final byte[] buf, final int posByte, final int
> >> posBit ) {
> >>  byte valByte = buf[posByte];
> >>  return valByte >> (8 - (posBit + 1)) & 0x0001;
> >> }
> >>
> >> I added a column in the Schema.java and a function in Packet.java to get
> >> the bit.  I was getting NPEs and I’m wondering now if the cause wasn’t
> my
> >> code but rather a problem with the bitwriter.  I’m going to play with
> this
> >> and see if I can get Drill to write a True/False value at all, and
> report
> >> back.  If the BitWriter is throwing NPE, I’ll create a JIRA for it.
> >>
> >> Thanks for your work on the PCAP format reader.
> >> —C
> >>
> >>
> >>
> >>> On Jan 3, 2018, at 17:33, Ted Dunning <ted.dunning@gmail.com> wrote:
> >>>
> >>> Don't think that will work.
> >>>
> >>> I tried what seemed obvious and got a NPE. Joys of Drill.
> >>>
> >>>
> >>>
> >>> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cgivre@gmail.com>
> wrote:
> >>>
> >>>> This isn’t the most elegant example, but should do the trick.
> >>>>
> >>>> https://github.com/cgivre/drill-network-functions/blob/
> >>>> master/src/main/java/org/apache/drill/contrib/function/
> IsPrivateIP.java
> >> <
> >>>> https://github.com/cgivre/drill-network-functions/blob/
> >>>> master/src/main/java/org/apache/drill/contrib/function/
> >> IsPrivateIP.java>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On Jan 3, 2018, at 16:09, Ted Dunning <ted.dunning@gmail.com>
wrote:
> >>>>>
> >>>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cgivre@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> HI Ted,
> >>>>>> This is really looking amazing.  Thank you so much for working
on
> >>>> this.  I
> >>>>>> wanted to ask whether you’ve tried the BitWriter to write
a Boolean
> >>>> value?
> >>>>>> I’ve done that in UDFs.
> >>>>>
> >>>>>
> >>>>> Point me to an example. I think booleans would be vastly better.
> >>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message