drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: PCAP Issues
Date Tue, 09 Jan 2018 16:26:28 GMT
The pcap format is there almost entirely just to give us a record structure.

The stream could give us that just as easily.

So that is one thing that could be done.

Another thought is that data could be buffered a (very) short time and
groups of packets could be sent as a single message. That would decrease
streaming overhead. I would vote against that change at first, but it would
make the pcap format relevant again.



On Tue, Jan 9, 2018 at 6:45 AM, John Omernik <john@omernik.com> wrote:

> I've been following this and and also now playing with them more.  A couple
> of tidbits that I think are worth re-mentioning:
>
> 1. pcapng the ability to read this too will be handy.
>
> 2. The flags are really cool, and the work here is great.
>
> The other thing I'd like to consider for this is with recent work on the
> Kafka plugin, and I am guessing a similar plugin with MapR Streams, what if
> we were able to have a tool like daemonlogger, or tcpdump read from a
> network interface and write directly to a Kafka/MapR Stream topic?
>
> My thought is this, you write to the topic, you partition based on a hash
> of known data components (like the 5-tuple of protocol, sport, dport,
> saddr, and daddr). From here, you could have things subscribe to the topic
> and process, you could etl, you could query.  There would be some concern
> at high scale, but even that has methods to address.  One of the challenges
> we have infosec is getting processing close enough to capture.... OR
> capture to file, move them around, have creates process to read and process
> files.
>
> It would be cool, to combine the pcap plugin directly with the kafka/stream
> plugin in drill and be able to read live pcaps!
>
> One big challenge is the pcap format has a header, but it seems to be a
> simple header that we "may" be able to forge upon dropping the needle on a
> stream of packet records.  I am just brainstorming aloud right now, (I
> should consider waiting until at least 3 cups of coffee before I make this
> inner dialog public)
>
> Curious on thoughts here.
>
>
>
>
> On Fri, Jan 5, 2018 at 10:01 AM, Charles Givre <cgivre@gmail.com> wrote:
>
> > I assumed (incorrectly as it turned out) that I was not reading the
> > offsets correctly or something.
> >
> >
> > > On Jan 5, 2018, at 10:59, Ted Dunning <ted.dunning@gmail.com> wrote:
> > >
> > > Yeah... I got the same result.
> > >
> > > I will push my latest shortly (without the boolean stuff)
> > >
> > > On Fri, Jan 5, 2018 at 7:52 AM, Charles Givre <cgivre@gmail.com>
> wrote:
> > >
> > >> Ted,
> > >> I’m wondering if we’ve uncovered an unsupported, undocumented feature
> in
> > >> Drill with respect to the BitWriter.  I made the following changes:
> > >>
> > >>
> > >> To PcapRecordReader.java, I added:
> > >>
> > >>
> > >> case "ack_flag":
> > >>  if (packet.isTcpPacket()) {
> > >>    int flag = (packet.getAckFlag()) ? 1 : 0;
> > >>    setBooleanColumnValue(flag, pci, count);
> > >>  }
> > >>  break;
> > >> ...
> > >>
> > >> private void setBooleanColumnValue(final int data, final
> > >> ProjectedColumnInfo pci, final int count) {
> > >>  ((NullableBitVector.Mutator) pci.vv.getMutator())
> > >>    .setSafe(count, data);
> > >> }
> > >>
> > >> Then I added this to PcapFormatUtils
> > >>
> > >> public static int getBit(final byte[] buf, final int posByte, final
> int
> > >> posBit ) {
> > >>  byte valByte = buf[posByte];
> > >>  return valByte >> (8 - (posBit + 1)) & 0x0001;
> > >> }
> > >>
> > >> I added a column in the Schema.java and a function in Packet.java to
> get
> > >> the bit.  I was getting NPEs and I’m wondering now if the cause wasn’t
> > my
> > >> code but rather a problem with the bitwriter.  I’m going to play with
> > this
> > >> and see if I can get Drill to write a True/False value at all, and
> > report
> > >> back.  If the BitWriter is throwing NPE, I’ll create a JIRA for it.
> > >>
> > >> Thanks for your work on the PCAP format reader.
> > >> —C
> > >>
> > >>
> > >>
> > >>> On Jan 3, 2018, at 17:33, Ted Dunning <ted.dunning@gmail.com>
wrote:
> > >>>
> > >>> Don't think that will work.
> > >>>
> > >>> I tried what seemed obvious and got a NPE. Joys of Drill.
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jan 3, 2018 at 1:31 PM, Charles Givre <cgivre@gmail.com>
> > wrote:
> > >>>
> > >>>> This isn’t the most elegant example, but should do the trick.
> > >>>>
> > >>>> https://github.com/cgivre/drill-network-functions/blob/
> > >>>> master/src/main/java/org/apache/drill/contrib/function/
> > IsPrivateIP.java
> > >> <
> > >>>> https://github.com/cgivre/drill-network-functions/blob/
> > >>>> master/src/main/java/org/apache/drill/contrib/function/
> > >> IsPrivateIP.java>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Jan 3, 2018, at 16:09, Ted Dunning <ted.dunning@gmail.com>
> wrote:
> > >>>>>
> > >>>>> On Wed, Jan 3, 2018 at 12:17 PM, Charles Givre <cgivre@gmail.com>
> > >> wrote:
> > >>>>>
> > >>>>>> HI Ted,
> > >>>>>> This is really looking amazing.  Thank you so much for
working on
> > >>>> this.  I
> > >>>>>> wanted to ask whether you’ve tried the BitWriter to write
a
> Boolean
> > >>>> value?
> > >>>>>> I’ve done that in UDFs.
> > >>>>>
> > >>>>>
> > >>>>> Point me to an example. I think booleans would be vastly better.
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message