Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 07A6E200CCF for ; Mon, 24 Jul 2017 16:31:14 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 05E5F16540A; Mon, 24 Jul 2017 14:31:14 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 50821165409 for ; Mon, 24 Jul 2017 16:31:13 +0200 (CEST) Received: (qmail 27423 invoked by uid 500); 24 Jul 2017 14:31:12 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 27414 invoked by uid 99); 24 Jul 2017 14:31:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jul 2017 14:31:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 181E5C1423 for ; Mon, 24 Jul 2017 14:31:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100 X-Spam-Level: X-Spam-Status: No, score=-100 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, URIBL_BLOCKED=0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id W4nMHInjjmtI for ; Mon, 24 Jul 2017 14:31:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id ABD505FDBF for ; Mon, 24 Jul 2017 14:31:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 06FB3E0D85 for ; Mon, 24 Jul 2017 14:31:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8C3C721EE5 for ; Mon, 24 Jul 2017 14:31:00 +0000 (UTC) Date: Mon, 24 Jul 2017 14:31:00 +0000 (UTC) From: "Arina Ielchiieva (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (DRILL-5432) Want a memory format for PCAP files MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 24 Jul 2017 14:31:14 -0000 [ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-5432: ------------------------------------ Fix Version/s: 1.11.0 > Want a memory format for PCAP files > ----------------------------------- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature > Reporter: Ted Dunning > Fix For: 1.11.0 > > > PCAP files [1] are the de facto standard for storing network capture data. In security and protocol applications, it is very common to want to extract particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and destination IP and port or by protocol. Beyond that, however, it would be very useful to be able to group packets by TCP session and eventually to look at packet contents. For now, however, the most critical requirement is that we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder that did lazy deserialization and could traverse hundreds of MB of PCAP data per second per core. This compares to roughly 2-3 MB/s for widely available Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.4.14#64029)