Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 96C74200C49 for ; Fri, 17 Mar 2017 17:38:19 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 94FED160B80; Fri, 17 Mar 2017 16:38:19 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7BF29160B70 for ; Fri, 17 Mar 2017 17:38:17 +0100 (CET) Received: (qmail 91200 invoked by uid 500); 17 Mar 2017 16:38:15 -0000 Mailing-List: contact commits-help@metron.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.incubator.apache.org Delivered-To: mailing list commits@metron.incubator.apache.org Received: (qmail 91191 invoked by uid 99); 17 Mar 2017 16:38:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Mar 2017 16:38:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E93B818F177 for ; Fri, 17 Mar 2017 16:38:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.068 X-Spam-Level: X-Spam-Status: No, score=-3.068 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_NEUTRAL=0.652, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id W33VHZSPVX6k for ; Fri, 17 Mar 2017 16:38:05 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id C59E35FBE6 for ; Fri, 17 Mar 2017 16:38:02 +0000 (UTC) Received: (qmail 34494 invoked by uid 99); 17 Mar 2017 14:51:22 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Mar 2017 14:51:22 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 4F9C1E001D; Fri, 17 Mar 2017 14:51:22 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: cestella@apache.org To: commits@metron.incubator.apache.org Date: Fri, 17 Mar 2017 14:51:27 -0000 Message-Id: <5a8c45be162b42c986165448a8361e97@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [06/15] incubator-metron git commit: METRON-766: Release 0.3.1 closes apache/incubator-metron#477 archived-at: Fri, 17 Mar 2017 16:38:19 -0000 http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-deployment/vagrant/full-dev-platform/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/vagrant/full-dev-platform/index.html b/current-book/metron-deployment/vagrant/full-dev-platform/index.html new file mode 100644 index 0000000..26a1336 --- /dev/null +++ b/current-book/metron-deployment/vagrant/full-dev-platform/index.html @@ -0,0 +1,374 @@ + + + + + + + + + Metron – Full Development Platform + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Full Development Platform

+

This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running on Virtualbox.

+

Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. To that end the vagrant file configuration defaults to disabling solr and yaf.

+
+

Getting Started

+
+

Prerequisites

+

The computer used to deploy Apache Metron will need to have the following components installed.

+ + +
+

macOS

+

Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent Homebrew project.

+ +
    + +
  1. +

    Install Homebrew by following the instructions at Homebrew.

    2. + +
  2. +

    Run the following command in a terminal to install all of the required tools.

    + +
    +
    +
      brew cask install vagrant virtualbox java
+  brew install maven git
+  brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ee1273bf919a5e4e50838513a9e55ea423e1d7ce/Formula/ansible.rb
+  brew switch ansible 2.0.0.2
+
    3. +
+
+

Deploy Metron

+ +
    + +
  1. +

    Build Metron

    + +
    +
    +
      cd incubator-metron
+  mvn clean package -DskipTests
+
    2. + +
  2. +

    Install Vagrant Hostmanager.

    + +
    +
    +
      vagrant plugin install vagrant-hostmanager
+
    3. + +
  3. +

    Deploy Metron

    + +
    +
    +
      cd metron-deployment/vagrant/full-dev-platform
+  vagrant up
+
    +

    Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.

    + +
    +
    +
      vagrant provision
+
    4. +
+
+

Explore Metron

+

Navigate to the following resources to explore your newly minted Apache Metron environment.

+ + +

Connecting to the host through SSH is as simple as running the following command.

+ +
+
+
vagrant ssh
+
+
+

Working with Metron

+

In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. The following commands will re-run the web role on the Vagrant image. This will install components (if necessary) and start the UI.

+ +
+
+
./run_ansible_role.sh web
+
+

or

+ +
+
+
vagrant --ansible-tags="web" provision
+
+
+

Using Tags

+

A collection of tags is specified as a comma separated list.

+ +
+
+
./run_ansible_role.sh "sensors,enrichment"
+
+
+

Tags are listed in the playbooks, some frequently used tags:

+ +
    + +
  • hdp-install - Install HDP
    • + +
  • hdp-deploy - Deploy and Start HDP Services (will start all Hadoop Services)
    • + +
  • sensors - Deploy and Start Sensors.
    • + +
  • enrichment - Deploy and Start Enrichment Topology.
    • +
+

Note also that there is a convenience script ./run_enrichment_role.sh which executes Vagrant with the enrichment tag.

+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-deployment/vagrant/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/vagrant/index.html b/current-book/metron-deployment/vagrant/index.html new file mode 100644 index 0000000..373405e --- /dev/null +++ b/current-book/metron-deployment/vagrant/index.html @@ -0,0 +1,255 @@ + + + + + + + + + Metron – Vagrant Deployment + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Vagrant Deployment

+

+ +
    + +
  • Codelab Platform
    • + +
  • Fast CAPA Test Platform
    • + +
  • Full Dev Platform
    • + +
  • Quick Dev Platform
    • +
+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-deployment/vagrant/quick-dev-platform/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-deployment/vagrant/quick-dev-platform/index.html b/current-book/metron-deployment/vagrant/quick-dev-platform/index.html new file mode 100644 index 0000000..123efc6 --- /dev/null +++ b/current-book/metron-deployment/vagrant/quick-dev-platform/index.html @@ -0,0 +1,327 @@ + + + + + + + + + Metron – Quick Development Platform + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Quick Development Platform

+

This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running on Virtualbox.

+

This image is designed for quick deployment of a single node Metron cluster running on Virtualbox. This platform is ideal for use by Metron developers. It uses a base image that has been pre-loaded with Ambari and HDP.

+

Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. To that end the vagrant file configuration defaults to disabling solr and yaf.

+
+

Getting Started

+
+

Prerequisites

+

As with the Full Development Platform (metron-deployment/vagrant/full-dev-platform), the computer used to deploy Apache Metron will need the following components installed.

+ +
+
+

Deploy Metron

+ +
    + +
  1. +

    Build Metron

    + +
    +
    +
      cd incubator-metron
+  mvn clean package -DskipTests
+
    2. + +
  2. +

    Install Vagrant Hostmanager.

    + +
    +
    +
      vagrant plugin install vagrant-hostmanager
+
    3. + +
  3. +

    Deploy Metron

    + +
    +
    +
      cd metron-deployment/vagrant/quick-dev-platform
+  vagrant up
+
    +

    Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.

    + +
    +
    +
      vagrant provision
+
    4. +
+
+

Explore Metron

+

Navigate to the following resources to explore your newly minted Apache Metron environment.

+ +
+
+

Working with Metron

+

As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run the following command.

+ +
+
+
./run_enrichment_role.sh
+
+

Connecting to the host through SSH is as simple as running the following command.

+ +
+
+
vagrant ssh
+
+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-docker/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-docker/index.html b/current-book/metron-docker/index.html new file mode 100644 index 0000000..a87945b --- /dev/null +++ b/current-book/metron-docker/index.html @@ -0,0 +1,470 @@ + + + + + + + + + Metron – Metron Docker + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Metron Docker

+

+

Metron Docker is a Docker Compose application that is intended for development and integration testing of Metron. Use this instead of Vagrant when:

+ +
    + +
  • You want an environment that can be built and spun up quickly
    • + +
  • You need to frequently rebuild and restart services
    • + +
  • You only need to test, troubleshoot or develop against a subset of services
    • +
+

Metron Docker includes these images that have been customized for Metron:

+ +
    + +
  • Kafka (with Zookeeper)
    • + +
  • HBase
    • + +
  • Storm (with all topologies deployed)
    • + +
  • Elasticsearch
    • + +
  • Kibana
    • +
+
+

Setup

+

Install Docker for Mac or Docker for Windows. The following versions have been tested:

+ +
    + +
  • Docker version 1.12.0
    • + +
  • docker-machine version 0.8.0
    • + +
  • docker-compose version 1.8.0
    • +
+

Build Metron from the top level directory with:

+ +
+
+
$ cd $METRON_HOME
+$ mvn clean install -DskipTests
+
+

You are welcome to use an existing Docker host but we prefer one with more resources. You can create one of those with this script:

+ +
+
+
$ export METRON_DOCKER_HOME=$METRON_HOME/metron-docker
+$ cd $METRON_DOCKER_HOME && ./scripts/create-docker-machine.sh
+
+

This will create a host called “metron-machine”. Anytime you want to run Docker commands against this host, make sure you run this first to set the Docker environment variables:

+ +
+
+
$ eval "$(docker-machine env metron-machine)"
+
+

If you wish to use a local docker-engine install, please set an environment variable BROKER_IP_ADDR to the IP address of your host machine. This cannot be the loopback address.

+
+

Usage

+

Navigate to the compose application root:

+ +
+
+
$ cd $METRON_DOCKER_HOME/compose/
+
+

The Metron Docker environment lifecycle is controlled by the docker-compose command. The service names can be found in the docker-compose.yml file. For example, to build and start the environment run this command:

+ +
+
+
$ eval "$(docker-machine env metron-machine)"
+$ docker-compose up -d
+
+

After all services have started list the containers and ensure their status is ‘Up’:

+ +
+
+
$ docker ps --format 'table {{.Names}}\t{{.Status}}'
+NAMES                    STATUS
+metron_storm_1           Up 5 minutes
+metron_hbase_1           Up 5 minutes
+metron_kibana_1          Up 5 minutes
+metron_kafkazk_1         Up 5 minutes
+metron_elasticsearch_1   Up 5 minutes
+
+

Various services are exposed through http on the Docker host. Get the host ip from the URL property:

+ +
+
+
$ docker-machine ls
+NAME             ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
+metron-machine   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.5
+
+

Then, assuming a host ip of 192.168.99.100, the UIs and APIs are available at:

+ + +

The Storm logs can be useful when troubleshooting topologies. They can be found on the Storm container in /usr/share/apache-storm/logs.

+

When done using the machine, shut it down with:

+ +
+
+
$ docker-compose down
+
+
+

Examples

+ + +
+

Deploy a new parser class

+

After adding a new parser to metron-parsers, build Metron from the top level directory:

+ +
+
+
$ cd $METRON_HOME
+$ mvn clean install -DskipTests
+
+

Then run these commands to redeploy the parsers to the Storm image:

+ +
+
+
$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose down
+$ docker-compose build storm
+$ docker-compose up -d
+
+
+

Connect to a container

+

Suppose there is a problem with Kafka and the logs are needed for further investigation. Run this command to connect and explore the running Kafka container:

+ +
+
+
$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose exec kafkazk bash
+
+
+

Create a sensor from sample data

+

A tool for producing test data in Kafka is included with the Kafka/Zookeeper image. It loops through lines in a test data file and outputs them to Kafka at the desired frequency. Create a test data file in ./kafkazk/data/ and rebuild the Kafka/Zookeeper image:

+ +
+
+
$ cd $METRON_DOCKER_HOME/compose
+$ printf 'first test data\nsecond test data\nthird test data\n' > ./kafkazk/data/TestData.txt
+$ docker-compose down
+$ docker-compose build kafkazk
+$ docker-compose up -d
+
+

This will deploy the test data file to the Kafka/Zookeeper container. Now that data can be streamed to a Kafka topic:

+ +
+
+
$ docker-compose exec kafkazk ./bin/produce-data.sh
+Usage:  produce-data.sh data_path topic [message_delay_in_seconds]
+
+# Stream data in TestData.txt to the 'test' Kafka topic at a frequency of 5 seconds (default is 1 second)
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/TestData.txt test 5 
+
+

The Kafka/Zookeeper image comes with sample Bro and Squid data:

+ +
+
+
# Stream Bro test data every 1 second
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
+
+# Stream Squid test data every 0.1 seconds
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/SquidExampleOutput.txt squid 0.1
+
+
+

Upload configs to Zookeeper

+

Parser configs and a global config configured for this Docker environment are included with the Kafka/Zookeeper image. Load them with:

+ +
+
+
$ docker-compose exec kafkazk bash
+# $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i $METRON_HOME/config/zookeeper
+# exit
+
+

Dump out the configs with:

+ +
+
+
$ docker-compose exec kafkazk bash
+# $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
+# exit
+
+
+

Manage a topology

+

The Storm image comes with a script to easily start parser topologies:

+ +
+
+
docker-compose exec storm ./bin/start_docker_parser_topology.sh sensor_name
+
+

The enrichment topology can be started with:

+ +
+
+
docker-compose exec storm ./bin/start_enrichment_topology.sh
+
+

The indexing topology can be started with:

+ +
+
+
docker-compose exec storm ./bin/start_elasticsearch_topology.sh
+
+

Topologies can be stopped using the Storm CLI. For example, stop the enrichment topology with:

+ +
+
+
docker-compose exec storm storm kill enrichments -w 0
+
+
+

Run sensor data end to end

+

First ensure configs were uploaded as described in the previous example. Then start a sensor and leave it running:

+ +
+
+
$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
+
+

Open a separate console session and verify the sensor is running by consuming a message from Kafka:

+ +
+
+
$ export METRON_DOCKER_HOME=$METRON_HOME/metron-docker
+$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
+
+

A new message should be printed every second. Now kill the consumer and start the Bro parser topology:

+ +
+
+
$ docker-compose exec storm ./bin/start_docker_parser_topology.sh bro
+
+

Bro data should be flowing through the bro parser topology and into the Kafka enrichments topic. The enrichments topic should be created automatically:

+ +
+
+
$ docker-compose exec kafkazk ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
+bro
+enrichments
+indexing
+
+

Verify parsed Bro data is in the Kafka enrichments topic:

+ +
+
+
docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic enrichments
+
+

Now start the enrichment topology:

+ +
+
+
docker-compose exec storm ./bin/start_enrichment_topology.sh
+
+

Parsed Bro data should be flowing through the enrichment topology and into the Kafka indexing topic. Verify enriched Bro data is in the Kafka indexing topic:

+ +
+
+
docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic indexing
+
+

Now start the indexing topology:

+ +
+
+
docker-compose exec storm ./bin/start_elasticsearch_topology.sh
+
+

Enriched Bro data should now be present in the Elasticsearch container:

+ +
+
+
$ docker-machine ls
+NAME             ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
+metron-machine   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.5
+
+$ curl -XGET http://192.168.99.100:9200/_cat/indices?v
+health status index                   pri rep docs.count docs.deleted store.size pri.store.size
+yellow open   .kibana                   1   1          1            0      3.1kb          3.1kb
+yellow open   bro_index_2016.12.19.18   5   1        180            0      475kb          475kb
+
+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-platform/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/index.html b/current-book/metron-platform/index.html new file mode 100644 index 0000000..0546ef1 --- /dev/null +++ b/current-book/metron-platform/index.html @@ -0,0 +1,249 @@ + + + + + + + + + Metron – Current Build + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Current Build

+

+

The latest build of metron-platform is 0.3.1.

+

We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.

+

+

metron-platform

+

Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at https://metron.incubator.apache.org/

+

+

Documentation

+

Please see documentation within each individual module for description and usage instructions. Sample topologies are provided under Metron_Topologies to get you started with the framework. We pre-assume knowledge of Hadoop, Storm, Kafka, and HBase.

+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-platform/metron-api/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-api/index.html b/current-book/metron-platform/metron-api/index.html new file mode 100644 index 0000000..fe2b9a6 --- /dev/null +++ b/current-book/metron-platform/metron-api/index.html @@ -0,0 +1,285 @@ + + + + + + + + + Metron – Metron PCAP Service + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Metron PCAP Service

+

+

The purpose of the Metron PCAP service is to provide a middle tier to negotiate retrieving packet capture data which flows into Metron. This packet data is of a form which libpcap based tools can read.

+
+

Starting the Service

+

You can start the service either via the init.d script installed, /etc/init.d/pcapservice or directly via the yarn jar command: yarn jar $METRON_HOME/lib/metron-api-$METRON_VERSION.jar org.apache.metron.pcapservice.rest.PcapService -port $SERVICE_PORT -query_hdfs_path $QUERY_PATH -pcap_hdfs_path $PCAP_PATH

+

where

+ +
    + +
  • METRON_HOME is the location of the metron installation
    • + +
  • METRON_VERSION is the version of the metron installation
    • + +
  • SERVICE_PORT is the port to bind the REST service to.
    • + +
  • QUERY_PATH is the temporary location to store query results. They are deleted after the service reads them.
    • + +
  • PCAP_PATH is the path to the packet data on HDFS
    • +
+
+

The /pcapGetter/getPcapsByIdentifiers endpoint

+

This endpoint takes the following query parameters and returns the subset of packets matching this query:

+ +
    + +
  • srcIp : The source IP to match on
    • + +
  • srcPort : The source port to match on
    • + +
  • dstIp : The destination IP to match on
    • + +
  • dstPort : The destination port to match on
    • + +
  • startTime : The start time in milliseconds
    • + +
  • endTime : The end time in milliseconds
    • + +
  • numReducers : Specify the number of reducers to use when executing the mapreduce job
    • + +
  • includeReverseTraffic : Indicates if filter should check swapped src/dest addresses and IPs
    • +
+
+

The /pcapGetter/getPcapsByQuery endpoint

+

This endpoint takes the following query parameters and returns the subset of packets matching this query. This endpoint exposes Stellar querying capabilities:

+ +
    + +
  • query : The Stellar query to execute
    • + +
  • startTime : The start time in milliseconds
    • + +
  • endTime : The end time in milliseconds
    • + +
  • numReducers : Specify the number of reducers to use when executing the mapreduce job
    • +
+

Example: curl -XGET "http://node1:8081/pcapGetter/getPcapsByQuery?query=ip_src_addr+==+'192.168.66.121'+and+ip_src_port+==+'60500'&startTime=1476936000000"

+

All of these parameters are optional. In the case of a missing parameter, it is treated as a wildcard.

+

Unlike the CLI tool, there is no paging mechanism. The REST API will stream back data as a single file.

+
+
+
+ +
+ + + +