From commits-return-3874-archive-asf-public=cust-asf.ponee.io@metron.apache.org Tue Sep 18 16:54:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3FAF01807A5 for ; Tue, 18 Sep 2018 16:54:44 +0200 (CEST) Received: (qmail 28518 invoked by uid 500); 18 Sep 2018 14:54:43 -0000 Mailing-List: contact commits-help@metron.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.apache.org Delivered-To: mailing list commits@metron.apache.org Received: (qmail 28419 invoked by uid 99); 18 Sep 2018 14:54:43 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Sep 2018 14:54:43 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id DDB4EE10F9; Tue, 18 Sep 2018 14:54:42 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: nickallen@apache.org To: commits@metron.apache.org Date: Tue, 18 Sep 2018 14:54:49 -0000 Message-Id: <0918f951013e4fb79b7b8eb94bb6cec0@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [08/21] metron git commit: METRON-1776 Update public web site to point at 0.6.0 new release (justinleet) closes apache/metron#1195 http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/development/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/development/index.html b/site/current-book/metron-deployment/development/index.html index ea99fdf..9c6f49e 100644 --- a/site/current-book/metron-deployment/development/index.html +++ b/site/current-book/metron-deployment/development/index.html @@ -1,13 +1,13 @@ - + Metron – Metron Development Environments @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Metron Development Environments
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/development/ubuntu14/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/development/ubuntu14/index.html b/site/current-book/metron-deployment/development/ubuntu14/index.html index 16b2d81..fb15a5e 100644 --- a/site/current-book/metron-deployment/development/ubuntu14/index.html +++ b/site/current-book/metron-deployment/development/ubuntu14/index.html @@ -1,13 +1,13 @@ - + Metron – Metron on Ubuntu 14 @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Metron on Ubuntu 14
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -190,8 +190,8 @@ vagrant up

    Navigate to the following resources to explore your newly minted Apache Metron environment.

    Connecting to the host through SSH is as simple as running the following command.

    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/index.html b/site/current-book/metron-deployment/index.html index a3ffcbd..ef7ddb1 100644 --- a/site/current-book/metron-deployment/index.html +++ b/site/current-book/metron-deployment/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -171,7 +171,10 @@ limitations under the License.

    Running Metron within the resource constraints of a single VM is incredibly challenging. Failing to respect this warning, will cause various services to fail mysteriously as the system runs into memory and processing limits.

    How?

    -

    To deploy Metron in a VM running on your computer, follow the instructions at development/centos6.

    +

    To deploy Metron in a VM running on your computer, follow the instructions at development/centos6.

    +
    +

    How do I address services crashing when running Metron on a single VM?

    +

    We recommend looking at Ambari and shutting down any services you may not be using. For example, we recommend turning off Metron Profiler, as this commonly causes REST services to crash when running on a single VM.

    How do I build RPM packages?

    This provides RPM packages that allow you to install Metron on an RPM-based operating system like CentOS.

    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/other-examples/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/other-examples/index.html b/site/current-book/metron-deployment/other-examples/index.html index 3a89cb0..41ee390 100644 --- a/site/current-book/metron-deployment/other-examples/index.html +++ b/site/current-book/metron-deployment/other-examples/index.html @@ -1,13 +1,13 @@ - + Metron – Other Example Deployments @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Other Example Deployments
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html b/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html index 5cf9775..f3d8dff 100644 --- a/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html +++ b/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html b/site/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html index 499e89b..8a59770 100644 --- a/site/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html +++ b/site/current-book/metron-deployment/packaging/ambari/elasticsearch-mpack/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/ambari/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/ambari/index.html b/site/current-book/metron-deployment/packaging/ambari/index.html index bac5758..7acd537 100644 --- a/site/current-book/metron-deployment/packaging/ambari/index.html +++ b/site/current-book/metron-deployment/packaging/ambari/index.html @@ -1,13 +1,13 @@ - + Metron – Ambari Management Pack Development @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Ambari Management Pack Development
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html b/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html index dd97717..ddf0663 100644 --- a/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html +++ b/site/current-book/metron-deployment/packaging/ambari/metron-mpack/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html b/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html index 8145c8f..4c85d3f 100644 --- a/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html +++ b/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/docker/deb-docker/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/docker/deb-docker/index.html b/site/current-book/metron-deployment/packaging/docker/deb-docker/index.html index a83a363..8dd1e3f 100644 --- a/site/current-book/metron-deployment/packaging/docker/deb-docker/index.html +++ b/site/current-book/metron-deployment/packaging/docker/deb-docker/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html b/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html index d684c0d..4d0cee3 100644 --- a/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html +++ b/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-deployment/packaging/packer-build/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-deployment/packaging/packer-build/index.html b/site/current-book/metron-deployment/packaging/packer-build/index.html index fb4b71b..63a85b0 100644 --- a/site/current-book/metron-deployment/packaging/packer-build/index.html +++ b/site/current-book/metron-deployment/packaging/packer-build/index.html @@ -1,13 +1,13 @@ - + Metron – Build Metron Images @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Build Metron Images
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-interface/metron-alerts/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-interface/metron-alerts/index.html b/site/current-book/metron-interface/metron-alerts/index.html index 9459bd1..f4c3d04 100644 --- a/site/current-book/metron-interface/metron-alerts/index.html +++ b/site/current-book/metron-interface/metron-alerts/index.html @@ -1,13 +1,13 @@ - + Metron – @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -118,7 +118,8 @@ limitations under the License.

    Prerequisites

      -
    • The Metron REST application should be up and running and Elasticsearch should have some alerts populated by Metron topologies
      • +
    • The Metron REST application should be up and running
      • +
    • Elasticsearch or Solr should have some alerts populated by Metron topologies, depending on which real-time store is enabled
    • The Management UI should be installed (which includes Express)
    • The alerts can be populated using Full Dev or any other setup
    • UI is developed using angular4 and uses angular-cli
      • @@ -202,7 +203,10 @@ rest:

      Global Configuration Properties

      source.type.field

      -

      The source type format used. Defaults to source:type.

    +

    The source type field name used in the real-time store. Defaults to source:type.

    +
    +

    threat.triage.score.field

    +

    The threat triage score field name used in the real-time store. Defaults to threat:triage:score.

    Usage

    After configuration is complete, the Management UI can be managed as a service:

    @@ -237,12 +241,34 @@ npm install

    NOTE: In the development mode ui by default connects to REST at http://node1:8082 for fetching data. If you wish to change it you can change the REST url at metron/metron-interface/metron-alerts/proxy.conf.json

    E2E Tests

    -

    An expressjs server is available for mocking the elastic search api.

    +
    +

    Caveats

    1. -

      Run e2e webserver :

      +

      E2E tests uses data from full-dev wherever applicable. The tests assume rest-api’s are available @http://node1:8082. It is recommended to shutdown all other Metron services while running the E2E tests including Parsers, Enrichment, Indexing and the Profiler.

      +
      2. +
    2. + +

      E2E tests are run on headless chrome. To see the chrome browser in action, remove the ‘–headless’ parameter of chromeOptions in metron/metron-interface/metron-alerts/protractor.conf.js file

      +
      3. +
    3. + +

      E2E tests delete all the data in HBase table ‘metron_update’ and Elastic search index ‘meta_alerts_index’ for testing against its test data

      +
      4. +
    4. + +

      E2E tests use protractor-flake to re-run flaky tests.

      +
      5. +
    +
    +

    Steps to run

    +
      + +
    1. + +

      An Express.js server is available for accessing the rest api. Run the e2e webserver:

      @@ -252,7 +278,7 @@ sh ./scripts/start-server-for-e2e.sh
    2. -

      run e2e test using the following command

      +

      Run e2e tests using the following command:

      @@ -260,12 +286,8 @@ sh ./scripts/start-server-for-e2e.sh npm run e2e
      3. -
    3. - -

      E2E tests uses data from full-dev wherever applicable. The tests assume rest-api’s are available @http://node1:8082

      -
    -

    NOTE: e2e tests covers all the general workflows and we will extend them as we need

    +

    NOTE: e2e tests cover all the general workflows and we will extend them as we need

    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-interface/metron-config/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-interface/metron-config/index.html b/site/current-book/metron-interface/metron-config/index.html index 0c4c073..c46f751 100644 --- a/site/current-book/metron-interface/metron-config/index.html +++ b/site/current-book/metron-interface/metron-config/index.html @@ -1,13 +1,13 @@ - + Metron – Metron Management UI @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Metron Management UI
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-interface/metron-rest/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-interface/metron-rest/index.html b/site/current-book/metron-interface/metron-rest/index.html index eb378b3..b0f91d7 100644 --- a/site/current-book/metron-interface/metron-rest/index.html +++ b/site/current-book/metron-interface/metron-rest/index.html @@ -1,13 +1,13 @@ - + Metron – Metron REST @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Metron REST
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -97,6 +97,7 @@ limitations under the License.
    • A running Metron cluster
      • +
    • A running real-time store, either Elasticsearch or Solr depending on which one is enabled
    • Java 8 installed
    • Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
    • A relational database
      • @@ -429,6 +430,23 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
    +

    Pcap Query

    +

    The REST application exposes endpoints for querying Pcap data. For more information about filtering options see Query Filter Utility.

    +

    There is an endpoint available that will return Pcap data in PDML format. Wireshark must be installed for this feature to work. Installing wireshark in CentOS can be done with yum -y install wireshark.

    +

    The REST application uses a Java Process object to call out to the pcap_to_pdml.sh script. This script is installed at $METRON_HOME/bin/pcap_to_pdml.sh by default. Out of the box it is a simple wrapper around the tshark command to transform raw pcap data to PDML. However it can be extended to do additional processing as long as the expected input/output is maintained. REST will supply the script with raw pcap data through standard in and expects PDML data serialized as XML.

    +

    Pcap query jobs can be configured for submission to a YARN queue. This setting is exposed as the Spring property pcap.yarn.queue. If configured, the REST application will set the mapreduce.job.queuename Hadoop property to that value. It is highly recommended that a dedicated YARN queue be created and configured for Pcap queries to prevent a job from consuming too many cluster resources. More information about setting up YARN queues can be found here.

    +

    Pcap query results are stored in HDFS. The location of query results when run through the REST app is determined by a couple factors. The root of Pcap query results defaults to /apps/metron/pcap/output but can be changed with the Spring property pcap.final.output.path. Assuming the default Pcap query output directory, the path to a result page will follow this pattern:

    + +
    +
    +
    /apps/metron/pcap/output/{username}/MAP_REDUCE/{job id}/page-{page number}.pcap
+
    + +

    Over time Pcap query results will accumulate in HDFS. Currently these results are not cleaned up automatically so cluster administrators should be aware of this and monitor them. It is highly recommended that a process be put in place to periodically delete files and directories under the Pcap query results root.

    +

    Users should also be mindful of date ranges used in queries so they don’t produce result sets that are too large. Currently there are no limits enforced on date ranges.

    +

    Queries can also be configured on a global level for setting the number of results per page via a Spring property pcap.page.size. By default, this value is set to 10 pcaps per page, but you may choose to set this value higher based on observing frequenetly-run query result sizes. This setting works in conjunction with the property for setting finalizer threadpool size when optimizing query performance.

    +

    Pcap query jobs have a finalization routine that writes their results out to HDFS in pages. Depending on the size of your pcaps, the number or results typically returned, page sizing (described above), and available CPU cores for running your REST application, your performance can be improved by adjusting the number of files that can be written to HDFS in parallel. To this end, there is a threadpool used for this finalization step that can be configured to use a specified number of threads. This setting is exposed as the Spring property pcap.finalizer.threadpool.size. A default value of “1” is used if not specified by the user. Generally speaking, you should see a performance gain when this value is set to anything higher than 1. A sizeable increase in performance can be achieved, especially for larger numbers of files of smaller size, by increasing the number of threads. It should be noted that this property is parsed as a String to allow for more complex parallelism values. In addition to normal integer values, you can specify a multiple of the number of cores. If it’s a string and ends with “C”, then strip the C and treat it as an integral multiple of the number of cores. If it’s a string and does not end with a C, then treat it as a number in string form.

    +

    API

    Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.

    @@ -439,7 +457,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" - + @@ -491,11 +509,27 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" + + + + + + + + + + + + + + + + - + - + @@ -593,7 +627,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab" - + @@ -1051,6 +1085,156 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
    +

    POST /api/v1/pcap/fixed

    +
      + +
    • Description: Executes a Fixed Filter Pcap Query.
      • +
    • Input: +
        + +
      • fixedPcapRequest - A Fixed Pcap Request which includes fixed filter fields like ip source address and protocol
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns a job status with job ID.
        • +
      +
      • +
    +
    +

    POST /api/v1/pcap/query

    +
      + +
    • Description: Executes a Query Filter Pcap Query.
      • +
    • Input: +
        + +
      • queryPcapRequest - A Query Pcap Request which includes Stellar query field
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns a job status with job ID.
        • +
      +
      • +
    +
    +

    GET /api/v1/pcap

    +
      + +
    • Description: Gets a list of job statuses for Pcap query jobs that match the requested state.
      • +
    • Input: +
        + +
      • state - Job state
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns a list of job statuses for jobs that match the requested state.
        • +
      +
      • +
    +
    +

    GET /api/v1/pcap/{jobId}

    +
      + +
    • Description: Gets job status for Pcap query job.
      • +
    • Input: +
        + +
      • jobId - Job ID of submitted job
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns a job status for the Job ID.
        • +
      • 404 - Job is missing.
        • +
      +
      • +
    +
    +

    GET /api/v1/pcap/{jobId}/pdml

    +
      + +
    • Description: Gets Pcap Results for a page in PDML format.
      • +
    • Input: +
        + +
      • jobId - Job ID of submitted job
        • +
      • page - Page number
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns PDML in json format.
        • +
      • 404 - Job or page is missing.
        • +
      +
      • +
    +
    +

    GET /api/v1/pcap/{jobId}/raw

    +
      + +
    • Description: Download Pcap Results for a page.
      • +
    • Input: +
        + +
      • jobId - Job ID of submitted job
        • +
      • page - Page number
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns Pcap as a file download.
        • +
      • 404 - Job or page is missing.
        • +
      +
      • +
    +
    +

    DELETE /api/v1/pcap/kill/{jobId}

    +
      + +
    • Description: Kills running job.
      • +
    • Input: +
        + +
      • jobId - Job ID of submitted job
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Kills passed job.
        • +
      +
      • +
    +
    +

    GET /api/v1/pcap/{jobId}/config

    +
      + +
    • Description: Gets job configuration for Pcap query job.
      • +
    • Input: +
        + +
      • jobId - Job ID of submitted job
        • +
      +
      • +
    • Returns: +
        + +
      • 200 - Returns a map of job properties for the Job ID.
        • +
      • 404 - Job is missing.
        • +
      +
      • +
    +

    POST /api/v1/search/search

      @@ -1865,8 +2049,8 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
    • Returns:
        -
      • 200 - nothing
        • -
      • 404 - document not found
        • +
      • 200 - Nothing
        • +
      • 404 - Document not found
    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-platform/Performance-tuning-guide.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/Performance-tuning-guide.html b/site/current-book/metron-platform/Performance-tuning-guide.html index 00d2907..b9134de 100644 --- a/site/current-book/metron-platform/Performance-tuning-guide.html +++ b/site/current-book/metron-platform/Performance-tuning-guide.html @@ -1,13 +1,13 @@ - + Metron – Metron Performance Tuning Guide @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Metron Performance Tuning Guide
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -55,15 +55,16 @@
  • Platform
    • @@ -781,7 +782,7 @@ enrichments enrichments 43 29754331 297
    -
    /usr/metron/0.5.0/bin/start_parser_topology.sh \
+
    /usr/metron/0.6.0/bin/start_parser_topology.sh \
     -e ~metron/.storm/storm-bro.config \
     -esc ~/.storm/spout-bro.config \
     -k $BROKERLIST \
@@ -966,7 +967,7 @@ export KAFKA_HOME=$HDP_HOME/kafka-broker
 export STORM_UI=http://node1:8744
 export ELASTIC=http://node1:9200
 export ZOOKEEPER=node1:2181
-export METRON_VERSION=0.5.0
+export METRON_VERSION=0.6.0
 export METRON_HOME=/usr/metron/${METRON_VERSION}
    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-platform/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/index.html b/site/current-book/metron-platform/index.html index 7819640..9fa8d70 100644 --- a/site/current-book/metron-platform/index.html +++ b/site/current-book/metron-platform/index.html @@ -1,13 +1,13 @@ - + Metron – Current Build @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Current Build
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -55,15 +55,16 @@
  • Platform
    • @@ -104,7 +105,7 @@ limitations under the License. -->

    Current Build

    -

    The latest build of metron-platform is 0.5.0.

    +

    The latest build of metron-platform is 0.6.0.

    We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.

    metron-platform

    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-platform/metron-api/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-api/index.html b/site/current-book/metron-platform/metron-api/index.html deleted file mode 100644 index ddcaec5..0000000 --- a/site/current-book/metron-platform/metron-api/index.html +++ /dev/null @@ -1,161 +0,0 @@ - - - - - - - - - Metron – Metron PCAP Service - - - - - - - -
    - - - -
    - -
    - -

    Metron PCAP Service

    -

    -

    The purpose of the Metron PCAP service is to provide a middle tier to negotiate retrieving packet capture data which flows into Metron. This packet data is of a form which libpcap based tools can read.

    -
    -

    Starting the Service

    -

    You can start the service either via the init.d script installed, /etc/init.d/pcapservice or directly via the yarn jar command: yarn jar $METRON_HOME/lib/metron-api-$METRON_VERSION.jar org.apache.metron.pcapservice.rest.PcapService -port $SERVICE_PORT -query_hdfs_path $QUERY_PATH -pcap_hdfs_path $PCAP_PATH

    -

    where

    -
      - -
    • METRON_HOME is the location of the metron installation
      • -
    • METRON_VERSION is the version of the metron installation
      • -
    • SERVICE_PORT is the port to bind the REST service to.
      • -
    • QUERY_PATH is the temporary location to store query results. They are deleted after the service reads them.
      • -
    • PCAP_PATH is the path to the packet data on HDFS
      • -
    -
    -

    The /pcapGetter/getPcapsByIdentifiers endpoint

    -

    This endpoint takes the following query parameters and returns the subset of packets matching this query:

    -
      - -
    • srcIp : The source IP to match on
      • -
    • srcPort : The source port to match on
      • -
    • dstIp : The destination IP to match on
      • -
    • dstPort : The destination port to match on
      • -
    • startTime : The start time in milliseconds
      • -
    • endTime : The end time in milliseconds
      • -
    • numReducers : Specify the number of reducers to use when executing the mapreduce job
      • -
    • includeReverseTraffic : Indicates if filter should check swapped src/dest addresses and IPs
      • -
    -
    -

    The /pcapGetter/getPcapsByQuery endpoint

    -

    This endpoint takes the following query parameters and returns the subset of packets matching this query. This endpoint exposes Stellar querying capabilities:

    -
      - -
    • query : The Stellar query to execute
      • -
    • startTime : The start time in milliseconds
      • -
    • endTime : The end time in milliseconds
      • -
    • numReducers : Specify the number of reducers to use when executing the mapreduce job
      • -
    -

    Example: curl -XGET "http://node1:8081/pcapGetter/getPcapsByQuery?query=ip_src_addr+==+'192.168.66.121'+and+ip_src_port+==+'60500'&startTime=1476936000000"

    -

    All of these parameters are optional. In the case of a missing parameter, it is treated as a wildcard.

    -

    Unlike the CLI tool, there is no paging mechanism. The REST API will stream back data as a single file.

    -
    -
    -
    -
    -
    -
    -
    -© 2015-2016 The Apache Software Foundation. Apache Metron, Metron, Apache, the Apache feather logo, - and the Apache Metron project logo are trademarks of The Apache Software Foundation. -
    -
    -
    - - http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-platform/metron-common/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-common/index.html b/site/current-book/metron-platform/metron-common/index.html index 8d0dc0e..caa5a3b 100644 --- a/site/current-book/metron-platform/metron-common/index.html +++ b/site/current-book/metron-platform/metron-common/index.html @@ -1,13 +1,13 @@ - + Metron – Contents @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Contents
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -55,15 +55,16 @@
  • Platform
    • @@ -231,6 +232,16 @@ limitations under the License.
    + + + + + + + + + + @@ -246,10 +257,30 @@ limitations under the License. + + + + + + + + + + + + + + + - + + + + + +
    POST /api/v1/alerts/ui/escalate
    POST /api/v1/alerts/ui/escalate
    GET /api/v1/alerts/ui/settings
    GET /api/v1/metaalert/update/status/{guid}/{status}
    POST /api/v1/pcap/fixed
    POST /api/v1/pcap/query
    GET /api/v1/pcap
    GET /api/v1/pcap/{jobId}
    GET /api/v1/pcap/{jobId}/pdml
    GET /api/v1/pcap/{jobId}/raw
    DELETE /api/v1/pcap/kill/{jobId}
    GET /api/v1/pcap/{jobId}/config
    GET /api/v1/search/search
    POST /api/v1/search/search
    POST /api/v1/search/search
    POST /api/v1/search/group
    POST /api/v1/search/group
    GET /api/v1/search/findOne
    PATCH /api/v1/update/patch
    PUT /api/v1/update/replace
    PUT /api/v1/update/replace
    GET /api/v1/user
    String profiler_period_units
    profiler.writer.batchSize Profiler Integer N/A
    profiler.writer.batchTimeout Profiler Integer N/A
    update.hbase.table REST/Indexing String String geo_hdfs_file
    enrichment.writer.batchSize Enrichment Integer N/A
    enrichment.writer.batchTimeout Enrichment Integer N/A
    geo.hdfs.file Enrichment String geo_hdfs_file
    source.type.field UI String N/A
    source_type_field
    threat.triage.score.field UI String threat_triage_score_field
    http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-platform/metron-data-management/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-data-management/index.html b/site/current-book/metron-platform/metron-data-management/index.html index dea600c..610c5c6 100644 --- a/site/current-book/metron-platform/metron-data-management/index.html +++ b/site/current-book/metron-platform/metron-data-management/index.html @@ -1,13 +1,13 @@ - + Metron – Resource Data Management @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Resource Data Management
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -55,15 +55,16 @@
  • Platform
    • http://git-wip-us.apache.org/repos/asf/metron/blob/a97e575f/site/current-book/metron-platform/metron-elasticsearch/index.html ---------------------------------------------------------------------- diff --git a/site/current-book/metron-platform/metron-elasticsearch/index.html b/site/current-book/metron-platform/metron-elasticsearch/index.html index cc360b1..bf3c630 100644 --- a/site/current-book/metron-platform/metron-elasticsearch/index.html +++ b/site/current-book/metron-platform/metron-elasticsearch/index.html @@ -1,13 +1,13 @@ - + Metron – Elasticsearch in Metron @@ -32,8 +32,8 @@
  • Metron/
  • Documentation/
  • Elasticsearch in Metron
    • -
  • | Last Published: 2018-06-07
    • -
  • Version: 0.5.0
    • +
  • | Last Published: 2018-09-12
    • +
  • Version: 0.6.0
    • @@ -55,15 +55,16 @@
  • Platform
    • @@ -405,13 +406,13 @@ limitations under the License.

    Using Metron with Elasticsearch 5.6.2

    -

    There is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field. See Ignoring Unmapped Fields for more information

    +

    There is a requirement that all sensors templates have a nested metron_alert field defined. This field is a dummy field. See Ignoring Unmapped Fields for more information

    Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service’s logs.

    Exception seen:

    -
    QueryParsingException[[nested] failed to find nested object under path [alert]];
+
    QueryParsingException[[nested] failed to find nested object under path [metron_alert]];

    There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field.

    @@ -424,7 +425,7 @@ export SENSOR="bro" curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true" -o "${SENSOR}.template" sed -i '' '2d;$d' ./${SENSOR}.template sed -i '' '/"properties" : {/ a\ -"alert": { "type": "nested"},' ${SENSOR}.template +"metron_alert": { "type": "nested"},' ${SENSOR}.template

    To manually verify this, you can optionally pretty print it again with:

    @@ -448,7 +449,7 @@ sed -i '' '/"properties" : {/ a\ 
    curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d '
 {
   "properties" : {
-    "alert" : {
+    "metron_alert" : {
       "type" : "nested"
     }
   }