Mailing-List: contact commits-help@nifi.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.incubator.apache.org
Date: Tue, 13 Jan 2015 18:36:37 +0000 (UTC)
From: "Teresa Jackson (JIRA)" <jira@apache.org>
To: commits@nifi.incubator.apache.org
Message-ID: <JIRA.12767211.1421174193000.72948.1421174197720@Atlassian.JIRA>
In-Reply-To: <JIRA.12767211.1421174193000@Atlassian.JIRA>
References: <JIRA.12767211.1421174193000@Atlassian.JIRA>
 <JIRA.12767211.1421174193689@arcas>
Subject: [jira] [Created] (NIFI-253) Enhancement to the Core to support
 metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Teresa Jackson created NIFI-253:
-----------------------------------

             Summary: Enhancement to the Core to support metrics 
                 Key: NIFI-253
                 URL: https://issues.apache.org/jira/browse/NIFI-253
             Project: Apache NiFi
          Issue Type: Wish
          Components: Core Framework
    Affects Versions: 0.0.1
            Reporter: Teresa Jackson


I'd like to propose an addition or enhancement be made to the Core to support volume management, trend analysis by way of databasing attributes and content so that it is query-able and made available for display. This information would then be used for statistical roll ups, metrics, trend analysis, etc..

Ideally, we'd do it by capturing running totals by receiving copies of local provenance events.  This component would be like local provenance in that it would retain the data for some configurable period of time, based on the amount of disk space allocated for that process.  In addition, these roll ups could be sent somewhere for even longer retention.

The goal is to keep as many hooks as possible to making it possible for other programs/services to ingest both the local provenance logs, and the rolled up summaries.  There's a growing base of people who are comfortable with NIFI graphs, and local provenance, so I think that it makes sense to build off that.

The issue I'm facing is that Provenance is fine for tracking one file if you have a starting point, but it is not designed to do counting, summarization and correlation of data. And it doesn't support advanced queries.

Here are some of the most immediate and pressing use cases for this design.

1.  How much traffic came in yesterday (or last week)?
2. Provide statistical counts on items of interest within a flow for a given flow/date range.
3.  When was the last file sent to "System X"?
4. Did anything get sent to "System Y"?
5. How much data was marked with a certain tag?
6. How much data was scanned?
7. How much data was detected?
8. How much of a particular type of data was received in bytes?
9. How much data was processed by file count?


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)