nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike de Rhino (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (NIFI-856) Add Processor for Lumberjack protocol
Date Sat, 15 Aug 2015 12:05:45 GMT

     [ https://issues.apache.org/jira/browse/NIFI-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mike de Rhino  updated NIFI-856:
--------------------------------
    Description: 
It would be great if NIFI could support the [lumberjack protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
so to enable the use of logstash forwarder as a source of data.

A lot of non Java shops tend to avoid installing Java at data producing nodes and instead
of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data
shipping mechanisms. 

Kafka is great but its architecture seem to be better focused on multi-DC environments instead
of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for each country
where you operate?)

[Heka|https://github.com/mozilla-services/heka] is fine, it has decent backpressure buffering
but no concept of acknowledgement on the receiving side of a TCP stream. If the other end
of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting
data through the pipe, oblivious to the woes at the other end.

Logstash forwarded in the other hand, is a quite simple tool, with a reasonable implementation
of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its
own issues).

It would be great if NIFI could serve as a middle man, receiving lumberjack messages and offloading
some of the hard work Logstash seems to struggle with (e.g. using NIFI to save to HDFS while
a downstream Logstash writes into ES).

  was:
It would be great if NIFI could support the lumberjack protocol so to enable the use of logstash
forwarder as a source of data.

A lot of non Java shops tend to avoid installing Java at data producing nodes and instead
of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data
shipping mechanisms. 

Kafka is great but its architecture seem to be better focused on multi-DC environments instead
of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for each country
where you operate?)

[Heka|https://github.com/mozilla-services/heka] is fine, it has decent backpressure buffering
but no concept of acknowledgement on the receiving side of a TCP stream. If the other end
of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting
data through the pipe, oblivious to the woes at the other end.

Logstash forwarded in the other hand, is a quite simple tool, with a reasonable implementation
of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its
own issues).

It would be great if NIFI could serve as a middle man, receiving [lumberjack messages|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
and offloading some of the hard work Logstash seems to struggle with (e.g. using NIFI to save
to HDFS while a downstream Logstash writes into ES).

        Summary: Add Processor for Lumberjack protocol  (was: Add suport to Lumberjack protocol)

> Add Processor for Lumberjack protocol
> -------------------------------------
>
>                 Key: NIFI-856
>                 URL: https://issues.apache.org/jira/browse/NIFI-856
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: Mike de Rhino 
>              Labels: features
>
> It would be great if NIFI could support the [lumberjack protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes and instead
of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data
shipping mechanisms. 
> Kafka is great but its architecture seem to be better focused on multi-DC environments
instead of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for
each country where you operate?)
> [Heka|https://github.com/mozilla-services/heka] is fine, it has decent backpressure buffering
but no concept of acknowledgement on the receiving side of a TCP stream. If the other end
of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting
data through the pipe, oblivious to the woes at the other end.
> Logstash forwarded in the other hand, is a quite simple tool, with a reasonable implementation
of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its
own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack messages
and offloading some of the hard work Logstash seems to struggle with (e.g. using NIFI to save
to HDFS while a downstream Logstash writes into ES).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message