nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-856) Add Processor for Lumberjack protocol
Date Mon, 04 Apr 2016 00:22:25 GMT

    [ https://issues.apache.org/jira/browse/NIFI-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223564#comment-15223564
] 

ASF GitHub Bot commented on NIFI-856:
-------------------------------------

Github user trixpan commented on the pull request:

    https://github.com/apache/nifi/pull/290#issuecomment-205082226
  
    @bbende @joewitt can one of you guys bump travis-ci ? 
    
    this should pass build without dramas


> Add Processor for Lumberjack protocol
> -------------------------------------
>
>                 Key: NIFI-856
>                 URL: https://issues.apache.org/jira/browse/NIFI-856
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: Mike de Rhino 
>              Labels: features
>
> It would be great if NIFI could support the [lumberjack protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes and instead
of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data
shipping mechanisms. 
> Kafka is great but its architecture seem to be better focused on multi-DC environments
instead of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for
each country where you operate?)
> [Heka|https://github.com/mozilla-services/heka] is fine, it has decent backpressure buffering
but no concept of acknowledgement on the receiving side of a TCP stream. If the other end
of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting
data through the pipe, oblivious to the woes at the other end.
> Logstash forwarder in the other hand, is a quite simple tool, with a reasonable implementation
of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its
own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack messages
and offloading some of the hard work Logstash seems to struggle with (e.g. using NIFI to save
to HDFS while a downstream Logstash writes into ES).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message