nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (NIFI-856) Add Processor for Lumberjack protocol
Date Mon, 04 Apr 2016 00:22:25 GMT


ASF GitHub Bot commented on NIFI-856:

Github user trixpan commented on the pull request:
    @bbende @joewitt can one of you guys bump travis-ci ? 
    this should pass build without dramas

> Add Processor for Lumberjack protocol
> -------------------------------------
>                 Key: NIFI-856
>                 URL:
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: Mike de Rhino 
>              Labels: features
> It would be great if NIFI could support the [lumberjack protocol|]
so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes and instead
of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data
shipping mechanisms. 
> Kafka is great but its architecture seem to be better focused on multi-DC environments
instead of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for
each country where you operate?)
> [Heka|] is fine, it has decent backpressure buffering
but no concept of acknowledgement on the receiving side of a TCP stream. If the other end
of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting
data through the pipe, oblivious to the woes at the other end.
> Logstash forwarder in the other hand, is a quite simple tool, with a reasonable implementation
of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its
own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack messages
and offloading some of the hard work Logstash seems to struggle with (e.g. using NIFI to save
to HDFS while a downstream Logstash writes into ES).

This message was sent by Atlassian JIRA

View raw message