nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Witt (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-1221) ListenSyslog should support batching
Date Sat, 28 Nov 2015 03:42:10 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15030364#comment-15030364
] 

Joseph Witt commented on NIFI-1221:
-----------------------------------

[~bbende] Can you please take a look at Mark's patch and then mine?  I'm a +1 to Mark's changes.
 Tkurc has reviewed as well and told me out of band he too is favorable.  Just would like
your review to make sure this still seems in range to what you were trying to accomplish.
 Thanks

> ListenSyslog should support batching
> ------------------------------------
>
>                 Key: NIFI-1221
>                 URL: https://issues.apache.org/jira/browse/NIFI-1221
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 0.4.0
>
>         Attachments: 0001-NIFI-1221-Support-batching-of-Syslog-messages.patch, 0001-NIFI-1221-fixing-test-failures-contrib-failed-line-m.patch
>
>
> Currently, performance of ListenSyslog is pretty reasonable. If I configure logging to
WARN level, and I use a Yield Duration of 0 ms, 3 threads, 25 ms run duration, then I can
push about 23,000 messages per second to a single NiFi node without any loss, with message
parsing enabled.
> However, I think we can do a lot better than that. Since these Syslog events are just
log messages, they lend themselves very well to concatenation. We should have a Max Batch
Size property as well as a Message Delimiter property.
> If using batch, though, it's important that we do not allow the Parse Messages property
to be true, since it doesn't really make sense to add attributes if we have multiple messages.
> However, since we cannot parse the messages if bundled together, we should have a separate
ParseSyslog processor that does parse them. This way, we can route specific events to a ParseSyslog
processor. For instance, using RouteText to pull out events of interest.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message