nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: session recover behaviour in nifi-jms-processor
Date Fri, 24 Feb 2017 14:30:16 GMT
Sorry, just noticed the typo. Instead of “limit the possibility of a message loads. . .”
should be "limit the possibility of a message loss…"

Cheers
Oleg
> On Feb 24, 2017, at 9:28 AM, Oleg Zhurakousky <ozhurakousky@hortonworks.com> wrote:
> 
> Dominic
> 
> Great to hear from you!
> As you can see from the inline comment in the code, the recover is there for a reason
primarily to ensure or should I say limit the possibility of a message loads in the event
of a processor and/or NiFi crash. 
> As you may be aware, in NiFi we do prefer message duplication over message loss. That
said, I do see several possibilities for improvement especially for the high traffic scenarios
you are describing. One such improvement would be to create a listening container version
of ConsumeJMS which has far more control over threading and session caching/sharing.
> 
> Would you mind racing a JIRA issue - https://issues.apache.org/jira/browse/NIFI/ describing
everything you just did and we’ll handle it.
> 
> Cheers
> Oleg
> 
>> On Feb 24, 2017, at 9:08 AM, Dominik Benz <dominik.benz@gmail.com> wrote:
>> 
>> Hi,
>> 
>> we're currently using Nifi to consume a relatively high-traffic JMS topic
>> (40-60 messages per second). 
>> 
>> Worked well in principle - however, we then noticed that the the outbound
>> rate (i.e. the number of messages we fetched) of the topic was consistently
>> slightly higher than its inbound rate (i.e. the actual number of messages
>> sent to the topic). This puzzled me, because (being the only subscriber to
>> the topic) I would expect inbound and outbound traffic to be identical
>> (given we can consume fast enough, which we can).
>> 
>> Digging deeper, I found that in 
>> 
>> org.apache.nifi.jms.processors.JMSConsumer
>> 
>> the method "consume" performs a session.recover:
>> 
>> 
>> 
>> session.recover (as written in the comment) basically stops message delivery
>> and re-starts from the last non-acked message. However, I think this leads
>> to the following issue in high-traffic contexts:
>> 
>> 1) several threads perform the JMS session callback in parallel
>> 2) each callback performs a session.recover
>> 3) during high traffic, the situation arises that the ACKs from another
>> thread may not (yet) have arrived at the JMS server
>> 4) this implies that the pointer of session.recover will reconsume the
>> not-yet-acked message from another thread
>> 
>> For verification, I performed so far the following steps:
>> 
>> (a) manual implementation of a simplistic synchronous JMS topic consumer ->
>> inbound/outbound identical as expected
>> (b) patched nifi-jms-processors and commented out session.recover() -
>> inbout/outbound identical as expected
>> 
>> Any thoughts on this? My current impression is that session.recover in its
>> current usage doesn't play well together with the multi-threaded JMS
>> consumers. Or do I have any misconception? 
>> 
>> Thanks & best regards,
>> Dominik
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-tp14940.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>> 
> 

Mime
View raw message