nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Subject Re: session recover behaviour in nifi-jms-processor
Date Fri, 24 Feb 2017 14:28:25 GMT
Dominic

Great to hear from you!
As you can see from the inline comment in the code, the recover is there for a reason primarily
to ensure or should I say limit the possibility of a message loads in the event of a processor
and/or NiFi crash. 
As you may be aware, in NiFi we do prefer message duplication over message loss. That said,
I do see several possibilities for improvement especially for the high traffic scenarios you
are describing. One such improvement would be to create a listening container version of ConsumeJMS
which has far more control over threading and session caching/sharing.

Would you mind racing a JIRA issue - https://issues.apache.org/jira/browse/NIFI/ describing
everything you just did and we’ll handle it.

Cheers
Oleg

> On Feb 24, 2017, at 9:08 AM, Dominik Benz <dominik.benz@gmail.com> wrote:
> 
> Hi,
> 
> we're currently using Nifi to consume a relatively high-traffic JMS topic
> (40-60 messages per second). 
> 
> Worked well in principle - however, we then noticed that the the outbound
> rate (i.e. the number of messages we fetched) of the topic was consistently
> slightly higher than its inbound rate (i.e. the actual number of messages
> sent to the topic). This puzzled me, because (being the only subscriber to
> the topic) I would expect inbound and outbound traffic to be identical
> (given we can consume fast enough, which we can).
> 
> Digging deeper, I found that in 
> 
>  org.apache.nifi.jms.processors.JMSConsumer
> 
> the method "consume" performs a session.recover:
> 
> 
> 
> session.recover (as written in the comment) basically stops message delivery
> and re-starts from the last non-acked message. However, I think this leads
> to the following issue in high-traffic contexts:
> 
> 1) several threads perform the JMS session callback in parallel
> 2) each callback performs a session.recover
> 3) during high traffic, the situation arises that the ACKs from another
> thread may not (yet) have arrived at the JMS server
> 4) this implies that the pointer of session.recover will reconsume the
> not-yet-acked message from another thread
> 
> For verification, I performed so far the following steps:
> 
> (a) manual implementation of a simplistic synchronous JMS topic consumer ->
> inbound/outbound identical as expected
> (b) patched nifi-jms-processors and commented out session.recover() -
> inbout/outbound identical as expected
> 
> Any thoughts on this? My current impression is that session.recover in its
> current usage doesn't play well together with the multi-threaded JMS
> consumers. Or do I have any misconception? 
> 
> Thanks & best regards,
>  Dominik
> 
> 
> 
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-tp14940.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

Mime
View raw message