nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Benz <>
Subject nifi-jms-processor message loss for non-durable shared topic subsriptions
Date Thu, 28 Feb 2019 09:32:35 GMT
We're using Nifi a lot to interact with a JMS middleware (Tibco EMS); see
also my previous post

For the latter problem, the new possibilities of configuring shared (JMS
2.0) subscriptions introduced by seemed promising, especially
for our clustered setup (3 Nifi instances, which could then share the load
of JMS consumption equally).

However, we are experiencing message loss in the following configuration of
the ConsumeJMS Processor:

destination type: TOPIC
Acknowlegement mode: CLIENT_ACKNOWLEDGE
durable subscription: FALSE
shared subscription: TRUE
subscription name: my-subscription

Strategy: timer-driven
Concurrent Tasks: 1
Run Schedule: 0 sec
Execution: Primary Node

We're currently running Nifi 1.7.0. The message loss occurs for
comparatively high-traffic topics. We'v established a controlled
reproducible setup of sending 1000 messages, from which we are losing
roughly 200-400. When increasing the concurrent tasks, we are loosing much
less (even down to 0-9 messages), but still substantially.

After checking the source code (especially
and its integration of spring-jms, my first suspicion was that we hit a
spring-jms problem similar to the one described here
We extended our test setup by a spring-jms consumer implementation similar
to the one of Nifi; however, in this setup, we're consistently receiving all
messages, which makes this bug as root cause less probable.

Interestingly, in the following to situations, we do NOT experience message
(1) non-shared subscription
(2) shared durable subscription
Both are not really an option for our case, as (1) still has the problem
when the primary node switches and (2) is currently not desired by our Tibco

My current impression is that the way how Nifi schedules / triggers the JMS
consumption introduces little "breaks" in the shared non-durable JMS
subscriptions; messages which occur during these breaks are consequently
lost. Before going deeper into this direction by myself, it would be great
to have an opinion if this direction is promising? Or any other thoughts /
experiences on this topic?

Sent from:

View raw message