From dev-return-18849-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Thu Feb 28 09:32:48 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id B79D1180657 for ; Thu, 28 Feb 2019 10:32:47 +0100 (CET) Received: (qmail 90941 invoked by uid 500); 28 Feb 2019 09:32:46 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 90930 invoked by uid 99); 28 Feb 2019 09:32:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Feb 2019 09:32:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 03B1818191B for ; Thu, 28 Feb 2019 09:32:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.173 X-Spam-Level: *** X-Spam-Status: No, score=3.173 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_PASS=-0.001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id EJ5V3z7BIwPh for ; Thu, 28 Feb 2019 09:32:42 +0000 (UTC) Received: from n7.nabble.com (n7.nabble.com [162.253.133.57]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 8D97A623E8 for ; Thu, 28 Feb 2019 09:32:42 +0000 (UTC) Received: from n7.nabble.com (localhost [127.0.0.1]) by n7.nabble.com (Postfix) with ESMTP id 99B6AA0A0823 for ; Thu, 28 Feb 2019 02:32:35 -0700 (MST) Date: Thu, 28 Feb 2019 02:32:35 -0700 (MST) From: Dominik Benz To: dev@nifi.apache.org Message-ID: <1551346355626-0.post@n7.nabble.com> Subject: nifi-jms-processor message loss for non-durable shared topic subsriptions MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit We're using Nifi a lot to interact with a JMS middleware (Tibco EMS); see also my previous post http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-jms-processor-not-closing-connection-properly-in-quot-primary-node-only-quot-execution-mode-td15967.html For the latter problem, the new possibilities of configuring shared (JMS 2.0) subscriptions introduced by https://issues.apache.org/jira/browse/NIFI-3983 seemed promising, especially for our clustered setup (3 Nifi instances, which could then share the load of JMS consumption equally). However, we are experiencing message loss in the following configuration of the ConsumeJMS Processor: PROPERTIES (...) destination type: TOPIC Acknowlegement mode: CLIENT_ACKNOWLEDGE durable subscription: FALSE shared subscription: TRUE subscription name: my-subscription SCHEDULING. (...) Strategy: timer-driven Concurrent Tasks: 1 Run Schedule: 0 sec Execution: Primary Node We're currently running Nifi 1.7.0. The message loss occurs for comparatively high-traffic topics. We'v established a controlled reproducible setup of sending 1000 messages, from which we are losing roughly 200-400. When increasing the concurrent tasks, we are loosing much less (even down to 0-9 messages), but still substantially. After checking the source code (especially https://github.com/apache/nifi/blob/rel/nifi-1.7.0/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSConsumer.java) and its integration of spring-jms, my first suspicion was that we hit a spring-jms problem similar to the one described here https://stackoverflow.com/questions/43083199/spring-jms-template-sync-receive-on-non-durable-subscriber-message-loss. We extended our test setup by a spring-jms consumer implementation similar to the one of Nifi; however, in this setup, we're consistently receiving all messages, which makes this bug as root cause less probable. Interestingly, in the following to situations, we do NOT experience message loss: (1) non-shared subscription (2) shared durable subscription Both are not really an option for our case, as (1) still has the problem when the primary node switches and (2) is currently not desired by our Tibco admins. My current impression is that the way how Nifi schedules / triggers the JMS consumption introduces little "breaks" in the shared non-durable JMS subscriptions; messages which occur during these breaks are consequently lost. Before going deeper into this direction by myself, it would be great to have an opinion if this direction is promising? Or any other thoughts / experiences on this topic? -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/