Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 454C6200D2E for ; Tue, 31 Oct 2017 13:49:56 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 43C561609EC; Tue, 31 Oct 2017 12:49:56 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8A0281609EB for ; Tue, 31 Oct 2017 13:49:55 +0100 (CET) Received: (qmail 49958 invoked by uid 500); 31 Oct 2017 12:49:54 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 49946 invoked by uid 99); 31 Oct 2017 12:49:54 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Oct 2017 12:49:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 7791118340E for ; Tue, 31 Oct 2017 12:49:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.044 X-Spam-Level: **** X-Spam-Status: No, score=4.044 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URI_HEX=1.313, URI_TRY_3LD=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id MCbFIImZdv7r for ; Tue, 31 Oct 2017 12:49:52 +0000 (UTC) Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id BA7565FD42 for ; Tue, 31 Oct 2017 12:49:51 +0000 (UTC) Received: by mail-wm0-f43.google.com with SMTP id t139so22985271wmt.1 for ; Tue, 31 Oct 2017 05:49:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=mfQVaCk1EN56AcvZU11nKZQGQ2Ht4JpgvJMlh7Lg5jQ=; b=B6ldTBBeBr1oRqFkeHf78hWng1Rw9noFRS3OtMGUb8kijFpwzjSRns1+0mxU17yR71 AWT1h5TTo23nlztLpBUD021sZIKqZ1ZDE2/l9aIM4cW2aaE5sc5rl4gIzxdbE8SS51qY ifFpaXNEtk6ftMSvIoE7BK9c2A3TYQ+0SOIzp9Kb0eu5ige6yEJlm2GJRuCRiF2lKe0P iiQEk+qxOBZnCqbVhpfoni4lYvqnpmBewR44WCTlfs7PVuVt5rxkXCIOd4Ebz88VD+NL FLCD0e0lmAgXlVTQv2DgjriXZqjCCHpqNAFqg3St5EIVqwLppJxQLrV5P00XUp88D0Bf uB+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=mfQVaCk1EN56AcvZU11nKZQGQ2Ht4JpgvJMlh7Lg5jQ=; b=OM8PZ7uHh57fdvdT+LxAL8c3nMbst8MyJ/A7A1XCwXO3Bf1fT1ZwiPKsFP+9Yw/2ei yC3xq/SqCx0zd0xM93tImVmtIXxzoMyXjBUPTFlAiSN8dqFoTYPTeW53BH2z8U8y/omt uv8WVtdkGPL/coxCaxWCrj4gxaxz43qp0RszC9RqtLtW/xXT+QulNyspRMq1fcOyDAxE wa1+g+Dmj9PrHSyHNdWKMaI2yPhGj46M1rYGxVE5Ei6XH5+r7OiHg662z7/76G9mOzag P7wChMRSHZGlYpbdIaKHSy5Ks1VxpkSGlop0MR/La1WP58rtHUjcUXvbFo9LzT3bGzbh uUNg== X-Gm-Message-State: AMCzsaVWEikvcfwOZE8/3HcxYhMihB2qWCq2eQYi53ekHvYYc4/7z6EZ VmVinRI34S1XCPJzQmaPnTfCF23dsXPaeEGjFfc= X-Google-Smtp-Source: ABhQp+QaMNct1UFrHOYBOc9d3ogXlEBtNs+jPtd9EZSocqNlBOhDHAmquNGsCqCe4kvatoTS4gDWzWFKcJealgDW+eg= X-Received: by 10.28.23.129 with SMTP id 123mr1735729wmx.54.1509454185034; Tue, 31 Oct 2017 05:49:45 -0700 (PDT) MIME-Version: 1.0 Sender: tbain98@gmail.com Received: by 10.223.134.42 with HTTP; Tue, 31 Oct 2017 05:49:44 -0700 (PDT) Received: by 10.223.134.42 with HTTP; Tue, 31 Oct 2017 05:49:44 -0700 (PDT) In-Reply-To: <1509340872141-0.post@n4.nabble.com> References: <1509092386548-0.post@n4.nabble.com> <1509340872141-0.post@n4.nabble.com> From: Tim Bain Date: Tue, 31 Oct 2017 06:49:44 -0600 X-Google-Sender-Auth: kfBoxpIoQ0CVGKAVzWJPR0Upe-8 Message-ID: Subject: Re: MQTT Subscriber gets disconnected frequently when there are large number of MQTT clients connected To: ActiveMQ Users Content-Type: multipart/alternative; boundary="001a114699be80f9a5055cd73051" archived-at: Tue, 31 Oct 2017 12:49:56 -0000 --001a114699be80f9a5055cd73051 Content-Type: text/plain; charset="UTF-8" OK. Those log lines indicate that the far side of the connection (the client) closed the connection when the near end (the broker) wasn't expecting it. The root cause may be something on the broker's side (for example, the broker got so busy that it stopped responding to the client, who reacted by closing the connection), but for now we need to follow the trail over to the client side and see what its logs say. Can you please post whatever content is in the client logs shortly before you see the content you posted from the broker's logs? Generic reasons why the broker could become unresponsive include not having enough heap (so the JVM spends most of its time GCing), having so many connections that it can't service them all quickly enough, disk I/O that's too slow, or of course maybe there's a bug. Some of those (e.g. spending too much time GCing) manifest as high CPU usage, and you said CPU usage is low/normal, but you may still want to double check GC and CPU usage from JConsole or something similar. You could also take a thread dump (via jstack or another method, there are several ways to do this) of your production broker and see if maybe there are lots of threads blocked waiting for a lock while trying to do real work (i.e. not the threads in a thread pool that are blocked trying to get a task from their queue, those are expected and don't indicate a problem). Tim On Oct 29, 2017 11:21 PM, "Shobhana" wrote: > Hi Tim, > > The log content quoted in my post was from broker log. > > I'll try to reproduce this in a test environment. Meanwhile if you could > think of any common (generic) reasons why broker may becomes unresponsive, > please share. > > Thanks, > Shobhana > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html > --001a114699be80f9a5055cd73051--