From users-return-32002-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Wed Sep 26 17:24:10 2012 Return-Path: X-Original-To: apmail-activemq-users-archive@www.apache.org Delivered-To: apmail-activemq-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B5737DE63 for ; Wed, 26 Sep 2012 17:24:10 +0000 (UTC) Received: (qmail 35100 invoked by uid 500); 26 Sep 2012 17:24:10 -0000 Delivered-To: apmail-activemq-users-archive@activemq.apache.org Received: (qmail 35048 invoked by uid 500); 26 Sep 2012 17:24:10 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 35038 invoked by uid 99); 26 Sep 2012 17:24:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Sep 2012 17:24:10 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vindimy@gmail.com designates 209.85.216.171 as permitted sender) Received: from [209.85.216.171] (HELO mail-qc0-f171.google.com) (209.85.216.171) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Sep 2012 17:24:04 +0000 Received: by qcad1 with SMTP id d1so553640qca.2 for ; Wed, 26 Sep 2012 10:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=9flbwbAourFiyVOmkDvaWakxmOQoH+hNtDd1M/8aoc8=; b=Cuz0mUQGa/0kdqnRN2CRSSLAfOg6arzlOYnNJlvSLcRW8VQKaatxHaZkysVFfpYNZ8 NpAr2AH7b9xTqmDFEhxozkS/YkesmpluwWIRbEL2O53PaPoYbDNAZ9IHaoha57ruC/zD FKhndKPgacyT8G1i/PmuA9SnfBYV40vF+5q+b/BKXq0azSUj84cPveCuLFNf/8/LHj/I sLnYYFd+d6uhviKqDdWNmSIWc2zVjEP82f7tWBmgf+z3XvLTolKCa/r+i5iVju8lOvoF tVgeeg/a3pmm52N+DnUDoJXOfXOTdP8xO6SovTBOdzRcbxc+3Bsfe8lzWAZIVE0U2D/o RszA== Received: by 10.224.217.136 with SMTP id hm8mr3459294qab.81.1348680223522; Wed, 26 Sep 2012 10:23:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.49.63.169 with HTTP; Wed, 26 Sep 2012 10:23:03 -0700 (PDT) In-Reply-To: References: From: DV Date: Wed, 26 Sep 2012 10:23:03 -0700 Message-ID: Subject: Re: KahaDB disk utilization To: users@activemq.apache.org Content-Type: text/plain; charset=UTF-8 Though I really like the idea of mKahaDB, it's unusable to us right now because of the bug described in AMQ-3841 - we frequently encounter it with AMQ 5.6 and the fix isn't due until 5.7 is out :/ - DV On Wed, Sep 26, 2012 at 2:54 AM, Claus Ibsen wrote: > Hi > > > > On Wed, Sep 26, 2012 at 5:14 AM, DV wrote: >> Are you absolutely certain that there are exactly 10M messages on the >> broker? I've seen a huge KahaDB store with only a few messages in the >> queue because the earlier published messages weren't de-queued >> ("stuck" messages). >> > > Yes and you can use multi KahaDB to distribute destinations in different files > which can help reduce disk space. You can read a bit about it here: > http://activemq.apache.org/kahadb.html > > >> On the other hand, maybe KahaDB sacrifices space for performance, kind >> of like MongoDB does. >> >> - DV >> >> On Wed, Sep 19, 2012 at 8:29 AM, Oleg Dulin wrote: >>> Dear Distinguished Colleagues: >>> >>> I've been trying to understand something. >>> >>> My queues take Java object messages. Serialized into bytes, each Java object >>> is about 1.5K on average. >>> >>> When I have a backlog of 10 million objects on one of the queues, the disk >>> utilization of kahadb is around 50gigs. But this math here: >>> >>> groovy:000> 1500*10000000/1024/1024/1024; >>> ===> 1.9698386192 >>> >>> means that 10M objects should only take up less than 2gigs. >>> >>> Even if I had 3 queues going with the same objects, the total would be 6gigs >>> max. So why the disk utilization overhead ? How much does a single JMS >>> message need ? >>> >>> >>> -- >>> Regards, >>> Oleg Dulin >>> NYC Java Big Data Engineer >>> http://www.olegdulin.com/ >>> >>> >> >> >> >> -- >> Best regards, Dmitriy V. > > > > -- > Claus Ibsen > ----------------- > Red Hat, Inc. > FuseSource is now part of Red Hat > Email: cibsen@redhat.com > Web: http://fusesource.com > Twitter: davsclaus > Blog: http://davsclaus.com > Author of Camel in Action: http://www.manning.com/ibsen -- Best regards, Dmitriy V.