From users-return-10114-apmail-jackrabbit-users-archive=jackrabbit.apache.org@jackrabbit.apache.org Thu Jan 08 18:18:24 2009 Return-Path: Delivered-To: apmail-jackrabbit-users-archive@locus.apache.org Received: (qmail 92219 invoked from network); 8 Jan 2009 18:18:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 8 Jan 2009 18:18:24 -0000 Received: (qmail 3369 invoked by uid 500); 8 Jan 2009 18:18:22 -0000 Delivered-To: apmail-jackrabbit-users-archive@jackrabbit.apache.org Received: (qmail 3354 invoked by uid 500); 8 Jan 2009 18:18:22 -0000 Mailing-List: contact users-help@jackrabbit.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@jackrabbit.apache.org Delivered-To: mailing list users@jackrabbit.apache.org Received: (qmail 3343 invoked by uid 99); 8 Jan 2009 18:18:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jan 2009 10:18:22 -0800 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [205.144.225.177] (HELO itspmx51.state.ms.us) (205.144.225.177) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jan 2009 18:18:14 +0000 Received: from itspmx51.state.ms.us (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id AE48E51E0 for ; Thu, 8 Jan 2009 12:13:31 -0600 (CST) Received: from hqsosn003.mdes.ms.gov (hqsosn002.mdes.ms.gov [10.80.224.7]) by itspmx51.state.ms.us (Postfix) with ESMTP id 7DFF75187 for ; Thu, 8 Jan 2009 12:13:31 -0600 (CST) Received: from hqsosn002 (hqsosn002.mdes.ms.gov [10.80.2.24] (may be forged)) by hqsosn003.mdes.ms.gov (8.12.10+Sun/8.12.10) with ESMTP id n08IAP2V009182 for ; Thu, 8 Jan 2009 12:10:25 -0600 (CST) Received: from hqccxpw114 ([10.80.240.114]) by hqsosn002.mdes.ms.gov (Sun Java System Messaging Server 6.2-7.05 (built Sep 5 2006)) with ESMTPA id <0KD50015WZN74U80@hqsosn002.mdes.ms.gov> for users@jackrabbit.apache.org; Thu, 08 Jan 2009 12:06:43 -0600 (CST) Date: Thu, 08 Jan 2009 12:17:52 -0600 From: Vijay Pandey Subject: RE: Large DMS - 43 GB - Out Of Memory Errors In-reply-to: <8ce7e61c0901081014gfa13ca5qaddfa65a153685c3@mail.gmail.com> To: users@jackrabbit.apache.org Message-id: <0KD50015XZN74U80@hqsosn002.mdes.ms.gov> MIME-version: 1.0 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Mailer: Microsoft Office Outlook, Build 11.0.5510 Content-type: text/plain; charset=iso-8859-1 Content-transfer-encoding: quoted-printable Thread-index: Aclxu3rP5k4hYszeT3OXgKE8bpsM/wAAckbg X-Virus-Checked: Checked by ClamAV on apache.org Yeah it is more than that --- we do have issues on performance but = currently the system is going out of memory every 2-3 hrs. -----Original Message----- From: Todd Seiber [mailto:todd.seiber@gmail.com]=20 Sent: Thursday, January 08, 2009 12:15 PM To: users@jackrabbit.apache.org Subject: Re: Large DMS - 43 GB - Out Of Memory Errors The Jackrabbit main site and wiki mention performance problems with over = 10K child nodes. Are you exceeding this number? On Thu, Jan 8, 2009 at 1:08 PM, Vijay Pandey = wrote: > We got the jackrabbit 1.5 upgraded without any issue and did the test = on > 1.5 > > on a smaller repository of size around 20 GB. > > This is how our content is setup > > a) Root Node > OurContentMainNode > Node 1 > This node has got around 80000 = flat(non-hierchical) > child nodes and each of these child nodes have = around 5 > to > 20 set of PDF docs (size varies from 15kb to = 70kb) > along with several other properties (around 12 - > string > type properties) > Node 2 -- > Node 3 > Node 4 > Node 5 > > > We are using JackRabbit through RMI -- we have 2 separate applications > accessing JackRabbit. > > This is what we found out through YourKit Java Profiler. We ran this = in > test > environment with only 1 user connected to JackRabbit. > > a) When we execute this piece of code on client side through RMI > > session.getRootNode().getNode("OurContentMainNode").getNode("Node1"); > > Just on the above operation we saw a spike of around 37 MB = memory on > the heap. > > After doing the force garbage collection (through yourkit profiler) = =96 we > saw > around 18 MB memory being reclaimed, but still there was around 19 MB > memory > being used in heap, we ran the snapshot to see which objects have = claimed > so > much of heap. This is what we found out. > > Sun.rmi.transport.WeakRef --- has the size of around 19 = MB > with the following structure. > > Sun.rmi.transport.WeakRef > Org.apache.jackrabbit.rmi.server.ServerNode > Org.apache.jackrabbit.core.NodeImpl > Org.apache.jackrabbit.core.NodeData > = Org.apache.jackrabbit.core.state.NodeState > > Org.apache.jackrabbit.core.state.ChildNodeEntries ----- this one = occupied > 19 > MB > Org.apache.commons.collections.map.LinkedMap > Java.util.HashMap > > > After doing some more force GC, around 10 mints later this memory was > reclaimed. > > > Is this issue just because we have too many nodes (non-hierarchical) = or > it's > a combination of too many non-hierarchical nodes and the JackRabbit = RMI > usage. > > Any hint how we can get over this problem will be great. > > Thanks > Vijay > > -----Original Message----- > From: Jukka Zitting [mailto:jukka.zitting@gmail.com] > Sent: Tuesday, January 06, 2009 3:13 AM > To: users@jackrabbit.apache.org > Subject: Re: Large DMS - 43 GB - Out Of Memory Errors > > Hi, > > On Tue, Jan 6, 2009 at 9:32 AM, Thomas M=FCller = > wrote: > > Upgrading is a good idea, you will have to do a full export/import > however. > > Only if you decide to start using the DataStore or bundle persistence > features, both of which should provide nice performance benefits in > this case. > > But you should be able to upgrade to Jackrabbit 1.5 simply by > replacing all the relevant jar files (note that 1.5 uses quite a bit > more jars than 1.0.1). Your existing configuration files and > repository data should work fine with Jackrabbit 1.5. > > BR, > > Jukka Zitting > > > ----------------------- > Hi, > > > > We are using Jackrabbit 1.0.1 now for more than 2 years. Currently we = are > facing a lot of issues related to the Out Of Memory exception, goes = down > around 3 times a day. These are the relevant details of the DMS. > > > > a) JackRabbit 1.0.1 - set up as a standalone RMI server > > b) Its primarily used to store PDF documents ( size vary from 50 = KB > to > 100 KB for each document) in binary format > > c) We have a nightly java batch programs that inserts around = 5000 to > 6000 of these documents every day > > d) During the day time these documents are accessed by the user = from > a > web application - we connect through RMI to the jackrabbit server and fetch > the document > > e) The database used is DB2 and the documents ( blobs) are = stored on > the file system ( setting for externalblobs is true) > > f) The total size of the repository has reached around 43 GB > > > > Will really appreciate if you all can throw some light on this and = what can > be done to resolve this issue? > > > > I am also thinking of migrating to 1.5 Jackrabbit, what I have read is > other > than various jar file changes; it should be compatible to the 1.x = release? > I > also read about the DataStore, but to use it do I have to do the full > export/import? > > > > Thanks > > Vijay > > --=20 Todd Seiber 830 Fishing Creek Rd. New Cumberland, PA 17070 h. 717-938-5778 c. 717-497-1742 e. todd.seiber@gmail.com