Return-Path: Delivered-To: apmail-camel-users-archive@www.apache.org Received: (qmail 82830 invoked from network); 19 Aug 2010 06:24:27 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 Aug 2010 06:24:27 -0000 Received: (qmail 69686 invoked by uid 500); 19 Aug 2010 06:24:27 -0000 Delivered-To: apmail-camel-users-archive@camel.apache.org Received: (qmail 69386 invoked by uid 500); 19 Aug 2010 06:24:24 -0000 Mailing-List: contact users-help@camel.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@camel.apache.org Delivered-To: mailing list users@camel.apache.org Received: (qmail 69378 invoked by uid 99); 19 Aug 2010 06:24:23 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 06:24:23 +0000 X-ASF-Spam-Status: No, hits=2.3 required=10.0 tests=FREEMAIL_FROM,SPF_HELO_PASS,SPF_SOFTFAIL,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of v_cheruvu@hotmail.com does not designate 216.139.236.158 as permitted sender) Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 06:24:01 +0000 Received: from sam.nabble.com ([192.168.236.26]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from ) id 1OlyXN-0004f4-0o for users@camel.apache.org; Wed, 18 Aug 2010 23:23:41 -0700 Date: Wed, 18 Aug 2010 23:23:41 -0700 (PDT) From: vcheruvu To: users@camel.apache.org Message-ID: <1282199021017-2640474.post@n5.nabble.com> In-Reply-To: References: <1282097628380-2638903.post@n5.nabble.com> Subject: Re: Java heap space issue with reading large CSV file MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org I have changed my logging level to INFO but it didn't solve memory issue. I have turned on JConsole noticed that, memory shoots up to 1.05GB when Camel reads in 218K lines, 45 MB file and then throws heap memory issue with Jconsole RMI which is trying to get memory info of the application. I am bit puzzled as to what happening in the Camel-File component when it reads in all the line from the csv file. I could only get around the issue only if I split the file into 10K lines per file. In this case, camel reads all the 20 files one by one, then it reaches memory usage upto 900MB and but doesn't crash with heap memory issue. Why didn't heap memory occurred in this scenario? is this because, JVM had sufficient time to GC objects? or application JVM settings for GC is poor? -Xms1024m -Xmx1024m -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8 -XX:NewSize=128m -XX:MaxNewSize=128m -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:PermSize=64m -XX:MaxPermSize=64m -XX:+UseAdaptiveSizePolicy I have attached jconsole memory graph for both cases. Both cases , I have set the same JVM settings. I guess it will be nice addition if we could have option for file consumer to limit number of lines to be read in for every poll regardless of number files. This will give some control on application to manage memory, throughput and stability. http://camel.465427.n5.nabble.com/file/n2640474/heapspace-singlefile.bmp heapspace-singlefile.bmp http://camel.465427.n5.nabble.com/file/n2640474/heapspace-multifile.bmp heapspace-multifile.bmp -- View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-issue-with-reading-large-CSV-file-tp2638903p2640474.html Sent from the Camel - Users mailing list archive at Nabble.com.