camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CAMEL-3497) Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
Date Sat, 08 Jan 2011 17:27:47 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979158#action_12979158
] 

Claus Ibsen commented on CAMEL-3497:
------------------------------------

I ran a test with 1.000.000 rows in a file
{code}
2011-01-08 18:25:44,216 [read #9 - Split] INFO  split                          - Received:
1000000 messages so far. Last group took: 50 millis which is: 20,000 messages per second.
average: 17,775.566
2011-01-08 18:25:44,217 [main           ] INFO  SplitterParallelBigFileTest    - Took 57.423
seconds
2011-01-08 18:25:44,218 [://target/split] INFO  route1                         - Done splitting
bigfile.txt
{code}

And the memory usage was at most 33mb at peak.

> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large
amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-3497
>                 URL: https://issues.apache.org/jira/browse/CAMEL-3497
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 2.5.0
>            Reporter: Ralf Steppacher
>            Assignee: Claus Ibsen
>             Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space
for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000
lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange>
object for every token delivered by the java.util.Scanner. The list is only cleared (going
out of scope) after all Future objects have been completed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message