camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CAMEL-3497) Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large amounts > of heap space for big original messages
Date Sat, 08 Jan 2011 10:49:46 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12979113#action_12979113
] 

Claus Ibsen commented on CAMEL-3497:
------------------------------------

Okay good news. I refactored the logic so Camel now aggregates the parallel tasks on-the-fly.

This makes a tremendous difference. Now I can split a file into 50.000 sub messages and process
that in 7 sec, using at most 18mb.
Before I would hit an issue at about 25.000-30.000 message and hit OOME with 130mb.

Since the logic is more complex because there is a separate tasks which aggregates on the
fly, while the other task submit new tasks, there is logic to signal between the two tasks.
They kinda need to agree when there are no more messages to split, and when it has aggregated
all of those.

> Splitter Component: Setting 'streaming="true" parallelProcessing="true"' consumes large
amounts > of heap space for big original messages
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-3497
>                 URL: https://issues.apache.org/jira/browse/CAMEL-3497
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-core
>    Affects Versions: 2.5.0
>            Reporter: Ralf Steppacher
>            Assignee: Claus Ibsen
>             Fix For: 3.0.0
>
>
> Setting 'streaming="true" parallelProcessing="true"' consumes large amounts of heap space
for big original messages. E.g. 1024m of heap is not enough to process an 80Mb with 500'000
lines, splitting it line by line.
> The problem seems to be the ArrayList in MulticastProcessor line 224. It contains a Future<Exchange>
object for every token delivered by the java.util.Scanner. The list is only cleared (going
out of scope) after all Future objects have been completed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message