camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Question: camel-sql out of memory error
Date Thu, 14 Aug 2014 13:22:17 GMT
Hi

IMHO you should only set that fetch limit if getMaxMessagesPerPoll >
0. So the user needs to set that option explicit if he/she want to
limit it.



On Wed, Aug 13, 2014 at 2:07 PM, Christian Müller
<christian.mueller@gmail.com> wrote:
> At present, we do not limit the fetch size on the JDBC driver level which
> will fix this issue (e.g.
> preparedStatement.setFetchSize(getMaxMessagesPerPoll())).
> The drawback is, we don't know how many rows are in the database which will
> match our query. We only can assume if fetchSize is equal to
> maxMessagesPerPoll, there are more rows to read.
>
> From my point of view, we should go for this change. If nobody has
> objections, I will will do the needed changes later this week...
>
> Best,
>
> Christian
> -----------------
>
> Software Integration Specialist
>
> Apache Member
> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer
> Apache Incubator PMC Member
>
> https://www.linkedin.com/pub/christian-mueller/11/551/642
>
>
> On Fri, Aug 8, 2014 at 10:34 PM, Matt Payne <Payne@mattpayne.org> wrote:
>
>> I am getting an  java.lang.OutOfMemoryError: Java heap space when using
>> camel sql component with a query that returns a large number of rows.
>>
>> Using a small value for maxMessagesPerPoll=10 does not help[1].
>>
>> When reading the source[2], I see:
>> 255   protected List<Map<String, Object>> queryForList(ResultSet rs)
throws
>> SQLException {
>> 256         ColumnMapRowMapper rowMapper = new ColumnMapRowMapper();
>> 257         RowMapperResultSetExtractor<Map<String, Object>> mapper =
new
>> RowMapperResultSetExtractor<Map<String, Object>>(rowMapper);
>> 258         List<Map<String, Object>> data = mapper.extractData(rs);
>> 259         return data;
>> 260     }
>>
>> It seems that all of the result set, rs, is going to be read regardless of
>> how large it is.   When stepping through via eclipse's debugger this is
>> what I see happening.
>>
>> It's unclear how to ask camel sql component to only take X rows at a time.
>>   Is there a way to do this please?
>>
>> Thanks! --Matt Payne
>>
>>
>> [1]
>> http://camel.465427.n5.nabble.com/Fetching-data-in-batches-td5718366.html
>> [2]
>>
>> https://git-wip-us.apache.org/repos/asf?p=camel.git;a=blob;f=components/camel-sql/src/main/java/org/apache/camel/component/sql/SqlEndpoint.java#l255
>>



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/

Mime
View raw message