camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkatesh.Krishnamoorthy <Venkatesh.Krishnamoor...@target.com>
Subject RE: Out of Memory For huge Json Response
Date Fri, 18 Apr 2014 15:28:18 GMT
Hi Fernando,

We get one time load of all data from the source system and this will happen from time to
time on an on-demand basis. The source system will push all data – huge json response over
http. The current camel code is trying to load this data on limited heap instead of in chunks
and stream it into a file. We want to avoid increase in jvm heap with increase in the response
size by using the best Camel method to stream data in chunks.
Appreciate your quick response and looking forward to a resolution.

Thanks & Regards,
Venkat


From: Fernando Ribeiro [mailto:fernando.ribeiro@upic.com.br]
Sent: Friday, April 18, 2014 10:19 AM
To: users@camel.apache.org
Cc: Rajbarath.Govindan; Venkatesh.Krishnamoorthy
Subject: Re: Out of Memory For huge Json Response

Why is the payload so big?

On Fri, Apr 18, 2014 at 9:32 AM, Karthik.Pothuru <Karthik.Pothuru@target.com<mailto:Karthik.Pothuru@target.com>>
wrote:
Dear Team,

We are facing Out of memory issue when  receiving a huge json response (around 1 GB ) when
hitting a REST webservice with in Target Enterprise. We are currently using camel Http component
.
Due to huge file, the entire data gets loaded into HEAP and the process is failing due to
that.

Our requirement is to read this response and store it into a file which will later be parsed
for further data processing.

Due to this issue, we are unable to go the next step of parsing. The same code works fine
when small volume of data is received through the API. (at present we remember processing
a response size of 150 MB and it was successful).

Kindly need your input how to read response with lesser memory foot print.

Thanks
Karthik Pothuru
Mob: +91 91766 97533<tel:%2B91%2091766%2097533>



--
Fernando Ribeiro
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message