flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Archana Ravindran <srarchan...@gmail.com>
Subject Re: Flume HTTP Source Handler Request size limit
Date Thu, 31 Aug 2017 08:28:50 GMT
You can try using blob handler in http source. Though blob is for binary,
we were able to push relatively large event.

For size limit, in our case we were not able to find default support in
flume. So we customized the code with a small function for checking the
size.
You can try similar.

Thanks,
Archana.


On Aug 31, 2017 12:29 PM, "Muhammad Yaseen" <yaseenaftab.gt@gmail.com>
wrote:

Hello,

I am using an http source for my Flume agent and would like to know if
there a size limit (explicit or implicit) on POST request body/data
(content length) for Flume's HTTP Source ?

I have a handler which is throwing* "MalformedJsonException: Unterminated
string at line 1 column 966657*" (number varies) . I know for a fact that
the JSON is correct, for some reason the Handler is dropping the POST body.
This leads to *"HTTPBadRequestException: Request has invalid JSON Syntax."*

The source receives a huge request with an array of JSON objects as events
(10000-30000 events per request body).
This problem occurs when the content length ( request.getContentLength() )
is large e.g. 2142675 but works fine when the length is smaller, e.g. 38320.

Has anyone experienced this ?

I am interested in the reason causing this problem.

Regards,
Muhammad Yaseen

Mime
View raw message