flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Simon <s...@cloudera.com>
Subject Re: JMS (Tibco) Source - input messages status during transaction
Date Thu, 20 Oct 2016 09:23:22 GMT
Indeed it can be stripped down to the question whether jms read
transactions are committed as a batch or as individual reads.

On Wednesday, 19 October 2016, Roberto Coluccio <roberto.coluccio@eng.it>
wrote:

> Hi,
>
> When leveraging a JMS (Tibco implementation) source to consume messages
> from a queue, are such messages immediately consumed , hence removed from
> the queue, even if the flume transaction hasn't been closed yet? What
> happens if the transaction is not committed (e.g. in case of errors)?
>
> For instance, in a topology made by:
>
> jms source -> memory or file channel -> hdfs sink
>
> with a configured batch size of 100, when 99 of them are read from JMS and
> written to the channel, if an error occurs when trying to read the 100th,
> to the best of my knowledge the whole batch of 100 is discarded, the
> transaction canceled, and the messages tried to be re-processed.
>
> In such a scenario, are the first 99 messages already consumed from JMS to
> be considered as lost, supposing the ones in the channel are lost as well
> (because of other issues)?
>
> thanks a lot in advance.
>
> Best regards,
> Roberto
>


-- 

*Attila Simon*
Software Engineer
Email:   sati@cloudera.com

[image: Cloudera Inc.]

Mime
View raw message