flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kajal K Das" <kajal....@in.ibm.com>
Subject Re: Looking for a specific input on JMS source
Date Thu, 31 Aug 2017 09:37:59 GMT
Thank you, Muhammad. Much appreciated.


Regards
Kajal Kr. Das
Solution Architect - Etihad Ops BI
IBM
SUT #: +91 (0)80 4318 2743 | India Mobile #: +91 (0)98301 15013 
E-mail: kajal.das@in.ibm.com 



From:   Muhammad Yaseen <yaseenaftab.gt@gmail.com>
To:     user@flume.apache.org
Date:   08/31/2017 12:47 PM
Subject:        Re: Looking for a specific input on JMS source



Hello,

I think you can achieve what you want with the JMSSourceConverter. Looking 
at the Flume documentation it says that "The JMS source allows pluggable 
converters, though it?s likely the default converter will work for most 
purposes. The default converter is able to convert Bytes, Text, and Object 
messages to FlumeEvents. In all cases, the properties in the message are 
added as headers to the FlumeEvent". 

Perhaps you can implement the JSMMessageConverter interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConverter.java
). The default converted also implements this interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java
) and also provides a good reference point for implementing you own.

You custom code would go into plugins.d directory in flume. More details 
here: https://flume.apache.org/FlumeUserGuide.html#jms-source

On Thu, Aug 31, 2017 at 12:05 PM, Kajal K Das <kajal.das@in.ibm.com> 
wrote:
Hi All,

We are designing a near-real time pipeline in which we will receive 
messages from IBM MQ. On the Hadoop side, we are planning to use Flume to 
ingest the messages using JMS source.
The messages will include some custom header information, which we need to 
extract as well. We discovered that interceptors cannot be configured with 
a JMS source.
In that case: what are the alternatives in order to extract the message 
headers using Flume? Or should we try something else (e.g. Kafka)?



Regards
Kajal Kr. Das






Mime
View raw message