flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Ross <richard.prion...@gmail.com>
Subject Re: Flume JMS source
Date Thu, 06 Mar 2014 20:10:51 GMT
Thats correct.

On Mar 6, 2014, at 1:09 PMEST, Madabhattula Rajesh Kumar wrote:

> 
> Hello Richard,
> 
> Thank you very much. I'll try this configuration. Could you please clarify below questions
> 
> a1.sources.r1.providerURL = tcp://<activemqserver>:61616 
> a1.sources.r1.destinationName = TESTQUEUE -----> This is queue name right?
> a1.sources.r1.destinationType = QUEUE 
> 
> Regards,
> Rajesh
> 
> 
> On Tue, Mar 4, 2014 at 8:13 PM, richard ross <richard.prionses@gmail.com> wrote:
> Hello Rajesh:
> 
> I just worked on a POC for the work that I am doing with JMS --> HDFS configuration
using ActiveMQ. I was able to use everything right out of the box, going with JMS Source -->
Memory Channel --> HDFS Sink, based on the guide (https://flume.apache.org/FlumeUserGuide.html).
Below is my example config that I was playing around with. Note that for ActiveMQ, you do
not want to specify the connectionFactory in the source configuration (https://flume.apache.org/FlumeUserGuide.html#jms-source).
A documentation defect was created for this point (https://issues.apache.org/jira/browse/FLUME-2339).
As for questions 3 and 4, I will let someone with more experience answer.
> 
> # Name the components on this agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
> 
> # Describe the source
> #a1.sources.r1.type = netcat
> #a1.sources.r1.bind = localhost
> #a1.sources.r1.port = 44444
> a1.sources.r1.type = jms
> a1.sources.r1.channels = c1
> a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
> a1.sources.r1.providerURL = tcp://<activemqserver>:61616
> a1.sources.r1.destinationName = TESTQUEUE
> a1.sources.r1.destinationType = QUEUE
> 
> # Describe the sink
> #a1.sinks.k1.type = logger
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d
> a1.sinks.k1.hdfs.filePrefix = messages
> a1.sinks.k1.hdfs.rollInterval = 30
> #a1.sinks.k1.hdfs.rollSize = 1024
> a1.sinks.k1.hdfs.rollCount = 1000
> a1.sinks.k1.hdfs.fileType = DataStream
> 
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
> 
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> 
> Hope this helps,
> Richard.
> 
> 
> On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar <mrajaforu@gmail.com>
wrote:
> Hello Team,
> 
> I'm new to flume. Could you please help me on below query
> How to configure JMS source(TIBCO and ActiveMQ) in flume?
> Do i need to write custom source for above requirement? (Or) Out Of the Box flume JMS
source implementation is fine?
> What is the best channel mechanism for above requirement?
> Each message will be sink into HDFS? (or) Can we consolidate some messages and sink into
HDFS?
> Thank you in advance
> 
> Regards,
> Rajesh
> 
> 


Mime
View raw message