flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madabhattula Rajesh Kumar <mrajaf...@gmail.com>
Subject Re: Flume JMS source
Date Thu, 06 Mar 2014 18:09:10 GMT
Hello Richard,

Thank you very much. I'll try this configuration. Could you please clarify
below questions

a1.sources.r1.providerURL = tcp://<activemqserver>:61616
a1.sources.r1.destinationName = TESTQUEUE -----> *This is queue name right?*
a1.sources.r1.destinationType = QUEUE

Regards,
Rajesh


On Tue, Mar 4, 2014 at 8:13 PM, richard ross <richard.prionses@gmail.com>wrote:

> Hello Rajesh:
>
> I just worked on a POC for the work that I am doing with JMS --> HDFS
> configuration using ActiveMQ. I was able to use everything right out of the
> box, going with JMS Source --> Memory Channel --> HDFS Sink, based on the
> guide (https://flume.apache.org/FlumeUserGuide.html). Below is my example
> config that I was playing around with. Note that for ActiveMQ, you do not
> want to specify the connectionFactory in the source configuration (
> https://flume.apache.org/FlumeUserGuide.html#jms-source). A documentation
> defect was created for this point (
> https://issues.apache.org/jira/browse/FLUME-2339). As for questions 3 and
> 4, I will let someone with more experience answer.
>
> # Name the components on this agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
>
> # Describe the source
> #a1.sources.r1.type = netcat
> #a1.sources.r1.bind = localhost
> #a1.sources.r1.port = 44444
> a1.sources.r1.type = jms
> a1.sources.r1.channels = c1
> a1.sources.r1.initialContextFactory =
> org.apache.activemq.jndi.ActiveMQInitialContextFactory
> a1.sources.r1.providerURL = tcp://<activemqserver>:61616
> a1.sources.r1.destinationName = TESTQUEUE
> a1.sources.r1.destinationType = QUEUE
>
> # Describe the sink
> #a1.sinks.k1.type = logger
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d
> a1.sinks.k1.hdfs.filePrefix = messages
> a1.sinks.k1.hdfs.rollInterval = 30
> #a1.sinks.k1.hdfs.rollSize = 1024
> a1.sinks.k1.hdfs.rollCount = 1000
> a1.sinks.k1.hdfs.fileType = DataStream
>
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
>
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
>
> Hope this helps,
> Richard.
>
>
> On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar <
> mrajaforu@gmail.com> wrote:
>
>> Hello Team,
>>
>> I'm new to flume. Could you please help me on below query
>>
>>    1. How to configure JMS source(TIBCO and ActiveMQ) in flume?
>>    2. Do i need to write custom source for above requirement? (Or) Out
>>    Of the Box flume JMS source implementation is fine?
>>    3. What is the best channel mechanism for above requirement?
>>    4. Each message will be sink into HDFS? (or) Can we consolidate some
>>    messages and sink into HDFS?
>>
>> Thank you in advance
>>
>> Regards,
>> Rajesh
>>
>
>

Mime
View raw message