flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Unable to transport the event to Avro Source
Date Wed, 20 Jun 2012 18:49:30 GMT
Thanks Will and Hari for your valuable guidance. Now i am able to transport
the events from my external program to the Avro Source.Its pretty simple
with 4 lines of code.

Regards,
Som Shekhar

On Wed, Jun 20, 2012 at 11:35 PM, shekhar sharma <shekhar2581@gmail.com>wrote:

> Thanks Will.. i am looking into it...
>
> Regards,
> Som
>
>
> On Wed, Jun 20, 2012 at 11:32 PM, shekhar sharma <shekhar2581@gmail.com>wrote:
>
>> Hari, can you suggest me what needs to be done to convert the events into
>> Avro format. I am also trying to find out, but it would be great if you can
>> give me some pointers to start.
>>
>> Regards,
>> Som
>>
>>
>> On Wed, Jun 20, 2012 at 11:30 PM, shekhar sharma <shekhar2581@gmail.com>wrote:
>>
>>> Thanks hari for prompt reply. I dont have a sink that writes to avro
>>> source.
>>> Actually i am writing a external program which will send the events to
>>> the Avro Source. you mean to say, my external program should convert the
>>> events into avro format and then i can send those event through a socket.
>>>
>>> Regards,
>>> SOm
>>>
>>>
>>> On Wed, Jun 20, 2012 at 11:22 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>>  Hi Som,
>>>>
>>>> AvroSource is designed to received only Avro formatted events, so if
>>>> you write arbitrary data to it, the AvroSource will behave in an undefined
>>>> way. If you have a sink that writes to the AvroSource, it should be using
>>>> the Flume SDK to send the event in the Flume Avro format. If you want to
>>>> write arbitrary data, I suggest taking a look at the netcat source.
>>>>
>>>> Thanks
>>>> Hari
>>>>
>>>> --
>>>> Hari Shreedharan
>>>>
>>>> On Wednesday, June 20, 2012 at 10:48 AM, shekhar sharma wrote:
>>>>
>>>> Hello,
>>>> I am encountering an exception when i am trying to transport the events
>>>> to flume avro source.
>>>> Basically i have created an agent consisting of avro source and my
>>>> custom sink.
>>>> Previously i was using exec source, but due to its limitations i am not
>>>> using it anymore.
>>>>
>>>> Since Avro Source is listening on a port, so i have created a simple
>>>> application, which opens a socket to that port and send the events through
>>>> this socket.
>>>> Socket connection is successful, but when i send the events i got the
>>>> following error:
>>>>
>>>> 2012-06-20 10:34:15,767 INFO source.AvroSource: Avro source
>>>> starting:AvroSource: { bindAddress:localhost port:41414 }
>>>> 2012-06-20 10:35:52,588 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>>> 127.0.0.1:46240 => /127.0.0.1:41414] OPEN
>>>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>>> 127.0.0.1:46240 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
>>>> 2012-06-20 10:35:52,612 INFO ipc.NettyServer: [id: 0x00f0b4a3, /
>>>> 127.0.0.1:46240 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:46240
>>>> 2012-06-20 10:35:52,792 WARN ipc.NettyServer: Unexpected exception from
>>>> downstream.
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>         at java.util.ArrayList.<init>(ArrayList.java:112)
>>>>         at
>>>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
>>>>         at
>>>> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
>>>>         at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
>>>>         at
>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
>>>>         at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
>>>>         at
>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
>>>>         at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
>>>>         at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>>>>         at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> Is there something else i need to perform?
>>>>
>>>> Regards,
>>>> Som Shekhar Sharma
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message