flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Flume-agent conf on ambari
Date Mon, 16 Nov 2015 08:34:06 GMT
I see your problem now. You are using "memory-channel" and "memoryChannel"
to refer to the same thing.
Change the 3rd line to:

hdfs-agent.channels= memoryChannel

On 16 November 2015 at 08:31, zaenal rifai <togatta.fudo@gmail.com> wrote:

> 1.Yes i get error, the error is " Agent configuration for hdfs-agent does
> not contain any valid channels. Marking it as invalid "
>     here is for more about the error
>
>     http://tempel.blankon.in/1527288
>
> 2.No, i don't have enable the kerberos
>
> 3.i'm not sure about the problem, i tried another method sink and it still
> failed.
>
> can you give me some simple example for agent configuration to collect log
> and write on hdfs ?
>
>
>
>
>
>
> On 16 November 2015 at 14:47, Gonzalo Herreros <gherreros@gmail.com>
> wrote:
>
>> I looks fine to me. Do you get any errors?
>> Do you have kerberos enabled? maybe is a security issue
>> Are you sure the problem is hdfs and not some netcat error? Try with
>> another sink to confirm that.
>>
>> Regards,
>> Gonzalo
>>
>> On 16 November 2015 at 07:35, zaenal rifai <togatta.fudo@gmail.com>
>> wrote:
>>
>>> Hello guys, i'm newbie on flume, i want to ask something about
>>> configuration on flume-agent which can write event on HDFS.
>>>
>>> for example which i can find on internet is like this.
>>>
>>> hdfs-agent.sources= netcat-collect
>>> hdfs-agent.sinks = hdfs-write
>>> hdfs-agent.channels= memory-channel
>>> hdfs-agent.sources.netcat-collect.type = netcat
>>> hdfs-agent.sources.netcat-collect.bind = 127.0.0.1
>>> hdfs-agent.sources.netcat-collect.port = 11111
>>> hdfs-agent.sinks.hdfs-write.type = hdfs
>>> hdfs-agent.sinks.hdfs-write.hdfs.path =/user/user/togatta/
>>> hdfs-agent.sinks.hdfs-write.rollInterval = 10
>>> hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=Text
>>> hdfs-agent.sinks.hdfs-write.hdfs.fileType=DataStream
>>> hdfs-agent.channels.memoryChannel.type = memory
>>> hdfs-agent.channels.memoryChannel.capacity=10000
>>> hdfs-agent.sources.netcat-collect.channels=memoryChannel
>>> hdfs-agent.sinks.hdfs-write.channel=memoryChannel
>>>
>>> this configuration will collect about log netcat event, but when i run
>>> it, it cant write on path hdfs which i had to config. So any suggest from
>>> all of you guys?
>>>
>>
>>
>

Mime
View raw message