flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zaenal rifai <togatta.f...@gmail.com>
Subject Re: Flume-agent conf on ambari
Date Mon, 16 Nov 2015 08:47:58 GMT
thanks gonzalo, it's running but still cant write on hdfs.

the error is
http://tempel.blankon.in/1527360

when Starting connection handler

it always Checking file:/home/hdfs/adaw.conf for changes

and i check on hdfs, there is no file

On 16 November 2015 at 15:34, Gonzalo Herreros <gherreros@gmail.com> wrote:

> I see your problem now. You are using "memory-channel" and "memoryChannel"
> to refer to the same thing.
> Change the 3rd line to:
>
> hdfs-agent.channels= memoryChannel
>
> On 16 November 2015 at 08:31, zaenal rifai <togatta.fudo@gmail.com> wrote:
>
>> 1.Yes i get error, the error is " Agent configuration for hdfs-agent does
>> not contain any valid channels. Marking it as invalid "
>>     here is for more about the error
>>
>>     http://tempel.blankon.in/1527288
>>
>> 2.No, i don't have enable the kerberos
>>
>> 3.i'm not sure about the problem, i tried another method sink and it
>> still failed.
>>
>> can you give me some simple example for agent configuration to collect
>> log and write on hdfs ?
>>
>>
>>
>>
>>
>>
>> On 16 November 2015 at 14:47, Gonzalo Herreros <gherreros@gmail.com>
>> wrote:
>>
>>> I looks fine to me. Do you get any errors?
>>> Do you have kerberos enabled? maybe is a security issue
>>> Are you sure the problem is hdfs and not some netcat error? Try with
>>> another sink to confirm that.
>>>
>>> Regards,
>>> Gonzalo
>>>
>>> On 16 November 2015 at 07:35, zaenal rifai <togatta.fudo@gmail.com>
>>> wrote:
>>>
>>>> Hello guys, i'm newbie on flume, i want to ask something about
>>>> configuration on flume-agent which can write event on HDFS.
>>>>
>>>> for example which i can find on internet is like this.
>>>>
>>>> hdfs-agent.sources= netcat-collect
>>>> hdfs-agent.sinks = hdfs-write
>>>> hdfs-agent.channels= memory-channel
>>>> hdfs-agent.sources.netcat-collect.type = netcat
>>>> hdfs-agent.sources.netcat-collect.bind = 127.0.0.1
>>>> hdfs-agent.sources.netcat-collect.port = 11111
>>>> hdfs-agent.sinks.hdfs-write.type = hdfs
>>>> hdfs-agent.sinks.hdfs-write.hdfs.path =/user/user/togatta/
>>>> hdfs-agent.sinks.hdfs-write.rollInterval = 10
>>>> hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=Text
>>>> hdfs-agent.sinks.hdfs-write.hdfs.fileType=DataStream
>>>> hdfs-agent.channels.memoryChannel.type = memory
>>>> hdfs-agent.channels.memoryChannel.capacity=10000
>>>> hdfs-agent.sources.netcat-collect.channels=memoryChannel
>>>> hdfs-agent.sinks.hdfs-write.channel=memoryChannel
>>>>
>>>> this configuration will collect about log netcat event, but when i run
>>>> it, it cant write on path hdfs which i had to config. So any suggest from
>>>> all of you guys?
>>>>
>>>
>>>
>>
>

Mime
View raw message